Your may want to patch your Linux servers on a regular basis (e.g using “yum/dnf update”). As always, it’s obviously recommended to :

1) Patch the TEST systems
2) Check if there is no side effects
3) Wait few days or weeks
4) Patch the PROD systems

The problem here is that between step 1 and 4 above, a new version of the packages can be available on the public repository you use.
Consequently after step 4, you’ll have a more recent version of you packages on PROD than on TEST, which is a situation you probably want to avoid.

In my previous post, I explained how to update Linux OS with Ansible by patching all packages or installing security fixes only.
In this one, I’ll show you how to keep your different environments consistent in term of packages version, even when the patching is delayed by several weeks.

Inventory

We will use the following simple inventory for this demo :

[test]
54.93.103.197   # srvtest1
35.156.114.202  # srvtest2

[prod]
54.93.245.3     # srvprod1
3.67.8.87       # srvprod1

The VMs are running in AWS and are deployed with Terraform (have a look to my other post if you want to know how to do it).
Both host groups have a variable “gv_env” defined :

$ tree inventories/group_vars/
inventories/group_vars/
├── prod.yml
└── test.yml

0 directories, 2 files
$ cat inventories/group_vars/*
gv_env: prod
gv_env: test
$ ansible-inventory -i inventories/hosts --graph --vars
@all:
  |--@prod:
  |  |--3.67.8.87
  |  |  |--{gv_env = prod}
  |  |--54.93.245.3
  |  |  |--{gv_env = prod}
  |  |--{gv_env = prod}
  |--@test:
  |  |--35.156.114.202
  |  |  |--{gv_env = test}
  |  |--54.93.103.197
  |  |  |--{gv_env = test}
  |  |--{gv_env = test}
  |--@ungrouped:

 

Playbook

The content of my playbook file is very limited. Its purpose is only to call a role :
---
- name: Patch TEST or PROD servers
  hosts: all
  gather_facts: true

  roles:
    - os_update
...

 

Role

The main task of my role is also very short :
---
- include_tasks: test.yml
  when:
    - ev_env_to_patch == "test"
    - inventory_hostname in groups['test']

- include_tasks: prod.yml
  when:
    - ev_env_to_patch == "prod"
    - inventory_hostname in groups['prod']
...
Its aim is to call the correct tasks file depending which servers (TEST or PROD) the playbook is running for.
The choice of the servers to patch is defined by the variable ev_env_to_patch. Nothing else in this file, as all the steps are described in the tasks files.

 

Task – test.yml

The first one is to list all packages that will be modified. Thus, the user can double-check what will be installed before moving on, or cancel the process if desired :

---
- name: Get packages that can be upgraded on TEST servers
  become: true
  ansible.builtin.yum:
    list: updates
    state: latest
    update_cache: yes
  register: reg_yum_output_all

- name: List packages that will be upgraded on TEST servers
  ansible.builtin.debug:
    msg: "{{ reg_yum_output_all.results | map(attribute='name') | list }}"

- name: Request user confirmation
  ansible.builtin.pause:
    prompt: |

      The packages listed above will be upgraded. Do you want to continue ?
      -> Press RETURN to continue.
      -> Press Ctrl+c and then "a" to abort.

 

Next step is to start the update :

- name: Upgrade packages on TEST servers
  become: true
  ansible.builtin.yum:
    name: '*'
    state: latest
    update_cache: yes
    update_only: no
  register: reg_upgrade_ok
  when: 
    - reg_yum_output_all is defined

- name: Print errors if upgrade failed
  ansible.builtin.debug:
    msg: "Packages upgrade failed"
  when: reg_upgrade_ok is not defined

 

If the Kernel has been updated, it’s strongly recommended to reboot the server. To check if a reboot is required, we can use the command “needs-restarting” provided here by the package dnf-utils :

- name: Install yum-utils on TEST servers
  become: true
  ansible.builtin.yum:
    name: 'yum-utils'
    state: latest
    update_cache: yes


- name: Check if a reboot is required
  become: true
  command: needs-restarting -r
  register: reg_reboot_required
  ignore_errors: true
  failed_when: false
  changed_when: reg_reboot_required.rc != 0
  notify:
    - Reboot server

 

As you probably noticed, the actions described so far are nearly the same as in my previous post.
Once the server is back the important step is to save in a text file the packages version freshly installed :

- name: Genereate the list of the upgraded packages on TEST servers
  become: true
  shell: "rpm -qa > /tmp/packages-set.txt"
  args:
    warn: false

 

You might have imported GPG public keys into your system to verify the signature of a package before installing it. If this is the case, the keys are also added to the RPM database. Therefore it’s required to remove them from the file :

- name: Delete the fake RPM packages gpg-pubkey from the list
  become: true
  ansible.builtin.lineinfile:
    path: /tmp/packages-set.txt
    regexp: 'gpg-pubkey-.*'
    state: absent

 

Once the file is ready, we fetch it from the Ansible Control Node :

- name: Fetch the file from the Ansible Control Node
  become: true
  ansible.builtin.fetch: 
    src: /tmp/packages-set.txt
    dest: /tmp/
    flat: yes

 

Now we need a directory on the PROD servers where the file will be copied to :

- name: Create directory to store the file on PROD servers
  become: true
  ansible.builtin.file:
    path: /etc/dbi
    state: directory
    mode: '0777'
  delegate_to: "{{ item }}"
  loop: "{{ groups['prod'] }}"

I used “delegate_to” to make this task executed by the PROD servers.

And finally, we copy the file to the PROD servers :

- name: Copy the file from the Ansible Control node to the PROD servers
  ansible.builtin.copy: 
    src: /tmp/packages-set.txt
    dest: /etc/dbi
    force: true
  delegate_to: "{{ item }}"
  loop: "{{ groups['prod'] }}"
...

Everything is done on the TEST servers : patching, reboot, file containing packages version transferred to the PROD servers.
Let’s see how to patch the PROD servers…

Task – prod.yml

First of all we need to ensure that the file “packages-set.txt” is still present on the servers :

---
- name: Check if the packages list file (/etc/dbi/packages-set.txt) is present on PROD servers
  ansible.builtin.stat:
    path: /etc/dbi/packages-set.txt
  register: reg_file_status

 

As the goal is to use it for the patching, we want the playbook to fail if it is not there :

- name: Fail when file /etc/dbi/packages-set.txt does not exist
  ansible.builtin.fail: 
    msg: /etc/dbi/packages-set.txt does not exist
  when: reg_file_status.stat.exists == false

 

And now… the (easy) trick :

- name: Upgrade PROD servers to the same packages version as TEST servers
  become: true
  shell: "cat /etc/dbi/packages-set.txt | xargs yum update-to -y"
- name: Check if a reboot is required
  become: true
  command: needs-restarting -r
  register: reg_reboot_required
  ignore_errors: true
  failed_when: false
  changed_when: reg_reboot_required.rc != 0
  notify:
    - Reboot server

 

Run the playbook on the TEST servers

To patch the TEST servers, we only need to execute the playbook with the extra-var “ev_env_to_patch=test” :

$ ansible-playbook playbooks/os_update.yml -e "ev_env_to_patch=test"

PLAY [Patch TEST or PROD servers] *******************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [35.156.114.202]
ok: [3.67.8.87]
ok: [54.93.245.3]
ok: [54.93.103.197]

TASK [os_update : include_tasks] *************************************************************************************************************************************************************************************************************
skipping: [54.93.245.3]
skipping: [3.67.8.87]
included: /ansible/roles/os_update/tasks/test.yml for 35.156.114.202, 54.93.103.197

TASK [os_update : Get packages that can be upgraded on TEST servers] *************************************************************************************************************************************************************************
ok: [54.93.103.197]
ok: [35.156.114.202]

TASK [os_update : List packages that will be upgraded on TEST servers] ***********************************************************************************************************************************************************************
ok: [54.93.103.197] => {
    "changed": false,
    "msg": [
        "bash",
        "bind-export-libs",
        "bind-libs-lite",
        "bind-license",
        "binutils",
        "btrfs-progs",
        "ca-certificates",
        "cloud-init",
        "cronie-anacron",
        "cronie",
        "curl",
        "cyrus-sasl-lib",
        "device-mapper-libs",
        "device-mapper",
        "dhclient",
        "dhcp-common",
        "dhcp-libs",
        "dmidecode",
        "firewalld-filesystem",
        "gettext-libs",
        "gettext",
        "glib2",
        "glibc-common",
        "glibc",
        "grub2-common",
        "grub2-pc-modules",
        "grub2-pc",
        "grub2-tools-extra",
        "grub2-tools-minimal",
        "grub2-tools",
        "grub2",
        "iproute",
        "kbd-legacy",
        "kbd-misc",
        "kbd",
        "kernel-tools-libs",
        "kernel-tools",
        "kpartx",
        "krb5-libs",
        "libblkid",
        "libcurl",
        "libgudev1",
        "libmount",
        "libsmartcols",
        "libuuid",
        "libwebp",
        "libxml2-python",
        "libxml2",
        "linux-firmware",
        "nspr",
        "nss-softokn-freebl",
        "nss-softokn",
        "nss-sysinit",
        "nss-tools",
        "nss-util",
        "nss",
        "openldap",
        "openssh-clients",
        "openssh-server",
        "openssh",
        "openssl-libs",
        "openssl",
        "oraclelinux-release-el7",
        "pciutils-libs",
        "pciutils",
        "python-firewall",
        "python-perf",
        "rpm-build-libs",
        "rpm-libs",
        "rpm-python",
        "rpm",
        "rsyslog",
        "selinux-policy-targeted",
        "selinux-policy",
        "sudo",
        "systemd-libs",
        "systemd-sysv",
        "systemd",
        "tzdata",
        "util-linux",
        "virt-what"
    ]
}
ok: [35.156.114.202] => {
    "changed": false,
    "msg": [
        "bash",
        "bind-export-libs",
        "bind-libs-lite",
        "bind-license",
        "binutils",
        "btrfs-progs",
        "ca-certificates",
        "cloud-init",
        "cronie-anacron",
        "cronie",
        "curl",
        "cyrus-sasl-lib",
        "device-mapper-libs",
        "device-mapper",
        "dhclient",
        "dhcp-common",
        "dhcp-libs",
        "dmidecode",
        "firewalld-filesystem",
        "gettext-libs",
        "gettext",
        "glib2",
        "glibc-common",
        "glibc",
        "grub2-common",
        "grub2-pc-modules",
        "grub2-pc",
        "grub2-tools-extra",
        "grub2-tools-minimal",
        "grub2-tools",
        "grub2",
        "iproute",
        "kbd-legacy",
        "kbd-misc",
        "kbd",
        "kernel-tools-libs",
        "kernel-tools",
        "kpartx",
        "krb5-libs",
        "libblkid",
        "libcurl",
        "libgudev1",
        "libmount",
        "libsmartcols",
        "libuuid",
        "libwebp",
        "libxml2-python",
        "libxml2",
        "linux-firmware",
        "nspr",
        "nss-softokn-freebl",
        "nss-softokn",
        "nss-sysinit",
        "nss-tools",
        "nss-util",
        "nss",
        "openldap",
        "openssh-clients",
        "openssh-server",
        "openssh",
        "openssl-libs",
        "openssl",
        "oraclelinux-release-el7",
        "pciutils-libs",
        "pciutils",
        "python-firewall",
        "python-perf",
        "rpm-build-libs",
        "rpm-libs",
        "rpm-python",
        "rpm",
        "rsyslog",
        "selinux-policy-targeted",
        "selinux-policy",
        "sudo",
        "systemd-libs",
        "systemd-sysv",
        "systemd",
        "tzdata",
        "util-linux",
        "virt-what"
    ]
}

TASK [os_update : Request user confirmation] *************************************************************************************************************************************************************************************************
[os_update : Request user confirmation]

The packages listed above will be upgraded. Do you want to continue ?
-> Press RETURN to continue.
-> Press Ctrl+c and then "a" to abort.
:
ok: [54.93.103.197]

TASK [os_update : Upgrade packages on TEST servers] ******************************************************************************************************************************************************************************************
changed: [35.156.114.202]
changed: [54.93.103.197]

TASK [os_update : Print errors if upgrade failed] ********************************************************************************************************************************************************************************************
skipping: [54.93.103.197]
skipping: [35.156.114.202]

TASK [os_update : Install yum-utils on TEST servers] *****************************************************************************************************************************************************************************************
ok: [54.93.103.197]
ok: [35.156.114.202]

TASK [os_update : Check if a reboot is required] *********************************************************************************************************************************************************************************************
changed: [54.93.103.197]
changed: [35.156.114.202]

TASK [os_update : Genereate the list of the upgraded packages on TEST servers] ***************************************************************************************************************************************************************
changed: [35.156.114.202]
changed: [54.93.103.197]

TASK [os_update : Delete the fake RPM packages gpg-pubkey from the list] *********************************************************************************************************************************************************************
changed: [35.156.114.202]
changed: [54.93.103.197]

TASK [os_update : Fetch the file from the Ansible Control Node] **********************************************************************************************************************************************************
changed: [35.156.114.202]
changed: [54.93.103.197]

TASK [os_update : Create directory to store the file on PROD servers] ************************************************************************************************************************************************************************
ok: [54.93.103.197 -> 54.93.245.3] => (item=54.93.245.3)
ok: [35.156.114.202 -> 54.93.245.3] => (item=54.93.245.3)
ok: [35.156.114.202 -> 3.67.8.87] => (item=3.67.8.87)
ok: [54.93.103.197 -> 3.67.8.87] => (item=3.67.8.87)

TASK [os_update : Copy the file from the Ansible Control node to the PROD servers] ***********************************************************************************************************************************************************
ok: [54.93.103.197 -> 54.93.245.3] => (item=54.93.245.3)
changed: [35.156.114.202 -> 54.93.245.3] => (item=54.93.245.3)
ok: [35.156.114.202 -> 3.67.8.87] => (item=3.67.8.87)
changed: [54.93.103.197 -> 3.67.8.87] => (item=3.67.8.87)

TASK [os_update : include_tasks] *************************************************************************************************************************************************************************************************************
skipping: [54.93.103.197]
skipping: [35.156.114.202]
skipping: [54.93.245.3]
skipping: [3.67.8.87]

RUNNING HANDLER [os_update : Reboot server] **************************************************************************************************************************************************************************************************
changed: [54.93.103.197]
changed: [35.156.114.202]

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
3.67.8.87                  : ok=1    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
35.156.114.202             : ok=13   changed=7    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
54.93.103.197              : ok=14   changed=7    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
54.93.245.3                : ok=1    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   

$

 

Run the playbook on the PROD servers

To patch the PROD servers, we only need to execute the playbook with the extra-var “ev_env_to_patch=prod” :

$ ansible-playbook playbooks/os_update.yml -e "ev_env_to_patch=prod"

PLAY [Patch TEST or PROD servers] *******************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
ok: [54.93.245.3]
ok: [3.67.8.87]
ok: [35.156.114.202]
ok: [54.93.103.197]

TASK [os_update : include_tasks] *************************************************************************************************************************************************************************************************************
skipping: [35.156.114.202]
skipping: [54.93.103.197]
skipping: [54.93.245.3]
skipping: [3.67.8.87]

TASK [os_update : include_tasks] *************************************************************************************************************************************************************************************************************
skipping: [54.93.103.197]
skipping: [35.156.114.202]
included: /ansible/roles/os_update/tasks/prod.yml for 54.93.245.3, 3.67.8.87

TASK [os_update : Check if the packages list file (/etc/dbi/packages-set.txt) is present on PROD servers] ************************************************************************************************************************************
ok: [3.67.8.87]
ok: [54.93.245.3]

TASK [os_update : Fail when file /etc/dbi/packages-set.txt does not exist] *******************************************************************************************************************************************************************
skipping: [54.93.245.3]
skipping: [3.67.8.87]

TASK [os_update : Upgrade PROD servers to the same packages version as TEST servers] *********************************************************************************************************************************************************
changed: [54.93.245.3]
changed: [3.67.8.87]

TASK [os_update : Check if a reboot is required] *********************************************************************************************************************************************************************************************
changed: [54.93.245.3]
changed: [3.67.8.87]

RUNNING HANDLER [os_update : Reboot server] **************************************************************************************************************************************************************************************************
changed: [3.67.8.87]
changed: [54.93.245.3]

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
3.67.8.87                  : ok=6    changed=3    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
35.156.114.202             : ok=1    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
54.93.103.197              : ok=1    changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
54.93.245.3                : ok=6    changed=3    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   

$

 

Easy, isn’t ? Happy patching !

Post Views: 9,233
Thumbnail [60x60]
by
Joël Cattin