Your may want to patch your Linux servers on a regular basis (e.g using “yum\/dnf update”). As always, it’s obviously recommended to :<\/p>\n
1) Patch the TEST systems
\n2) Check if there is no side effects
\n3) Wait few days or weeks
\n4) Patch the PROD systems<\/p>\n
The problem here is that between step 1 and 4 above, a new version of the packages can be available on the public repository you use.
\nConsequently after step 4, you’ll have a more recent version of you packages on PROD than on TEST, which is a situation you probably want to avoid.<\/p>\n
In my previous post<\/a>, I explained how to update Linux OS with Ansible by patching all packages or installing security fixes only. We will use the following simple inventory for this demo :<\/p>\n The VMs are running in AWS and are deployed with Terraform<\/a> (have a look to my other post<\/a> if you want to know how to do it). <\/p>\n <\/p>\n <\/p>\n The first one is to list all packages that will be modified. Thus, the user can double-check what will be installed before moving on, or cancel the process if desired :<\/p>\n <\/p>\n Next step is to start the update :<\/p>\n <\/p>\n If the Kernel has been updated, it’s strongly recommended to reboot the server. To check if a reboot is required, we can use the command “needs-restarting” provided here by the package dnf-utils :<\/p>\n <\/p>\n As you probably noticed, the actions described so far are nearly the same as in my previous post. <\/p>\n You might have imported GPG public keys into your system to verify the signature of a package before installing it. If this is the case, the keys are also added to the RPM database. Therefore it’s required to remove them from the file :<\/p>\n <\/p>\n Once the file is ready, we fetch it from the Ansible Control Node :<\/p>\n <\/p>\n Now we need a directory on the PROD servers where the file will be copied to :<\/p>\n I used “delegate_to” to make this task executed by the PROD servers.<\/p>\n And finally, we copy the file to the PROD servers :<\/p>\n Everything is done on the TEST servers : patching, reboot, file containing packages version transferred to the PROD servers. First of all we need to ensure that the file “packages-set.txt” is still present on the servers :<\/p>\n <\/p>\n As the goal is to use it for the patching, we want the playbook to fail if it is not there :<\/p>\n <\/p>\n And now… the (easy) trick :<\/p>\n <\/p>\n To patch the TEST servers, we only need to execute the playbook with the extra-var “ev_env_to_patch=test” :<\/p>\n <\/p>\n To patch the PROD servers, we only need to execute the playbook with the extra-var “ev_env_to_patch=prod<\/strong>” :<\/p>\n <\/p>\n Easy, isn’t ? Happy patching !<\/p>\n","protected":false},"excerpt":{"rendered":" Your may want to patch your Linux servers on a regular basis (e.g using “yum\/dnf update”). As always, it’s obviously recommended to : 1) Patch the TEST systems 2) Check if there is no side effects 3) Wait few days or weeks 4) Patch the PROD systems The problem here is that between step 1 […]<\/p>\n","protected":false},"author":30,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1321,1320,42,149],"tags":[150,151,152,73,153,154,155,156,157,158],"type_dbi":[],"class_list":["post-581","post","type-post","status-publish","format-standard","hentry","category-ansible","category-devops","category-operating-systems","category-security","tag-ansible","tag-devops","tag-dnf","tag-linux","tag-operating-system","tag-os","tag-patching","tag-prod","tag-test","tag-yum"],"acf":[],"yoast_head":"\n
\nIn this one, I’ll show you how to keep your different environments consistent in term of packages version, even when the patching is delayed by several weeks.<\/p>\nInventory<\/h3>\n
[test]\n54.93.103.197 # srvtest1\n35.156.114.202 # srvtest2\n\n[prod]\n54.93.245.3 # srvprod1\n3.67.8.87 # srvprod1<\/pre>\n
\nBoth host groups have a variable “gv_env” defined :<\/p>\n$ tree inventories\/group_vars\/\ninventories\/group_vars\/\n\u251c\u2500\u2500 prod.yml\n\u2514\u2500\u2500 test.yml\n\n0 directories, 2 files<\/pre>\n
$ cat inventories\/group_vars\/*\ngv_env: prod\ngv_env: test<\/pre>\n
$ ansible-inventory -i inventories\/hosts --graph --vars\n@all:\n |--@prod:\n | |--3.67.8.87\n | | |--{gv_env = prod}\n | |--54.93.245.3\n | | |--{gv_env = prod}\n | |--{gv_env = prod}\n |--@test:\n | |--35.156.114.202\n | | |--{gv_env = test}\n | |--54.93.103.197\n | | |--{gv_env = test}\n | |--{gv_env = test}\n |--@ungrouped:<\/pre>\nPlaybook<\/h3>\n
---\n- name: Patch TEST or PROD servers\n hosts: all\n gather_facts: true\n\n roles:\n - os_update\n...<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
Role<\/h3>\n
---\n- include_tasks: test.yml\n when:\n - ev_env_to_patch == \"test\"\n - inventory_hostname in groups['test']\n\n- include_tasks: prod.yml\n when:\n - ev_env_to_patch == \"prod\"\n - inventory_hostname in groups['prod']\n...<\/pre>\n<\/div>\n
Task – test.yml<\/h3>\n
---\n- name: Get packages that can be upgraded on TEST servers\n become: true\n ansible.builtin.yum:\n list: updates\n state: latest\n update_cache: yes\n register: reg_yum_output_all\n\n- name: List packages that will be upgraded on TEST servers\n ansible.builtin.debug:\n msg: \"{{ reg_yum_output_all.results | map(attribute='name') | list }}\"\n\n- name: Request user confirmation\n ansible.builtin.pause:\n prompt: |\n\n The packages listed above will be upgraded. Do you want to continue ?\n -> Press RETURN to continue.\n -> Press Ctrl+c and then \"a\" to abort.<\/pre>\n- name: Upgrade packages on TEST servers\n become: true\n ansible.builtin.yum:\n name: '*'\n state: latest\n update_cache: yes\n update_only: no\n register: reg_upgrade_ok\n when: \n - reg_yum_output_all is defined\n\n- name: Print errors if upgrade failed\n ansible.builtin.debug:\n msg: \"Packages upgrade failed\"\n when: reg_upgrade_ok is not defined<\/pre>\n<\/div>\n
- name: Install yum-utils on TEST servers\n become: true\n ansible.builtin.yum:\n name: 'yum-utils'\n state: latest\n update_cache: yes\n\n\n- name: Check if a reboot is required\n become: true\n command: needs-restarting -r\n register: reg_reboot_required\n ignore_errors: true\n failed_when: false\n changed_when: reg_reboot_required.rc != 0\n notify:\n - Reboot server<\/pre>\n<\/div>\n<\/div>\n
\nOnce the server is back the important step is to save in a text file the packages version freshly installed :<\/p>\n- name: Genereate the list of the upgraded packages on TEST servers\n become: true\n shell: \"rpm -qa > \/tmp\/packages-set.txt\"\n args:\n warn: false<\/pre>\n
- name: Delete the fake RPM packages gpg-pubkey from the list\n become: true\n ansible.builtin.lineinfile:\n path: \/tmp\/packages-set.txt\n regexp: 'gpg-pubkey-.*'\n state: absent<\/pre>\n
- name: Fetch the file from the Ansible Control Node\n become: true\n ansible.builtin.fetch: \n src: \/tmp\/packages-set.txt\n dest: \/tmp\/\n flat: yes<\/pre>\n
- name: Create directory to store the file on PROD servers\n become: true\n ansible.builtin.file:\n path: \/etc\/dbi\n state: directory\n mode: '0777'\n delegate_to: \"{{ item }}\"\n loop: \"{{ groups['prod'] }}\"<\/pre>\n- name: Copy the file from the Ansible Control node to the PROD servers\n ansible.builtin.copy: \n src: \/tmp\/packages-set.txt\n dest: \/etc\/dbi\n force: true\n delegate_to: \"{{ item }}\"\n loop: \"{{ groups['prod'] }}\"\n...<\/pre>\n
\nLet’s see how to patch the PROD servers…<\/p>\nTask – prod.yml<\/h3>\n
---\n- name: Check if the packages list file (\/etc\/dbi\/packages-set.txt) is present on PROD servers\n ansible.builtin.stat:\n path: \/etc\/dbi\/packages-set.txt\n register: reg_file_status<\/pre>\n
- name: Fail when file \/etc\/dbi\/packages-set.txt does not exist\n ansible.builtin.fail: \n msg: \/etc\/dbi\/packages-set.txt does not exist\n when: reg_file_status.stat.exists == false<\/pre>\n
- name: Upgrade PROD servers to the same packages version as TEST servers\n become: true\n shell: \"cat \/etc\/dbi\/packages-set.txt | xargs yum update-to -y\"<\/pre>\n
- name: Check if a reboot is required\n become: true\n command: needs-restarting -r\n register: reg_reboot_required\n ignore_errors: true\n failed_when: false\n changed_when: reg_reboot_required.rc != 0\n notify:\n - Reboot server<\/pre>\n
Run the playbook on the TEST servers<\/h3>\n
$ ansible-playbook playbooks\/os_update.yml -e \"ev_env_to_patch=test\"\n\nPLAY [Patch TEST or PROD servers] *******************************************************************************************************************************************************************************\n\nTASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************\nok: [35.156.114.202]\nok: [3.67.8.87]\nok: [54.93.245.3]\nok: [54.93.103.197]\n\nTASK [os_update : include_tasks] *************************************************************************************************************************************************************************************************************\nskipping: [54.93.245.3]\nskipping: [3.67.8.87]\nincluded: \/ansible\/roles\/os_update\/tasks\/test.yml for 35.156.114.202, 54.93.103.197\n\nTASK [os_update : Get packages that can be upgraded on TEST servers] *************************************************************************************************************************************************************************\nok: [54.93.103.197]\nok: [35.156.114.202]\n\nTASK [os_update : List packages that will be upgraded on TEST servers] ***********************************************************************************************************************************************************************\nok: [54.93.103.197] => {\n \"changed\": false,\n \"msg\": [\n \"bash\",\n \"bind-export-libs\",\n \"bind-libs-lite\",\n \"bind-license\",\n \"binutils\",\n \"btrfs-progs\",\n \"ca-certificates\",\n \"cloud-init\",\n \"cronie-anacron\",\n \"cronie\",\n \"curl\",\n \"cyrus-sasl-lib\",\n \"device-mapper-libs\",\n \"device-mapper\",\n \"dhclient\",\n \"dhcp-common\",\n \"dhcp-libs\",\n \"dmidecode\",\n \"firewalld-filesystem\",\n \"gettext-libs\",\n \"gettext\",\n \"glib2\",\n \"glibc-common\",\n \"glibc\",\n \"grub2-common\",\n \"grub2-pc-modules\",\n \"grub2-pc\",\n \"grub2-tools-extra\",\n \"grub2-tools-minimal\",\n \"grub2-tools\",\n \"grub2\",\n \"iproute\",\n \"kbd-legacy\",\n \"kbd-misc\",\n \"kbd\",\n \"kernel-tools-libs\",\n \"kernel-tools\",\n \"kpartx\",\n \"krb5-libs\",\n \"libblkid\",\n \"libcurl\",\n \"libgudev1\",\n \"libmount\",\n \"libsmartcols\",\n \"libuuid\",\n \"libwebp\",\n \"libxml2-python\",\n \"libxml2\",\n \"linux-firmware\",\n \"nspr\",\n \"nss-softokn-freebl\",\n \"nss-softokn\",\n \"nss-sysinit\",\n \"nss-tools\",\n \"nss-util\",\n \"nss\",\n \"openldap\",\n \"openssh-clients\",\n \"openssh-server\",\n \"openssh\",\n \"openssl-libs\",\n \"openssl\",\n \"oraclelinux-release-el7\",\n \"pciutils-libs\",\n \"pciutils\",\n \"python-firewall\",\n \"python-perf\",\n \"rpm-build-libs\",\n \"rpm-libs\",\n \"rpm-python\",\n \"rpm\",\n \"rsyslog\",\n \"selinux-policy-targeted\",\n \"selinux-policy\",\n \"sudo\",\n \"systemd-libs\",\n \"systemd-sysv\",\n \"systemd\",\n \"tzdata\",\n \"util-linux\",\n \"virt-what\"\n ]\n}\nok: [35.156.114.202] => {\n \"changed\": false,\n \"msg\": [\n \"bash\",\n \"bind-export-libs\",\n \"bind-libs-lite\",\n \"bind-license\",\n \"binutils\",\n \"btrfs-progs\",\n \"ca-certificates\",\n \"cloud-init\",\n \"cronie-anacron\",\n \"cronie\",\n \"curl\",\n \"cyrus-sasl-lib\",\n \"device-mapper-libs\",\n \"device-mapper\",\n \"dhclient\",\n \"dhcp-common\",\n \"dhcp-libs\",\n \"dmidecode\",\n \"firewalld-filesystem\",\n \"gettext-libs\",\n \"gettext\",\n \"glib2\",\n \"glibc-common\",\n \"glibc\",\n \"grub2-common\",\n \"grub2-pc-modules\",\n \"grub2-pc\",\n \"grub2-tools-extra\",\n \"grub2-tools-minimal\",\n \"grub2-tools\",\n \"grub2\",\n \"iproute\",\n \"kbd-legacy\",\n \"kbd-misc\",\n \"kbd\",\n \"kernel-tools-libs\",\n \"kernel-tools\",\n \"kpartx\",\n \"krb5-libs\",\n \"libblkid\",\n \"libcurl\",\n \"libgudev1\",\n \"libmount\",\n \"libsmartcols\",\n \"libuuid\",\n \"libwebp\",\n \"libxml2-python\",\n \"libxml2\",\n \"linux-firmware\",\n \"nspr\",\n \"nss-softokn-freebl\",\n \"nss-softokn\",\n \"nss-sysinit\",\n \"nss-tools\",\n \"nss-util\",\n \"nss\",\n \"openldap\",\n \"openssh-clients\",\n \"openssh-server\",\n \"openssh\",\n \"openssl-libs\",\n \"openssl\",\n \"oraclelinux-release-el7\",\n \"pciutils-libs\",\n \"pciutils\",\n \"python-firewall\",\n \"python-perf\",\n \"rpm-build-libs\",\n \"rpm-libs\",\n \"rpm-python\",\n \"rpm\",\n \"rsyslog\",\n \"selinux-policy-targeted\",\n \"selinux-policy\",\n \"sudo\",\n \"systemd-libs\",\n \"systemd-sysv\",\n \"systemd\",\n \"tzdata\",\n \"util-linux\",\n \"virt-what\"\n ]\n}\n\nTASK [os_update : Request user confirmation] *************************************************************************************************************************************************************************************************\n[os_update : Request user confirmation]\n\nThe packages listed above will be upgraded. Do you want to continue ?\n-> Press RETURN to continue.\n-> Press Ctrl+c and then \"a\" to abort.\n:\nok: [54.93.103.197]\n\nTASK [os_update : Upgrade packages on TEST servers] ******************************************************************************************************************************************************************************************\nchanged: [35.156.114.202]\nchanged: [54.93.103.197]\n\nTASK [os_update : Print errors if upgrade failed] ********************************************************************************************************************************************************************************************\nskipping: [54.93.103.197]\nskipping: [35.156.114.202]\n\nTASK [os_update : Install yum-utils on TEST servers] *****************************************************************************************************************************************************************************************\nok: [54.93.103.197]\nok: [35.156.114.202]\n\nTASK [os_update : Check if a reboot is required] *********************************************************************************************************************************************************************************************\nchanged: [54.93.103.197]\nchanged: [35.156.114.202]\n\nTASK [os_update : Genereate the list of the upgraded packages on TEST servers] ***************************************************************************************************************************************************************\nchanged: [35.156.114.202]\nchanged: [54.93.103.197]\n\nTASK [os_update : Delete the fake RPM packages gpg-pubkey from the list] *********************************************************************************************************************************************************************\nchanged: [35.156.114.202]\nchanged: [54.93.103.197]\n\nTASK [os_update : Fetch the file from the Ansible Control Node] **********************************************************************************************************************************************************\nchanged: [35.156.114.202]\nchanged: [54.93.103.197]\n\nTASK [os_update : Create directory to store the file on PROD servers] ************************************************************************************************************************************************************************\nok: [54.93.103.197 -> 54.93.245.3] => (item=54.93.245.3)\nok: [35.156.114.202 -> 54.93.245.3] => (item=54.93.245.3)\nok: [35.156.114.202 -> 3.67.8.87] => (item=3.67.8.87)\nok: [54.93.103.197 -> 3.67.8.87] => (item=3.67.8.87)\n\nTASK [os_update : Copy the file from the Ansible Control node to the PROD servers] ***********************************************************************************************************************************************************\nok: [54.93.103.197 -> 54.93.245.3] => (item=54.93.245.3)\nchanged: [35.156.114.202 -> 54.93.245.3] => (item=54.93.245.3)\nok: [35.156.114.202 -> 3.67.8.87] => (item=3.67.8.87)\nchanged: [54.93.103.197 -> 3.67.8.87] => (item=3.67.8.87)\n\nTASK [os_update : include_tasks] *************************************************************************************************************************************************************************************************************\nskipping: [54.93.103.197]\nskipping: [35.156.114.202]\nskipping: [54.93.245.3]\nskipping: [3.67.8.87]\n\nRUNNING HANDLER [os_update : Reboot server] **************************************************************************************************************************************************************************************************\nchanged: [54.93.103.197]\nchanged: [35.156.114.202]\n\nPLAY RECAP ***********************************************************************************************************************************************************************************************************************************\n3.67.8.87 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 \n35.156.114.202 : ok=13 changed=7 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 \n54.93.103.197 : ok=14 changed=7 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 \n54.93.245.3 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 \n\n$<\/pre>\nRun the playbook on the PROD servers<\/h3>\n
$ ansible-playbook playbooks\/os_update.yml -e \"ev_env_to_patch=prod\"\n\nPLAY [Patch TEST or PROD servers] *******************************************************************************************************************************************************************************\n\nTASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************\nok: [54.93.245.3]\nok: [3.67.8.87]\nok: [35.156.114.202]\nok: [54.93.103.197]\n\nTASK [os_update : include_tasks] *************************************************************************************************************************************************************************************************************\nskipping: [35.156.114.202]\nskipping: [54.93.103.197]\nskipping: [54.93.245.3]\nskipping: [3.67.8.87]\n\nTASK [os_update : include_tasks] *************************************************************************************************************************************************************************************************************\nskipping: [54.93.103.197]\nskipping: [35.156.114.202]\nincluded: \/ansible\/roles\/os_update\/tasks\/prod.yml for 54.93.245.3, 3.67.8.87\n\nTASK [os_update : Check if the packages list file (\/etc\/dbi\/packages-set.txt) is present on PROD servers] ************************************************************************************************************************************\nok: [3.67.8.87]\nok: [54.93.245.3]\n\nTASK [os_update : Fail when file \/etc\/dbi\/packages-set.txt does not exist] *******************************************************************************************************************************************************************\nskipping: [54.93.245.3]\nskipping: [3.67.8.87]\n\nTASK [os_update : Upgrade PROD servers to the same packages version as TEST servers] *********************************************************************************************************************************************************\nchanged: [54.93.245.3]\nchanged: [3.67.8.87]\n\nTASK [os_update : Check if a reboot is required] *********************************************************************************************************************************************************************************************\nchanged: [54.93.245.3]\nchanged: [3.67.8.87]\n\nRUNNING HANDLER [os_update : Reboot server] **************************************************************************************************************************************************************************************************\nchanged: [3.67.8.87]\nchanged: [54.93.245.3]\n\nPLAY RECAP ***********************************************************************************************************************************************************************************************************************************\n3.67.8.87 : ok=6 changed=3 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 \n35.156.114.202 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 \n54.93.103.197 : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 \n54.93.245.3 : ok=6 changed=3 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 \n\n$<\/pre>\n