Archives des Ansible - dbi Blog

Customer case study – automating SQL Server TLS Encryption with Ansible and Certificates (Architecture)

Amine Haloui — Fri, 15 May 2026 19:35:21 +0000

When working with SQL Server environments, securing client connections can become an important requirement, especially when TLS encryption must be implemented using certificates. In this context, a customer asked us to develop an Ansible playbook and role to automate the configuration of TLS for SQL Server. The certificates are generated from the customer PKI and provided as PEM files containing the server certificate, the private key, and the certificate chain.

However, some extractions and conversions are required before these certificates can be used on Windows and configured for SQL Server.

Here, the idea is to propose a solution (the architecture) that prepares the certificate, imports it on the SQL Server host, and configures SQL Server to use it.

We will also see how to separate the preparation and activation steps in order to reduce the impact on the SQL Server service.

In this blog post, we will describe the global approach and the Ansible logic used to implement certificate-based TLS encryption for SQL Server.

Implementation logic

Before configuring TLS encryption on SQL Server, the first point was to understand the certificate format provided by the customer PKI.

In our case, the generated file is a .pem file. This file contains the server certificate used for TLS, the private key and the certificate chain with the intermediate and root certificates.

As this format cannot be directly used as-is on the Windows side for SQL Server, some extraction and conversion steps are required.

The general idea is to use the Ansible control node as a working area.

The PEM file is first copied into a temporary folder where the different parts of the certificate are extracted:

the leaf certificate
the intermediate certificate
the root certificate
the private key

These elements are then used to build a PFX file which can be imported on the Windows SQL Server host.

The PFX is installed in the LocalMachine\My certificate store while the intermediate and root certificates are imported into the appropriate Windows certificate stores.

The implementation has been designed around three different execution modes: stage, activate, and full.

The stage mode is used to prepare the certificate without any impact on the SQL Server service. It copies the PEM file, performs the extractions, builds the PFX file, copies it to the managed Windows node and imports the certificates into the Windows certificate stores. No registry change is performed, and the SQL Server service is not restarted. This mode is useful when we want to prepare the server in advance before switching SQL Server to the new certificate.

The activate mode assumes that the certificate is already present on the Windows server. Its role is to configure SQL Server to use the installed certificate and depending on the selected option, restart the SQL Server service or leave the change pending until the next planned reboot.

This can be useful when the certificate activation must be aligned with an existing maintenance window, for example during monthly OS patching.

The full mode executes the complete configuration from end to end. It performs the extraction and conversion steps, imports the certificates, grants the required permissions, configures SQL Server to use the expected certificate, and restarts the SQL Server service only if required. To avoid unnecessary impact, the role relies on the certificate thumbprint. If the expected certificate is already configured, no change is applied and the SQL Server service is not restarted. This behavior is important for idempotency.

For example, if the full mode is executed after an activate mode, nothing should be changed if the certificate is already the correct one. The same logic applies if the playbook is executed by mistake while the certificate has not been renewed.

Another point to manage is the restart of the SQL Server service. SQL Server loads the certificate configuration when the service starts. Therefore, when a new certificate is configured, the change is only effective after a restart of the SQL Server service.

For this reason the role should provide an option to control whether the restart is performed immediately or postponed to the next planned reboot.

We also have to consider DNS aliases. The standard use case is to generate a certificate containing at least the short name and the FQDN of the SQL Server host in the subjectAltName. If DNS aliases are used by client applications, they can also be added to the certificate SAN.

For example:

[alt_names]
DNS.1 = A-WS2022-2.lab.local
DNS.2 = A-WS2022-2

Finally, the customer confirmed that the private key included in the PEM file is not encrypted.

This simplifies the conversion process to PFX, but it also means that the PEM file must be handled carefully during the Ansible execution, especially in temporary folders and during file transfers. With this approach, the role provides a controlled way to prepare, activate, or fully configure TLS encryption for SQL Server while keeping the impact on the SQL Server service under control.

Logical workflow

The complete workflow can be represented as follows:

Architecture summary

The certificate manipulation is performed on the Ansible control node.

The Windows certificate import and SQL Server configuration are performed on the managed Windows SQL Server host.

This separation is useful because the PEM processing and PFX generation are handled with Linux tools such as OpenSSL while the certificate installation, private key permissions, registry configuration and SQL Server restart are handled through Windows modules and PowerShell. The design also supports a controlled deployment approach.

The certificate can first be staged without service impact then activated later during a maintenance window.

The full mode can be used when the complete implementation must be executed in a single run. The use of the certificate thumbprint is important for idempotency. It allows the role to detect whether SQL Server is already configured with the expected certificate and avoids unnecessary service restarts when no change is required.

Remarks

For certain reasons we do not disclose the code of the created role.

Thank you. Amine Haloui

L’article Customer case study – automating SQL Server TLS Encryption with Ansible and Certificates (Architecture) est apparu en premier sur dbi Blog.

How to Standardize SQL Server Disks on VMs using Ansible

Nathan Courtine — Mon, 23 Mar 2026 17:50:46 +0000

INTRODUCTION

Today, the benefits of automation no longer need much explanation: saving time, reducing human error, and ensuring every environment remains aligned with internal standards. What is less obvious, however, is how using an Ansible Playbook can provide advantages that more traditional scripting approaches — such as large PowerShell scripts — struggle to offer. That is exactly what I want to explore here.

When you complete an automated deployment of a SQL Server environment on Windows Server, there is a real sense of achievement. You have invested time and effort, and you expect that investment to pay off thanks to the reliability and repeatability of automation.

But everything changes when the next Windows Server upgrade or SQL Server version arrives… or when corporate standards evolve. Suddenly, you need to reopen a multi-thousand‑line PowerShell script and:

Integrate the required changes while keeping execution stable,
Avoid subtle but potentially critical regressions,
Maintain clear and usable logging,
Retest the entire automation workflow,
Troubleshoot new issues introduced by the modifications.

This is precisely the type of situation where Ansible becomes a far better long‑term investment. Its architecture and philosophy offer several advantages:

Native idempotence, ensuring the same result even after multiple runs,
A declarative YAML approach, focusing on the desired end state rather than the execution steps,
Windows Server and SQL Server modules, providing built‑in idempotence and saving significant time,
Agentless connectivity, simplifying deployment on new machines,
A modular structure (roles, modules, variables), making adaptation and reuse of your automation much easier.

In this article, I will give you a concrete overview by walking you through how to configure the disks required for SQL Server using Ansible.

1-Map iSCSI controllers to disk numbers

When developing an Ansible Playbook, one fundamental principle is to design for idempotence from the very start—not just rely on idempotent modules.

On Windows, disk numbering is not guaranteed: it depends on several factors – how disks are detected at startup, the firmware, and so on.
As a result, disk numbers may change from one reboot to another.

To ensure consistent and reliable execution of your deployment, this behavior must be accounted for directly in the design of your Playbook.
Otherwise, it may introduce wrong behaviors, and lead to:

formatting the wrong disk,
mounting volumes on incorrect devices,
completely breaking the SQL Server provisioning workflow.

In other words, idempotence is no longer guaranteed.

To ensure stable and predictable executions, you must determine dynamically the correct disk numbering at each execution.

You can use Get-Disk PowerShell command to achieve your goal, by searching iSCSI controller number and LUN position from Location property.

$adapter = {{ disk.adapter }}
$lun = {{ disk.lun}}
(Get-Disk | Where-Object {
      $_.Location -match "Adapter $adapter\s+:.*\s+LUN $lun"
    }).number

We have done our mapping between VM specifications and Windows disk numbers.

2-Loop SQL Server disks

Since we often have several disks to configure — Data, Logs, TempDB — we need to perform the same actions repeatedly on each disk:

dynamically determine the disk number,
initialize it in GPT,
create the partition and format the volume in NTFS with a 64 KB allocation unit size,
assign an access path (drive letter or mountpoint),
apply certain specific configuration settings, such as disabling indexing,
verify the compliance of the disk configuration.

As these actions are identical for all disks, the best approach is to factorize the tasks.
The Ansible pattern, for such scenario, is to loop that call in a dedicated Task File.

---

- name: Manage all disk properties based on Location and Target numbers
  ansible.builtin.include_tasks: disks_properties.yml
  loop:
    - name: data
      location: "{{ disk_specs.data.location }}"
      target: "{{ disk_specs.data.target }}"
      label: "{{ disk_specs.data.label }}"
      letter: "{{ disk_specs.data.letter }}"
    - name: logs
      location: "{{ disk_specs.logs.location }}"
      target: "{{ disk_specs.logs.target }}"
      label: "{{ disk_specs.logs.label }}"
      letter: "{{ disk_specs.logs.letter }}"
    - name: tempdb
      location: "{{ disk_specs.tempdb.location }}"
      target: "{{ disk_specs.tempdb.target }}"
      label: "{{ disk_specs.tempdb.label }}"
      letter: "{{ disk_specs.tempdb.letter }}"
  loop_control:
    loop_var: disk

...

3- Implement SQL Server disk configuration

Since we performed our loop in the previous section on the disks_properties.yml file, we can now implement the configuration actions inside this file.
First, we will retrieve the disk number and then begin configuring the disk according to best practices and our internal standards.

To guarantee idempotence, we will mark this step as not changed: this is only a Get action:

---

- name: Identify the {{ disk.name }} disk number
  ansible.windows.win_shell: |
    $adapter = {{ disk.target }}
    $lun = {{ disk.location }}
    (Get-Disk | Where-Object {
      $_.Location -match "Adapter $adapter\s+:.*\s+LUN $lun"
    }).number
  register: disk_num
  changed_when: false

Then, we will register the disk number as an Ansible Fact for all this task file execution call.

- name: Set fact for {{ disk.name }} disk number
  ansible.builtin.set_fact:
    "disk_number_{{ disk.name }}": "{{ disk_num.stdout | trim | int }}"

We can now initialize the disk using community.windows module. Of course, use Ansible module if possible.

The parameter disk_bps.partition_style is a variable of my Ansible Role, to guarantee GPT will be used.

- name: Initialize disks
  community.windows.win_initialize_disk:
    disk_number: "{{ lookup('vars', 'disk_number_' + disk.name) }}"
    style: "{{ disk_bps.partition_style }}"

From there, we can create our partition:

- name: Create partition with letter {{ disk.letter }} for disk {{ disk.name }}
  community.windows.win_partition:
    drive_letter: "{{ disk.letter }}"
    partition_size: "-1"
    disk_number: "{{ lookup('vars', 'disk_number_' + disk.name) }}"

And now format our volume with allocation unit size 64KB:

- name: Create a partition letter {{ disk.letter }} on disk {{ disk.name }} with label {{ disk.label }}
  community.windows.win_format:
    drive_letter: "{{ disk.letter }}"
    allocation_unit_size: "{{ disk_bps.allocation_unit_size_bytes }}"
    new_label: "{{ disk.label }}"

...

As I mentioned earlier in previous section, we can also add tasks relative to some specific standards or a tasks to guarantee disk compliance.

4- Execute the Playbook

Now that our Ansible Role windows_disks is ready, we can call it through a Playbook.
Of course, we must adjust the reality of the iSCSI configuration of the Virtual Machine.

---

- name: Configure Disks by detecting Disk Number
  hosts: Raynor
  gather_facts: false
  vars:
    disk_specs:
      data:
        location: 0
        target: 1
        label: SQL_DATA
        letter: E
      logs:
        location: 0
        target: 2
        label: SQL_TLOG
        letter: L
      tempdb:
        location: 0
        target: 3
        label: SQL_TEMPDB
        letter: T
  tasks:
    - name: gather facts
      ansible.builtin.setup:
      changed_when: false
      tags: [always]
    - name: Configure Disks
      ansible.builtin.import_role:
        name: windows_disks
      tags: windows_disks


...

CONCLUSION

We have had an overview of how Ansible makes automation easier to maintain and to evolve, by focusing on the logic of our deployment and not on the code to achieve it.
Now, updating your standards or upgrading versions will no longer require rewriting scripts, but mainly adapting variables.

However, it is important to be aware that idempotence must also be maintained through design.

L’article How to Standardize SQL Server Disks on VMs using Ansible est apparu en premier sur dbi Blog.

Errorhandling in Ansible

Martin Bracher — Thu, 18 Sep 2025 13:54:32 +0000

Errors in tasks – abort or ignore?

Per default, if a task in a playbook fails, then the execution of the playbook is stopped for that host.

- name: PLAY1
  hosts: localhost
  gather_facts: no
  tasks:
    - name: let this shell-command fail
      ansible.builtin.shell: exit 1

    - name: let this shell-command complete
      ansible.builtin.shell: exit 0

As you can see, the 2nd task is not executed. If you want to continue in such a case, the ignore_errors parameter is your friend

    - name: let this shell-command fail
      ansible.builtin.shell: exit 1
      ignore_errors: true

Custom error conditions

Per default, Ansible evaluates the exit-code of the module, in case of the shell-module, the exit-code of the last command.

But for some commands, that is not adequate. Example: The Oracle commandline tool sqlplus to submit sql-commands will have an exit-code of 0 if it can connect to the database-instance. It is not related to the result of your SQL-commands. Error-messages in Oracle are prefixed by ORA-.

So, if you want to check for application errors, you have to implement it yourself. For that, you can use the failed_when option.

    - name: let this shell-command fail
      ansible.builtin.shell: |
        . ~/.profile
        echo "select count(*) from all_users;" | sqlplus / as sysdba
      register: number_of_users
      failed_when: "'ORA-' in number_of_users.stdout"

Caution: In this case the exit-code of the shell is no longer evaluated. To also get the exit-code of the sqlplus call (e.g., sqlplus can not connect to the database, or sqlplus binary not found), you have to add this (default) condition:

      failed_when: "number_of_users.rc != 0 or 'ORA-' in number_of_users.stdout"

But caution! Not all modules will have an rc field.

Tuning: Perform several checks at once

Conceptually, Ansible is not the fastest tool. For each task, it will usually login with ssh to the remote server. If you have to run several checks in the shell, then, instead of running each in a separate task, you can run all these check-commands in one shell-task, and evaluate the result afterwards.

    - name: run many check commands
      ansible.builtin.shell: |
        mount | grep ' on /u00 '  #check if /u00 is mounted
        rpm -q ksh 2>&1  #check if ksh is installed
        exit 0 # do not fail if rpm exit != 0; we will do our own errorhandling
      register: checks

    - name: fail if no /u00 is mounted
      fail:
        msg: "/u00 is not mounted"
      when: "' on /u00 ' not in checks.stdout"

    - name: No ksh found, try to install it
      yum:
        name: ksh
        state: present
      when: "'package ksh is not installed' in checks.stdout"

If you only want to throw an error, then you can do it directly in the shell-task:

when: "' on /u00 ' not in checks.stdout" or 'package ksh is not installed' in checks.stdout

But if you parse the output afterwards, you can run tasks to fix the error.

Sometimes it is difficult to parse the output if some commands return the same output, e.g. “OK”.
If your check-commands always return exactly 1 line, then you can directly parse the output of the command. The output of the 3rd command is in checks.stdout_lines[2].
In the above example that will not work because grep will return the exit-code 0 (not found) or 1 (found) plus the found line. So, expand it as: mount | grep ' on /u00 ' || echo error

Print errormessages more readable

Do not fail the task itself, it is very usually unreadable because all information is printed on one line.

Instead, use ignore_errors: true and failed_when: false in the task. Do the errorhandling in a separate task with a customized errormessage. To print the multiline list of stdout_lines, use debug: otherwise you can directy use ansible.builtin.fail: with a customized message:

    - name: force sqlplus to throw error because of missing environment
      ansible.builtin.shell: |
        /u00/app/oracle/product/19.7.0/bin/sqlplus -L / as sysdba @myscript.sql 2>&1
      register: checks
      ignore_errors: true
      failed_when: false

    - name: Check for errors of task before
      debug: var=checks.stdout_lines
      failed_when: checks.rc != 0 or 'ORA-' in checks.stdout
      when: checks.rc != 0 or 'ORA-' in checks.stdout

Re-run failed hosts

As an example for this scenario: A customer of mine will do an offline backup of the development databases. And I have to run an Ansible playbook against all databases. If the playbook for this host is run at the backup time, it will fail because the database is down. But after some minutes the database will be restarted.

What we can do now?

Wait until the database is up again. That is possible, see the example of ansible.builtin.wait_for in my blog post Parallel execution of Ansible roles. But for this scenario it is a waste of time. The database can be stopped (not for backup) and will not be restarted within the next few minutes.

Try later after a while. My playbook for all hosts (parallel forks=5) takes about 1 hour. The idea now is to remember the host with the stopped database and to continue with the next host. After the play finished for all hosts, restart the play for the remembered hosts.

The 1st play running against all database hosts:
gets the status of the databases on the host
assigns the database instances to a is_running and a not_open list
Include the role to run against the running databases
Dynamically add the host to the group re_run_if_not_open if there are not_open databases

The next play only runs for the re_run_if_not_open group
Include the role to run against the (hopefully now running) databases
If the database then is still down, we assume it is stopped permanently.

L’article Errorhandling in Ansible est apparu en premier sur dbi Blog.

Parallel execution of Ansible roles

Martin Bracher — Tue, 10 Jun 2025 18:56:18 +0000

Introduction

You can run a playbook for specific host(s), a group of hosts, or “all” (all hosts of the inventory).

Ansible will then run the tasks in parallel on the specified hosts. To avoid an overload, the parallelism – called “forks” – is limited to 5 per default.

A task with a loop (e.g. with_items:) will be executed serially per default. To run it in parallel, you can use the “async” mode.

But unfortunately, this async mode will not work to include roles or other playbooks in the loop. In this blog post we will see a workaround to run roles in parallel (on the same host).

Parallelization over the ansible hosts

In this example, we have 3 hosts (dbhost1, dbhost2, dbhost3) in the dbservers group
(use ansible-inventory --graph to see all your groups) and we run the following sleep1.yml playbook

- name: PLAY1
  hosts: [dbservers]
  gather_facts: no
  tasks:
    - ansible.builtin.wait_for: timeout=10

The tasks of the playbook will run in parallel on all hosts of the dbservers group, but not more at the same time as specified with the “forks” parameter. (specified in ansible.cfg, shell-variable ANSIBLE_FORKS, commandline parameter –forks)
https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_strategies.html

$ time ansible-playbook sleep1.yml --forks 2
...
ok: [dbhost1]  #appears after 10sec
ok: [dbhost2]  #appears after 10sec
ok: [dbhost3]  #appears after 20sec
...
real    0m22.384s

With forks=2 the results of dbhost1 and dbhost2 will both be returned after 10 seconds (sleep 10 in parallel). dbhost3 has to wait until one of the running tasks is completed. So the playbook will complete after approx. 20 seconds. If forks is 1, then it takes 30s, if forks is 3, it takes 10s (plus overhead).

Parallelization of loops

Per default, a loop is not run in parallel

- name: PLAY2A
  hosts: localhost
  tasks:
    - set_fact:
        sleepsec: [ 1, 2, 3, 4, 5, 6, 7 ]

    - name: nonparallel loop
      ansible.builtin.wait_for: "timeout={{item}} "
      with_items: "{{sleepsec}}"
      register: loop_result

This sequential run will take at least 28 seconds.

To run the same loop in parallel, use “async”
https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_async.html

- name: PLAY2B
  hosts: localhost
  gather_facts: no
  tasks:
    - name: parallel loop
      ansible.builtin.wait_for: "timeout={{item}}"
      with_items: "{{sleepsec}}"
      register: loop_result
      async: 600  # Maximum runtime in seconds. Adjust as needed.
      poll: 0     # Fire and continue (never poll here)

    # in the meantime, you can run other tasks

    - name: Wait for parallel loop to finish (poll)
      async_status:
        jid: "{{ item.ansible_job_id }}"
      register: loop_check
      until: loop_check.finished
      delay: 1      # Check every 1 seconds
      retries: 600  # Retry up to 600 times. 
                    # delay*retries should be "async:" above
      with_items: "{{ loop_result.results }}"

In the first task we start all sleeps in parallel. It will timeout after 600 seconds. We will not wait for the result (poll: 0). A later task polls the background processes until all parallel loops are finished. This execution only takes a little bit more than 7 seconds (the longest sleep plus some overhead). Between the loop and the poll you can add other tasks to use the waiting time for something more productive. Or if you know your loop takes at least 1 minute, then you can add to reduce the overhead of the polling loop, an ansible.builtin.wait_for: "timeout=60".

For example, we have an existing role to create and configure a new useraccount with many, sometimes longer running steps, e.g. add to LDAP, create NFS share, create a certificate, send a welcome-mail, ….; most of these tasks are not bound to a specific host, and will run on “localhost” calling a REST-API.

The following code example is a dummy role for copy/paste to see how it works with parallel execution.

# roles/create_user/tasks/main.yml    
    - debug: var=user
    - ansible.builtin.wait_for: timeout=10

Now we have to create many useraccounts and would like to do that in parallel. We use the code above and adapt it:

- name: PLAY3A
  hosts: localhost
  gather_facts: no
  tasks:
    - set_fact:
        users: [ 'Dave', 'Eva', 'Hans' ]

    - name: parallel user creation
      ansible.builtin.include_role: name=create_user
      with_items: "{{users}}"
      loop_control:
        loop_var: user
      register: loop_result
      async: 600
      poll: 0

But unfortunately, Ansible will not accept include_role:
ERROR! 'poll' is not a valid attribute for a IncludeRole

The only solution is to rewrite the role and to run every task with the async mode.

But is there no better solution to re-use existing roles? Let’s see…

Parallel execution of roles in a loop

As we already know

Ansible can run playbooks/tasks in parallel over different hosts (hosts parameter of the play).
Ansible can run tasks with a loop in parallel with the async option, but
Ansible can NOT run tasks with a loop in parallel for include_role or include_tasks

So, the trick will be to run the roles on “different” hosts. There is a special behavior of localhost. Well-known is the localhost IP 127.0.0.1; But also 127.0.0.2 to 127.255.255.254 refer to localhost (check it with ‘ping’). For our create-user script: we will run it on “different” localhosts in parallel. For that, we create a host-group at runtime with localhost addresses. The number of these localhost IP’s is equal to the number of users to create.

users[0] is Dave. It will be created on 127.0.0.1
users[1] is Eva. It will be created on 127.0.0.2
users[2] is Hans. It will be created on 127.0.0.3
…

- name: create dynamic localhosts group
  hosts: localhost
  gather_facts: no
  vars:
    users: [ 'Dave', 'Eva', 'Hans' ]
  tasks:
    # Create a group of localhost IP's; 
    # Ansible will treat it as "different" hosts.
    # To know, which locahost-IP should create which user:
    # The last 2 numbers of the IP matches the element of the {{users}} list:
    # 127.0.1.12 -> (1*256 + 12)-1 = 267 -> users[267]
    # -1: first Array-Element is 0, but localhost-IP starts at 127.0.0.1
    - name: create parallel execution localhosts group
      add_host:
        name: "127.0.{{item|int // 256}}.{{ item|int % 256 }}"
        group: localhosts
      with_sequence:  start=1  end="{{users|length}}" 

- name: create useraccounts
  hosts: [localhosts]  # [ 127.0.0.1, 127.0.0.2, ... ]
  connection: local
  gather_facts: no
  vars:
    users: [ 'Dave', 'Eva', 'Hans' ]
  # this play runs in parallel over the [localhosts] 
  tasks:
    - set_fact:
        ip_nr: "{{ inventory_hostname.split('.') }}"

    - name: parallel user creation
      ansible.builtin.include_role:
        name: create_user
      vars:
        user: "{{ users[ (ip_nr[2]|int*256 + ip_nr[3]|int-1) ] }}"

In this example: With forks=3 it runs in 11 seconds. With forks=1 (no parallelism) it takes 32 seconds.

The degree of parallelism (forks) depends on your use-case and your infrastructure. If you have to restore files, probably the network-bandwith, disk-I/O or the number of tape-slots is limited. Choose a value of forks that does not overload your infrastructure.

If some tasks or the whole role has to be run on another host than localhost (e.g. create a local useraccount on a server), then you can use delegate_to: "{{remote_host}}".

This principle can ideally be used for plays that are not bound to a specific host, usually for tasks that will run from localhost and calling a REST-API without logging in with ssh to a server.

Summary

Ansible is optimized to run playbooks on different hosts in parallel. The degree of parallelism can be limited by the “forks” parameter (default 5).

Ansible can run loops in parallel with the async mode. Unfortunately that does not work if we include a role or tasks.

The workaround to run roles in parallel on the same host is to assign every loop item to a different host, and then to run the role on different hosts. For the different hosts we can use the localhost IP’s between 127.0.0.1 and 127.255.255.254 to build a dynamic host-group; the number corresponds to number of loop items

L’article Parallel execution of Ansible roles est apparu en premier sur dbi Blog.

Ansible: reduce output in loops – or replace loops

Martin Bracher — Fri, 23 May 2025 10:32:09 +0000

Reduce the output

If you do a loop in Ansible, it always prints the item variable for each loop. “item” is the current list element. If you are looping over a list with small values, then the output is OK.

    - set_fact:
        myList: [ "a", "b", "c", "d" ]

   - name: simple loop
      debug:
        msg: "value of item is {{ item }}."
      with_items: "{{myList}}"
      register: loop_result

TASK [simple loop] ***********************************************
ok: [localhost] => (item=a) => {
    "msg": "value of item is a."
}
ok: [localhost] => (item=b) => {
    "msg": "value of item is b."
}
ok: [localhost] => (item=c) => {
    "msg": "value of item is c."
}
ok: [localhost] => (item=d) => {
    "msg": "value of item is d."
}

But if the loop variable is a list of dict, especially with many or big values, then the log-output of Ansible gets very unreadable.

Example: you have to deploy more than one trusted certificate to an Oracle wallet (a PKCS#12 container file). The size of a certificate is usually 1-4 KB. That means, if you deploy 4 certificates à 3KB, your screen is spammed by approx. 12’000 characters. And I think you are not interested in the certificate content. Isn’t it?

Variable trusted_certs (certificate shortened for better readability)

       trusted_certs:
          - { issuer: "letsencrypt",
              certificate: "-----BEGIN CERTIFICATE-----\nMIIFBjCCAu6gAwIBAgI....MEZSa8DA\n-----END CERTIFICATE-----" }
          - { issuer: "verisign",
              certificate: "-----BEGIN CERTIFICATE-----\nMIICkDCCAXgCAQIwHDq....kMJVW2X=\n-----END CERTIFICATE-----" }

With the following task we add the certificate to the Oracle wallet

    - name: Add trusted certificates to wallet
      ansible.builtin.shell:  |
        echo "{{ item.certificate }}" > {{ item.issuer }}.cert.pem
        orapki wallet add -wallet . -cert {{ item.issuer }}.cert.pem -trusted_cert -pwd {{ walletpw }}
      with_items: "{{trusted_certs}}"

But how we can get rid of this annoying output?

One possibility is to use the no_log: true option. Then, item will be replaced by “None” in the output.

- name: Add trusted certificates to wallet
  ansible.builtin.shell:  |
    echo "{{ item.certificate }}" > {{ item.issuer }}.cert.pem
    orapki wallet add -wallet . -cert {{ item.issuer }}.cert.pem -trusted_cert -pwd {{ walletpw }}
  with_items: "{{trusted_certs}}"
  no_log: true

Nice 🙂 But in case of errors, we have a problem to see what went wrong 🙁

Another approach is: Instead of looping over the certificate list, create a list with the index numbers of your certificate list.

    - name: Add trusted certificates to wallet
      ansible.builtin.shell:  |
        echo "{{ trusted_certs[item|int].certificate }}" > {{ trusted_certs[item|int].issuer }}.cert.pem
        orapki wallet add -wallet . -cert {{ trusted_certs[item|int].issuer }}.cert.pem -trusted_cert -pwd {{ walletpw }}
      with_sequence: start=0 end={{ trusted_certs|length-1 }}

We will create a sequence to identify the list elements using with_sequence: The 1st list element is [0] (start=0) and the last one is the number of elements -1 (`end={{ trusted_certs|length-1 }}`).

For example, to access the 1st certificate (index number 0) we can use {{ trusted_certs[0].certificate }}. In the loop we replace the [0] by [index|int] to access all elements. Caution! Ansible is not able to recognize the sequence as numbers and treats it as string, so you have to explicitly convert it to int.

And now, the output is very compact and readable 🙂

Replace loops

Loops are easy to implement and to understand. It is nice for a few list elements, but if you have thousands of elements, then you get a big amount of output, and it is slow.

If you want to see what certificates are added (only the issuer field), you can do it similar to the task to add certificates:

    - name: Show trusted certificates to add
      ansible.builtin.debug:  var=trusted_certs[item|int].certificate
      with_sequence: start=0 end={{ trusted_certs|length-1 }}

Or you can use json_query

    - name: Show trusted certificates to add (with json_query)
      ansible.builtin.debug:  var=trusted_certs|json_query('[].issuer')

json_query: [] is the filter criteria. In this case, we will not filter the list elements. And .issuer means, to only return the issuer field and not the certificate field.

If you run a playbook for many servers (e.g. hosts: [dbservers], a group with all database hosts) and you need information for the hosts from an ldap directory (or a REST web-service), then it is often faster to get once the whole data and then to extract the information for the hosts with json_query, instead of querying each host individually in ldap.

Example: If you have to know all databases on a server and you have stored all the connect strings (orclNetDescString) in an ldap directory, then you can get the information from there. For simplicity, we assume that:

orclNetDescString: contains no spaces, all keywords are in uppercase and HOST=...contains the {{inventory_hostname}}
cn: contains the database/instance name

    - name: "get databases on the server"
      community.general.ldap_search:
        server_uri: "{{ ldap_server }}"
        bind_dn: ""
        dn: "cn=OracleContext,dc=domain,dc=com"
        filter: "(&(orclNetDescString=*HOST*{{inventory_hostname}})(objectclass=orclNetService))"
        scope: "children"
        attrs:
          - cn
          - orclNetDescString
        register: ldap1
    - debug: var=ldap1.results

It will return all databases for this host, in this example exactly one:

ldap1.results: [
  {
    "cn": "DB1"
    "dn": "cn=DB1,cn=oracleContext,dc=domain,dc=com",
    "orclNetDescString": "(DESCRIPTION =(ADDRESS=(PROTOCOL=TCP)(HOST=srv01)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=DB01.domain.com)))"
  }
]

You have to do that for the [dbservers] hosts. If you have 1000 database servers, you will run this task 1000 times.

Instead of querying LDAP for each hostname, we can query it for all hosts, and then extract the required data from the result for the current host with json_query.

    - name: "get databases of all servers"
      run_once: true
      community.general.ldap_search:
        server_uri: "{{ ldap_server }}"
        bind_dn: ""
        dn: "cn=OracleContext,dc=domain,dc=com"
        filter: "(objectclass=orclNetService)"
        scope: "children"
        attrs:
          - cn
          - orclNetDescString
      register: ldap2

With run_once it will only be executed once on the 1st host, but the result is available for all hosts. All the hosts then can query this result to get the records of the own host:

- name: show all databases on this host
  debug: var=ldap2.results|json_query(jquery)
  vars:
    jquery: "[? contains(orclNetDescString, `HOST={{inventory_hostname}}`)].cn"

Summary

The output of a loop (loop, with_items, …) is annoying if the list item contains many or big values. To reduce the output you can:

use the no_log: true option. Then the item-values will no longer be printed. But in case of an error, you will no longer get the error-message.

Use the index of the list-elements. In the output you will see the index-number, and in case of errors, you will also see the error-message.

For filtering data, it is more efficient to use json_query instead of looping over all list items.

If you get data from an LDAP directory or a Webservice, try to fetch once all data and then to extract from this result the host-specific data with json_query.

L’article Ansible: reduce output in loops – or replace loops est apparu en premier sur dbi Blog.

YaK Core – The Holy Grail for Deploying Ansible Code Everywhere

Hervé Schweitzer — Tue, 29 Apr 2025 13:52:46 +0000

YaK core Multi-Platform open source Automation Tool simplifies the deployment of Ansible playbooks through a clean UI and API. It offers an intuitive interface where users can upload playbooks, configure parameters, and deploy them seamlessly across various platforms, and all managed through a centralized inventory stored in a PostgreSQL database. With YaK Core, developers can focus on writing application code without worrying about infrastructure setup or management.

YaK consists of two parts: YaK Core, which is open source, and YaK Components, which can be installed on top. These YaK Components are platform-agnostic service packages (e.g., PostgreSQL, Oracle DB, MongoDB, Kubernetes, etc.), written in Ansible by experts. They provide essential operational features such as backup, patching, upgrades, and high availability. If you’d like to learn more about the available YaK components, feel free to contact us!

But that’s not all. YaK Core also lets you create your own YaK Components . Once created, your component becomes immediately available for deployment across all platforms supported by YaK Core.

In this blog, I’ll show you how easy it is to create your own YaK Component using Ansible, upload it to YaK Core, and deploy it across any supported platform.

YaK Demo platform provisioning

To get started with YaK Core Multi-Platform open source solution, visit https://yak4all .io and provision your own YaK demo environment (take 5 minutes to be ready).

Demo

Build your YaK Component

To build a YaK Component, you need to declare at least the following three files

1. playbooks/create_linux_users.yml
2. manifest.yml
3. yak_variables_specifications/basic_variables_specifications.yml

1. The Ansible Playbook

playbooks/create_linux_users.yml
This file is simply your Ansible playbook, nothing more. The only requirement is that the code uses variables, which will be exposed in the UI for configuration. The example playbook below will create a user and optionally grant them sudo privileges.

---
- name: Create Linux users
  hosts: linux_hosts
  become: true
  gather_facts: true

  tasks:
    - debug:
        var: user

    - name: Create users
      ansible.builtin.user:
        name: "{{ item.username }}"
        create_home: "{{ item.create_home | default(true) }}"
        state: present
      loop: "{{ user }}"

    - name: Add users to sudoers
      community.general.sudoers:
        name: "yak-sudoer-{{ item.username }}"
        user: "{{ item.username }}"
        commands: ALL
        state: present
      loop: "{{ user }}"
      when: item.is_sudoer
        
  post_tasks:
    - name: Update component state
      delegate_to: localhost
      yak.core.yak_component_state_update:
        component_state_name: 'deployed'
...

2. Manifest file

manifest.yml
This file contains the basic information about your component and specifies which playbooks can be executed.

name: linux_users

version:
  major: 1
  minor: 0
  patch: 0

sub_component_types:
  - display_label: Linux users
    name: create_linux_users
    features:
      - display_label: Create Linux users
        name: create_linux_users
        playbook_name: playbooks/create_linux_users.yml

    inventory_maps:
      - group_name: linux_hosts
        group_nicename: Linux hosts
        group_description: Host on which the users will be created
        group_min_hosts: 1
        group_max_hosts: 100
        type: host
        os_type: Linux

3. Variable specification file

yak_variables_specifications/basic_variables_specifications.yml
Now you can define and provide all the specifications for the variables you want to make configurable, with all the required settings .

- variableName: user
  niceName: Users to create
  dataType: array
  children:
    - variableName: username
      niceName: Username
      dataType: string
      mandatory: true
      defaultValue: yak
      isOneOffSetting: false
      usage: Name of the user to create

    - variableName: create_home
      niceName: Create Home directory
      dataType: boolean
      mandatory: true
      defaultValue: true
      isOneOffSetting: false
      usage: Tick the box if you want to create a Home directory for the user (/home/)

    - variableName: is_sudoer
      niceName: Grant sudo privileges
      dataType: boolean
      mandatory: true
      defaultValue: true
      isOneOffSetting: false
      usage: Tick the box if you want to grant "ALL" privileges escalation to the user

That’s it! You now have all the necessary files for your first YaK Component. Next, create a ZIP package and upload it to your deployed YaK Demo environment.

To make things easier, I’ve created a ZIP file that you can upload directly. : create_linux_user.zip

Setup a Server

For this task, simply follow the documentation below up to Step 4: Deploy your server https://dbi-services.gitbook.io/yak-user-doc/introduction/yak-demo

Declare and deploy your Component

You’re now ready to declare and deploy your component!

1. Declare

YaK UI -> Components -> Declare -> Component_type : linux_users -> Save

2 Deploy

YaK UI -> Components -> Select LinuxUser -> Action -> Create Linux User -> Confirm

Conclusion

This component can now be deployed on any cloud platform or integrated on On-Premises environment using the YaK UI, and can also be deployed in parallel on up to 100 servers, as specified in your Manifest file.

With this solution, you can provide your colleagues with an intuitive and efficient way to work with Ansible playbooks, enhancing their overall experience .

For more Information about YaK see the blogs available here : https://www.dbi-services.com/blog/yak

L’article YaK Core – The Holy Grail for Deploying Ansible Code Everywhere est apparu en premier sur dbi Blog.

Migrating AWX from one Kubernetes cluster to another: A custom approach

Donovan Winter — Mon, 14 Oct 2024 08:25:54 +0000

In this guide, we’ll walk through migrating an AWX instance from one Kubernetes infrastructure to another, with two important considerations. First, both AWX instances are on completely different networks, meaning there’s no direct connectivity between them. Second, we aim to replicate the credentials (including passwords) stored in AWX, which requires careful handling. This approach differs from the official documentation due to these two specific constraints.

Step 1: Backup AWX on the old infrastructure

To back up AWX on the old infrastructure, we’ll use the AWXBackup resource provided by the AWX Operator. This will capture all necessary configurations, including credentials, job templates, and database data.

Create the AWXBackup resource

apiVersion: awx.ansible.com/v1beta1
kind: AWXBackup
metadata:
  name: awx-backup
  namespace: 
spec:
  deployment_name:

Apply the backup configuration

kubectl apply -f awxbackup.yaml

Verify the backup
Check the status of the AWXBackup resource to ensure the backup is complete

kubectl get awxbackup -n

Access the backup data
AWXBackup creates a PVC to store the backup data. We need to retrieve it.

kubectl get pvc -n

Mount the backup PVC
Create a temporary pod to access the backup files

apiVersion: v1
kind: Pod
metadata:
  name: awx-backup-access
  namespace: 
spec:
  containers:
  - name: backup-container
    image: busybox:latest
    command: ["/bin/sh", "-c", "sleep 3600"]
    volumeMounts:
    - mountPath: /backup-data
      name: awx-backup-pvc
  volumes:
  - name: awx-backup-pvc
    persistentVolumeClaim:
      claimName:

Compress the backup
Once inside the pod, go to the backup directory and archive the latest directory

kubectl exec -it awx-backup-access -n  -- /bin/sh
cd /backup-data
ls -l
## Find the latest directory
tar -czvf /awx_backup.tar.gz

Copy the archive locally
Use kubectl cp to copy the archive from the pod to your local machine

kubectl cp /awx-backup-access:/backup-data/awx_backup.tar.gz ./awx_backup.tar.gz

Clean up the temporary pod
Once the backup is copied, delete the temporary pod

kubectl delete pod awx-backup-access -n

Recover the decryption key for secret keys

kubectl get secrets -n  -secret-key -o jsonpath='{.data.secret_key}' && echo;

Save the base 64 encrypted key, we will need it for during the restoring step.

Step 2: Setup the new AWX instance

On the new infrastructure, we first need to install AWX via the AWX Operator.

Install the AWX Operator
For this step follow the official documentation of AWX Operator
Maybe you will need to deploy AWX using a local repository, you can read my other article: Deploy awx-operator with Helm using images from a local registry

Verify the AWX deployment
Check that the new AWX instance is up and running

kubectl get awx -n

Step 3: Backup AWX on the new infrastructure

Next, we need to create an AWXBackup on the new infrastructure.

Create an AWXBackup for the backup data
Create the awxbackup.yaml file:

apiVersion: awx.ansible.com/v1beta1
kind: AWXBackup
metadata:
  name: awx-backup-migration
  namespace: 
spec:
  deployment_name:

Apply the backup configuration

kubectl apply -f awxbackup.yaml

Verify the backup
Check the status of the AWXBackup resource to ensure the backup is complete

kubectl get awxbackup -n

Identify the backup PVC

kubectl get pvc -n

Step 4: Transfer and restore the backup on the new infrastructure

Now that the new AWX is set up, we’ll transfer the backup data and restore it.

Transfer the backup archive
Copy the awx_backup.tar.gz file to the new infrastructure by uploading it to the new backup PVC using a temporary pod

Create a temporary pod to restore data
Create the awx-restore-access.yaml file

apiVersion: v1
kind: Pod
metadata:
  name: awx-backup-restore
  namespace: 
spec:
  containers:
  - name: restore-container
    image: busybox:latest
    command: ["/bin/sh", "-c", "sleep 3600"]
    volumeMounts:
    - mountPath: /backup-data
      name: awx-backup-pvc
  volumes:
  - name: awx-backup-pvc
    persistentVolumeClaim:
      claimName:

kubectl apply -f awx-restore-access.yaml

Use kubectl cp to upload the archive to the pod

kubectl cp ./awx-backup-migration.tar.gz /awx-restore-access:/awx_backup.tar.gz

Replace the data from archive
Inside the pod, extract the archive

kubectl exec -it awx-backup-restore -n  -- /bin/sh
cd /backup-data
ls -l
## Find the latest directory

rm -rf /backup-data//{tower.db,awx-objects}
cd
tar -xzvf /awx_backup.tar.gz

cp backup-data//tower.db /backup-data//.
cp backup-data//awx-objects /backup-data//.

vi /backup-data//secret.yml
## Replace the value of the variable
## secrets:
##   secretKeySecret:
##     data: {secret_key: ###insert here the base64 of the decryption key recover at the end of the Step 1### }

Create the AWXRestore resource
Create an AWXRestore resource to apply the backup

apiVersion: awx.ansible.com/v1beta1
kind: AWXRestore
metadata:
  name: awx-backup-restore
  namespace: 
spec:
  deployment_name: 
  backup_name: awx-backup-migration
  no_log: false
  force_drop_db: true

Apply the AWXRestore

kubectl apply -f awxrestore.yaml

Monitor the restoration
Ensure the restoration completes successfully

kubectl get awxrestore -n

Step 5: Log in to the new infrastructure AWX

You can log in as admin (the password will be that of the old infrastructure)
Explore the various AWX resources and check that everything has been migrated correctly
Run a template job to validate correct operation

Conclusion

By following this procedure, we’ve successfully migrated an AWX instance across two isolated Kubernetes clusters while maintaining full fidelity of the AWX credentials and configurations.

L’article Migrating AWX from one Kubernetes cluster to another: A custom approach est apparu en premier sur dbi Blog.

Ansible loops: A guide from basic to advanced examples

DevOps — Tue, 09 Jul 2024 08:22:30 +0000

If you are writing roles with Ansible, you must already have thought about implementing a loop, a loop of loops with Ansible, and wonder how. The ability to execute tasks in loops is primordial. This guide will provide multiple loop examples in Ansible, starting with a basic loop and progressing to more advanced scenarios.

Basic loop

Let’s start with the most basic loop as an introduction.

In the following playbook, called playbook.yml, a list of numbers is created, from 1 to 5. Then a loop on the debug task displays each number.

# playbook.yml
- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:
  - set_fact:
      numbers: [1,2,3,4,5]
  
  - name: Most basic loop
    debug:
      msg: '{{ item }}'
    loop: '{{ numbers }}'

You can test it by running the command “ansible-playbook playbook.yml”.

$ ansible-playbook playbook.yml 

PLAY [Use vars from dbservers] ****************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [Most basic loop] ************************************************************************************************************************************************************************
ok: [localhost] => (item=1) => {
    "msg": 1
}
ok: [localhost] => (item=2) => {
    "msg": 2
}
ok: [localhost] => (item=3) => {
    "msg": 3
}
ok: [localhost] => (item=4) => {
    "msg": 4
}
ok: [localhost] => (item=5) => {
    "msg": 5
}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Loop of loops

In most use cases, you want to loop multiple tasks sequentially. The first thought is to use a block statement, but a block doesn’t accept a loop. The solution is to use “ansible.builtin.include_tasks” and loop on the task file.

The usage of loop_control is recommended to rename the loop_var name and not use “item”. See below the example with the “playbook.yml” and “loop.yml” files.

# playbook.yml
- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:
  - set_fact:
      numbers: [1,2,3,4,5]

  - name: loop multiple tasks with include_task
    ansible.builtin.include_tasks:
      file: loop.yml
    loop: '{{ numbers }}'
    loop_control:
      loop_var: number

# loop.yml
- debug:
    msg: 'First task of loop.yml'

- debug:
    var: number

The results of running the playbook should be as below.

$ ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [loop multiple tasks with include_task] **************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=5)

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 1
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 2
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 3
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 4
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 5
}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=16   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Conditional on include_tasks

Adding a condition on the include_tasks is only evaluated once. It means that if the condition is True, it will iterate until the end even though the condition might become False, which is the intended result of the condition of include_tasks. See the following example.

# playbook.yml
- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:

  - set_fact:
      numbers: [1,2,3,4,5]

  - set_fact: 
      continue_task: true

  - when: continue_task == true
    name: When on include task only (evaluted at the beginning)
    ansible.builtin.include_tasks:
      file: condition.yml
    loop: '{{ numbers }}'
    loop_control:
      loop_var: number

# condition.yml
- debug:
    var: number

- when: number >= 3
  set_fact:
    continue_task: false

- debug:
    msg: 'current number: {{ number }}. Condition.yml running on number < 3'

Results of the running playbook.

$ ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [When on include task only (evaluted at the beginning)] **********************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=5)

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 1
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 1. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 2
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 2. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 3
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 3. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 4
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 4. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 5
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 5. Condition.yml running on number < 3"
}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=20   changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

If you want the condition to apply on every task, you can either add the statement “when” on every task in “condition.yml”, or you can use the statement “apply” on “ansible.builtin.include_tasks”. Choose the solution that suits the best for the role or tasks you are writing.

The below example will use the statement “apply”.

# playbook.yml

- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:

  - set_fact:
      numbers: [1,2,3,4,5]

  - set_fact: 
      continue_task: true

  - name: When applied on all tasks included (evaluated each time)
    ansible.builtin.include_tasks:
      file: condition.yml
      apply:  
        when: continue_task == true
    loop: '{{ numbers }}'
    loop_control:
      loop_var: number

Output of the playbook. The results should print the number until “3” but not the second debug message “current number 3. Condition.yml running on number < 3”.

$ ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [When applied on all tasks included (evalued each time)] *********************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=5)

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 1
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 1. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 2
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 2. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 3
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=13   changed=0    unreachable=0    failed=0    skipped=9    rescued=0    ignored=0

Advanced loop: do tasks until succeed, or fail at Xth attempt

In this last example, I want to make a loop of tasks. The problem is that multiple tasks may fail for multiple reasons (network, unavailability of external services, awaiting process from external services, etc.). Therefore I want to retry the task at least 5th times, before exiting the playbook run.

The example will include a commented task, to provide a more concrete real use case example. The real example is the following. Fetch the ID of an item from an external service, use this ID to fetch its status to the external service, and only proceed if the item status is completed (successful, completed) or fail the task if it returns a failure state. The task may fail on the fetch of ID (due to network issues or unavailability of the external service) and on the status fetching if it is still ongoing, which will result in retrying the whole task. Note that it is not completely optimized to make it simpler to understand and read (for example we could ignore the fetching of ID if it was already gotten in a previous iteration).

For the simplicity of the setup, a simple condition on “number” is used instead, to showcase the retry of tasks and failure of the play. It causes the playbook’s execution to fail if the number is above 3.

# playbook.yml

- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:

  - set_fact:
      numbers: [1,2,3,4,5]

  - set_fact: 
      continue_task: true


  - name: Example of a complex loop. Loop on number, do function(number) until suceed, or fail at the fifth attempt
    ansible.builtin.include_tasks:
      file: function.yml
    loop: '{{ numbers }}'
    loop_control:
      loop_var: number

# function.yml
- block:
    - when: init_count | default(true)
      ansible.builtin.set_fact:
        retry_count: 0

    - debug:
        msg: 'Current number is {{ number }}, and current retry count is {{ retry_count }}'

    # Do an action, use the result to do another action or checks (for example a wget, curl, or another request to get an ID)
    # For the simplicity of the example, I simply do an echo
    - name: API - get ID
      ansible.builtin.shell: 'echo {{ number }}'
      register: _api_result

    # Use the result from the precedent task
    - name: Use the ID to check another API if process is succesful 
      ## An exemple of a use case, using the id
      # ansible.builtin.uri:
      #   url: 'https://example.com/status?id={{ _api_result.stdout }}'
      #   method: GET
      #   status_code: 200
      # register: _check_status
      # until:
      #   - _check_status.json is defined
      #   - _check_status.json.status in ["SUCCESSFUL", "COMPLETED", "FAILURE"]
      # failed_when: _check_status.json is not defined or _check_status.json.status in ["FAILURE"]
      # delay: '5'
      # retries: '3'

      ## For the simplicity, I just used a failed_when on debug
      debug:
        msg: 'Testing that api result is a > 3'
      failed_when: _api_result.stdout|int > 3

  rescue:
    - when: _check_status.json is defined and _check_status.json.status in ["FAILURE"]
      name: Fail if process return Failure
      ansible.builtin.fail:
        msg: status failure

    - name: Fail Task in case of total failure after a certain amount of retry
      ansible.builtin.fail:
        msg: "5 retries attempted, failed perform desired result"
      when: retry_count | int >= 5 

    # Pause the playbook if necessary.
    # - ansible.builtin.pause:
    #     seconds: '5'

    - name: Increment Retry Count
      ansible.builtin.set_fact:
        retry_count: "{{ retry_count | int + 1 }}"

    # Retry the function.yml and indicate to increment the counter.
    - name: Retry function
      ansible.builtin.include_tasks: function.yml
      vars:
        init_count: false

Result of the execution.

$ ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [When applied on all tasks included (evalued each time)] *********************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=5)

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 1
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 1. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 2
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 2. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 3
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=13   changed=0    unreachable=0    failed=0    skipped=9    rescued=0    ignored=0   

kke@DBI-LT-KKE ansible_loop % ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [Example of a complex loop. Loop on number, do function(number) until suceed, or fail at the fifth attempt] ******************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=5)

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 1, and current retry count is 0"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
ok: [localhost] => {
    "msg": "Testing that api result is a > 3"
}

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 2, and current retry count is 0"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
ok: [localhost] => {
    "msg": "Testing that api result is a > 3"
}

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 3, and current retry count is 0"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
ok: [localhost] => {
    "msg": "Testing that api result is a > 3"
}

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 0"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 1"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 2"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 3"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 4"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 5"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "5 retries attempted, failed perform desired result"}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=42   changed=9    unreachable=0    failed=1    skipped=16   rescued=6    ignored=0

Conclusion

Loop is a necessity in IT, the base for automation. At first, it might be disorientating with Ansible, but it becomes quite easy to understand and use with a little practice. The official documentation provided by Ansible also covers a lot and explains well the concept of loop, it will be your best friend in your journey to master Ansible.

Links

Ansible – Official documentation
Blog – Faster Ansible
Blog – Ansible Automates – Event Driven & Lightspeed
Blog – Specify hosts in ansible-playbook command line
Blog – Ansible Event Driven Automation
Blog – Create and manage Ansible Execution Environments

L’article Ansible loops: A guide from basic to advanced examples est apparu en premier sur dbi Blog.

Using WebHooks with GitLab to launch external tools

Oracle Team — Mon, 27 May 2024 15:02:10 +0000

by Alexandre Nestor

Introduction

WebHooks are HTTP callbacks that are triggered by events. In case of GitlLab these can be events like pushing some code, or creating an issue, or pushing some comments, create a merge request and so on.

Somehow, is a way for an application to send data, trigger events, in real time to another application, even if the target application don’t ask for the data.

In this post I will explain how to add a WebHook to a project and how to execute some external code when en event is triggered.

I will use an Flask application, which will be triggered by GitLab WebHook.

VM configuration

The test VM is created on the OCI cloud. It is a RedHat VM but that is not very important.

First I have to open the port 8200 (used by Flask). RedHat use by default firewalld as firewall:

# add 8200 port using tcp to the public zone
[root@wb ~]# firewall-cmd --zone=public --add-port=8200/tcp
success
# list opened ports
[root@wb ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: ens3
sources:
services: dhcpv6-client http ssh
ports: 8200/tcp
protocols:
forward: no
masquerade: no
forward-ports:
source-ports:
icmp-blocks:
rich rules:

The Flask application

For example I create a small Flask application which is waiting for an event on 8200 port. The easiest application is to retrieve the HTTP frame.

# app.py 
from flask import Flask, request, abort
import json

app = Flask(__name__)

@app.route('/my_webhook', methods=['POST'])
def my_webhook():
    if request.method == 'POST':
        print(json.dumps(request.json, indent=2))
        return 'success', 200
    else:
        abort(400)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8200, debug=True)

The Flask will launch a http server which is listen on all interfaces (0.0.0.0) en on port 8200
The path to be used in the HTTP POST call is my_webhook
Also I used the debug mode to get the most traces.
The Flask application is waiting a POST request (line 10). And answer success.
Otherwise, return 400 HTTP code.

Start the Flask application

(venv) [opc@wb ~]$ python app.py
 * Serving Flask app 'app' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: on
 * Running on all addresses.
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://172.168.0.40:8200/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 295-390-357

GitLab configuration

You must have the role maitainer on your project, to have the rights to manage WebHooks.

Select Settings -> WebHooks from the left Menu

Once The WebHook was saved it can be tested. From Test button you can choose an event type to test.

Let’s try with a push event:

When the event is triggered in GitLab, the Flask application must dump the http frame on the screen:

 * Running on http://172.168.0.40:8200/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 295-390-357
.....
{
  "object_kind": "push",   
  "event_name": "push",
  "before": "6aa30335b09*****53183fb28bb84aada5cf",
  "after": "a48e187ef217a******62498378e5023d355",
  "ref": "refs/heads/main",
  "checkout_sha": "a48e187ef21********9490562498378e5023d355",
  "message": null,
  "user_id": 11774125,
  "user_name": "your username ",
  "user_username": "your.usernam",
  "user_email": "",
  "user_avatar": "https://gitlab.com/uploads/-/system/user/avatar/11774125/avatar.png",
  "project_id": 38878393,
  "project": {
    "id": 38878393,
    .....

On line 7 we can see the push event from GitLab.

Conclusion

The use of WebHooks from GitLab to run external code represents another facility in DevOps tools.

Their implementation is easy and immediate.

An example of use case is the integration with Ansible Automation Controller, to run Ansible jobs.

L’article Using WebHooks with GitLab to launch external tools est apparu en premier sur dbi Blog.

Faster Ansible

Middleware Team — Mon, 08 Apr 2024 15:03:21 +0000

Even if Ansible is powerful and flexible, it can be considered “slow”. It will be anyway faster, and more consistent, than doing the same steps manually. Nevertheless, we will experiment to make it even faster. I found few of them on the Internet, but rarely with figures of what to expect.

In this blog post, I will cover one of them and run different scenarios. We will also dig inside some internal mechanism used by Ansible.

SSH Connections

Ansible is connection intensive as it opens, and closes, many ssh connections to the targeted hosts.

I found two possible ways to count the amount of connections from control to agents nodes:

Add -vvv option to the ansible-playbook command.
grep audit.log file:

tail -f /var/log/audit/audit | grep USER_LOGIN

First option is really too much verbose, but I used it with the first playbook below to confirm the second option give the same count.

Simple Playbook

To demonstrate that, let’s start with a very minimal playbook without fact gathering:

---
- name: Test playbook
  hosts: all
  gather_facts: false
  pre_tasks:
    - name: "ping"
      ansible.builtin.ping:
...

This playbook triggered 8 ssh connections to the target host. If I enable facts gathering, count goes to 14 connections. Again, this is quiet a lot knowing that playbook does not do much beside check target is alive.

To summarize:

gather_facts	connections	timing (s)
false	8	1,277
true	14	1,991

ping playbook results

What are All These Connection For?

To determine what are these connections doing, we can analyze verbose (really verbose!!) output of Ansible playbook without fact gathering.

First Connection

First command of first connection is echo ~opc && sleep 0 which will return the home directory of ansible user.

Second

Second command is already scary:

( umask 77 && mkdir -p "` echo /home/opc/.ansible/tmp `"&& mkdir "` echo /home/opc/.ansible/tmp/ansible-tmp-1712570792.3350322-23674-205520350912482 `" && echo ansible-tmp-1712570792.3350322-23674-205520350912482="` echo /home/opc/.ansible/tmp/ansible-tmp-1712570792.3350322-23674-205520350912482 `" ) && sleep 0

It set the umask for the commands to follow.
Create tmp directory to store any python script on target
In that directory, create a directory to store the script for this specific task
Makes this ssh command return the temporary variable with full path to the task script directory
sleep 0

This one is mainly to ensure directory structure exists on the target.

Third

I will not paste this one here as it is very long and we can easily guess what it does with log just before:

Attempting python interpreter discovery

Roughly, what it does, it tries many versions of python.

Fourth

Next, it will run a python script with discovered python version to determine Operating System type and release.

Fifth

Fifth connection is actually a sftp command to copy module content (AnsiballZ_ping.py). AnsiballZ is a framework to embed module into script itself. This allows to be run modules with a single Python copy.

Seventh

This one is simply ensuring execution permission is set on temporary directory (ie. ansible-tmp-1712570792.3350322-23674-205520350912482) as well the python script (ie. AnsiballZ_ping.py).

Eighth and Last Connection

Lastly, the execution of the ping module itself:

/usr/bin/python3.9 /home/opc/.ansible/tmp/ansible-tmp-1712570792.3350322-23674-205520350912482/AnsiballZ_ping.py && sleep 0

Optimization

To reduce the amount of connection, there is one possible option: Pipelining

To enable that, I simply need to add following line in ansible.cfg:

pipelining = true

Or set ANSIBLE_PIPELINING environment variable to true.

How does it improve our playbook execution time:

gather_facts	connections	timing (s)
false	3 (-62%)	0,473 (-63%)
true	4 (-71%)	1,275 (-36%)

ping playbook results with pipelining

As we can see there is a significant reduction on the amount of ssh connections as well as a reduction of the playbook duration.

In this configuration, only 3 connections are made:

python interpreter discovery (connection #3)
OS type discovery (connection #4)
python module execution (connection #8). AnsiballZ data is piped to that process.

With the pipelining option, I also noticed that the Ansible temporary directory is not created.

Of course, we can’t expect such big speed-up on a real life playbook. So, we should do it now.

Deploy WebLogic Server Playbook

Let’s use the WebLogic YaK component to deploy a single WebLogic instance. It includes dbi service best practices, latest CPU patches and SSL configuration. The “normal” run takes 13 minutes 30 seconds when the pipelined run takes 12 minutes 13 seconds. This is 10% faster.

This is nice, but not as good as previous playbook. Why is that? Because most of the time is not spent in ssh connections, but with actual work (running WebLogic installer, starting services, patching with OPatch, etc).

What Next?

With such results, you might wonder why isn’t it enabled by default? As per documentation, there is a limitation:

This can conflict with privilege escalation (become). For example, when using sudo operations you must first disable ‘requiretty’ in the sudoers file for the target hosts, which is why this feature is disabled by default.
Ansible documentation

Until now, with all tests I have made, I never encountered that limitation. Did you?

L’article Faster Ansible est apparu en premier sur dbi Blog.