DevOps, auteur/autrice sur dbi Blog

Ansible loops: A guide from basic to advanced examples

DevOps — Tue, 09 Jul 2024 08:22:30 +0000

If you are writing roles with Ansible, you must already have thought about implementing a loop, a loop of loops with Ansible, and wonder how. The ability to execute tasks in loops is primordial. This guide will provide multiple loop examples in Ansible, starting with a basic loop and progressing to more advanced scenarios.

Basic loop

Let’s start with the most basic loop as an introduction.

In the following playbook, called playbook.yml, a list of numbers is created, from 1 to 5. Then a loop on the debug task displays each number.

# playbook.yml
- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:
  - set_fact:
      numbers: [1,2,3,4,5]
  
  - name: Most basic loop
    debug:
      msg: '{{ item }}'
    loop: '{{ numbers }}'

You can test it by running the command “ansible-playbook playbook.yml”.

$ ansible-playbook playbook.yml 

PLAY [Use vars from dbservers] ****************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [Most basic loop] ************************************************************************************************************************************************************************
ok: [localhost] => (item=1) => {
    "msg": 1
}
ok: [localhost] => (item=2) => {
    "msg": 2
}
ok: [localhost] => (item=3) => {
    "msg": 3
}
ok: [localhost] => (item=4) => {
    "msg": 4
}
ok: [localhost] => (item=5) => {
    "msg": 5
}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Loop of loops

In most use cases, you want to loop multiple tasks sequentially. The first thought is to use a block statement, but a block doesn’t accept a loop. The solution is to use “ansible.builtin.include_tasks” and loop on the task file.

The usage of loop_control is recommended to rename the loop_var name and not use “item”. See below the example with the “playbook.yml” and “loop.yml” files.

# playbook.yml
- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:
  - set_fact:
      numbers: [1,2,3,4,5]

  - name: loop multiple tasks with include_task
    ansible.builtin.include_tasks:
      file: loop.yml
    loop: '{{ numbers }}'
    loop_control:
      loop_var: number

# loop.yml
- debug:
    msg: 'First task of loop.yml'

- debug:
    var: number

The results of running the playbook should be as below.

$ ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [loop multiple tasks with include_task] **************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/loop.yml for localhost => (item=5)

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 1
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 2
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 3
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 4
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "First task of loop.yml"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 5
}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=16   changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Conditional on include_tasks

Adding a condition on the include_tasks is only evaluated once. It means that if the condition is True, it will iterate until the end even though the condition might become False, which is the intended result of the condition of include_tasks. See the following example.

# playbook.yml
- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:

  - set_fact:
      numbers: [1,2,3,4,5]

  - set_fact: 
      continue_task: true

  - when: continue_task == true
    name: When on include task only (evaluted at the beginning)
    ansible.builtin.include_tasks:
      file: condition.yml
    loop: '{{ numbers }}'
    loop_control:
      loop_var: number

# condition.yml
- debug:
    var: number

- when: number >= 3
  set_fact:
    continue_task: false

- debug:
    msg: 'current number: {{ number }}. Condition.yml running on number < 3'

Results of the running playbook.

$ ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [When on include task only (evaluted at the beginning)] **********************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=5)

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 1
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 1. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 2
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 2. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 3
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 3. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 4
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 4. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 5
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 5. Condition.yml running on number < 3"
}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=20   changed=0    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

If you want the condition to apply on every task, you can either add the statement “when” on every task in “condition.yml”, or you can use the statement “apply” on “ansible.builtin.include_tasks”. Choose the solution that suits the best for the role or tasks you are writing.

The below example will use the statement “apply”.

# playbook.yml

- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:

  - set_fact:
      numbers: [1,2,3,4,5]

  - set_fact: 
      continue_task: true

  - name: When applied on all tasks included (evaluated each time)
    ansible.builtin.include_tasks:
      file: condition.yml
      apply:  
        when: continue_task == true
    loop: '{{ numbers }}'
    loop_control:
      loop_var: number

Output of the playbook. The results should print the number until “3” but not the second debug message “current number 3. Condition.yml running on number < 3”.

$ ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [When applied on all tasks included (evalued each time)] *********************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=5)

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 1
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 1. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 2
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 2. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 3
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=13   changed=0    unreachable=0    failed=0    skipped=9    rescued=0    ignored=0

Advanced loop: do tasks until succeed, or fail at Xth attempt

In this last example, I want to make a loop of tasks. The problem is that multiple tasks may fail for multiple reasons (network, unavailability of external services, awaiting process from external services, etc.). Therefore I want to retry the task at least 5th times, before exiting the playbook run.

The example will include a commented task, to provide a more concrete real use case example. The real example is the following. Fetch the ID of an item from an external service, use this ID to fetch its status to the external service, and only proceed if the item status is completed (successful, completed) or fail the task if it returns a failure state. The task may fail on the fetch of ID (due to network issues or unavailability of the external service) and on the status fetching if it is still ongoing, which will result in retrying the whole task. Note that it is not completely optimized to make it simpler to understand and read (for example we could ignore the fetching of ID if it was already gotten in a previous iteration).

For the simplicity of the setup, a simple condition on “number” is used instead, to showcase the retry of tasks and failure of the play. It causes the playbook’s execution to fail if the number is above 3.

# playbook.yml

- name: Loop examples
  hosts: localhost
  connection: local
  gather_facts: False
  tasks:

  - set_fact:
      numbers: [1,2,3,4,5]

  - set_fact: 
      continue_task: true


  - name: Example of a complex loop. Loop on number, do function(number) until suceed, or fail at the fifth attempt
    ansible.builtin.include_tasks:
      file: function.yml
    loop: '{{ numbers }}'
    loop_control:
      loop_var: number

# function.yml
- block:
    - when: init_count | default(true)
      ansible.builtin.set_fact:
        retry_count: 0

    - debug:
        msg: 'Current number is {{ number }}, and current retry count is {{ retry_count }}'

    # Do an action, use the result to do another action or checks (for example a wget, curl, or another request to get an ID)
    # For the simplicity of the example, I simply do an echo
    - name: API - get ID
      ansible.builtin.shell: 'echo {{ number }}'
      register: _api_result

    # Use the result from the precedent task
    - name: Use the ID to check another API if process is succesful 
      ## An exemple of a use case, using the id
      # ansible.builtin.uri:
      #   url: 'https://example.com/status?id={{ _api_result.stdout }}'
      #   method: GET
      #   status_code: 200
      # register: _check_status
      # until:
      #   - _check_status.json is defined
      #   - _check_status.json.status in ["SUCCESSFUL", "COMPLETED", "FAILURE"]
      # failed_when: _check_status.json is not defined or _check_status.json.status in ["FAILURE"]
      # delay: '5'
      # retries: '3'

      ## For the simplicity, I just used a failed_when on debug
      debug:
        msg: 'Testing that api result is a > 3'
      failed_when: _api_result.stdout|int > 3

  rescue:
    - when: _check_status.json is defined and _check_status.json.status in ["FAILURE"]
      name: Fail if process return Failure
      ansible.builtin.fail:
        msg: status failure

    - name: Fail Task in case of total failure after a certain amount of retry
      ansible.builtin.fail:
        msg: "5 retries attempted, failed perform desired result"
      when: retry_count | int >= 5 

    # Pause the playbook if necessary.
    # - ansible.builtin.pause:
    #     seconds: '5'

    - name: Increment Retry Count
      ansible.builtin.set_fact:
        retry_count: "{{ retry_count | int + 1 }}"

    # Retry the function.yml and indicate to increment the counter.
    - name: Retry function
      ansible.builtin.include_tasks: function.yml
      vars:
        init_count: false

Result of the execution.

$ ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [When applied on all tasks included (evalued each time)] *********************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/condition.yml for localhost => (item=5)

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 1
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 1. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 2
}

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "current number: 2. Condition.yml running on number < 3"
}

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "number": 3
}

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
skipping: [localhost]

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=13   changed=0    unreachable=0    failed=0    skipped=9    rescued=0    ignored=0   

kke@DBI-LT-KKE ansible_loop % ansible-playbook playbook.yml

PLAY [Loop examples] **************************************************************************************************************************************************************************

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [set_fact] *******************************************************************************************************************************************************************************
ok: [localhost]

TASK [Example of a complex loop. Loop on number, do function(number) until suceed, or fail at the fifth attempt] ******************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=1)
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=2)
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=3)
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=4)
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost => (item=5)

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 1, and current retry count is 0"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
ok: [localhost] => {
    "msg": "Testing that api result is a > 3"
}

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 2, and current retry count is 0"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
ok: [localhost] => {
    "msg": "Testing that api result is a > 3"
}

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 3, and current retry count is 0"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
ok: [localhost] => {
    "msg": "Testing that api result is a > 3"
}

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
ok: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 0"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 1"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 2"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 3"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 4"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
skipping: [localhost]

TASK [Increment Retry Count] ******************************************************************************************************************************************************************
ok: [localhost]

TASK [Retry function] *************************************************************************************************************************************************************************
included: /Users/kke/dbi/blog/ansible_loop/function.yml for localhost

TASK [ansible.builtin.set_fact] ***************************************************************************************************************************************************************
skipping: [localhost]

TASK [debug] **********************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "Current number is 4, and current retry count is 5"
}

TASK [API - get ID] ***************************************************************************************************************************************************************************
changed: [localhost]

TASK [Use the ID to check another API if process is succesful] ********************************************************************************************************************************
fatal: [localhost]: FAILED! => {
    "msg": "Testing that api result is a > 3"
}

TASK [Fail if process return Failure] *********************************************************************************************************************************************************
skipping: [localhost]

TASK [Fail Task in case of total failure after a certain amount of retry] *********************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "5 retries attempted, failed perform desired result"}

PLAY RECAP ************************************************************************************************************************************************************************************
localhost                  : ok=42   changed=9    unreachable=0    failed=1    skipped=16   rescued=6    ignored=0

Conclusion

Loop is a necessity in IT, the base for automation. At first, it might be disorientating with Ansible, but it becomes quite easy to understand and use with a little practice. The official documentation provided by Ansible also covers a lot and explains well the concept of loop, it will be your best friend in your journey to master Ansible.

Links

Ansible – Official documentation
Blog – Faster Ansible
Blog – Ansible Automates – Event Driven & Lightspeed
Blog – Specify hosts in ansible-playbook command line
Blog – Ansible Event Driven Automation
Blog – Create and manage Ansible Execution Environments

L’article Ansible loops: A guide from basic to advanced examples est apparu en premier sur dbi Blog.

My first day at SUSECON 2024

DevOps — Tue, 18 Jun 2024 06:00:00 +0000

It is mid-June, and I have the opportunity with my colleague Arnaud Berbier to go to Berlin. Not for visiting the city, not for sightseeing, but for a business conference. This year, SUSECON set its location in the German capital, and as a partner, it was a kind of must for dbi services to be next to SUSE.

SUSECON, the annual global conference organized by SUSE, is a convention for IT professionals like us to explore the latest developments in open-source software and SUSE technologies. This event features keynotes, technical sessions, and hands-on labs, providing a platform for learning, networking, and collaboration.

Just the time to arrive in front of the Estrel building, we were warmly welcomed by Nico Lamberti, Partner Executive ALPS at SUSE. A quick photo, and we headed inside the building. Then we met Emiel Brok. A quick chat, and he already pointed the mic to me, and we started a short interview. I’m quite shy (so I won’t put any link to it .. sorry Emiel ) and not used to such an improvised kind of speaking, but I knew I had to push through my hesitation for this important event!

The registration desk then, and all was well organized. Step by step, we registered, changed to the following desk to get our badge, and another desk to get our swag bag. Finally, we were fully prepared for the event.

The partner summit was going to happen at 2 p.m. So we had time to attend a session.

“How YOU can contribute to high-quality documentation”

This was the first session I attended. Led by Meike Chabowski, I was interested in how SUSE was driving their documentation. Which kind of technology, which kind of organization,

She highlighted first the goal of documentation (at least at SUSE): to produce and deliver high quality documentation. What for? To provide added value, to drive custom, partner, and community (yes, open-source philosophy is there). And then she enumerated first the different sort of documentation at SUSE:

Release notes, for a brief description of new technologies and features,
Technical Reference Documentation, to describe and explain design, setup and configuration of SUSE products,
SUSE Best Practices, practical, issue-focused, and solution-focused documentation,
Product documentation, aka Technical documentation for all products,
Topic based article.

I was really curious how SUSE manage their documentation. At dbi services, in the DevOps team, and even for personal documentation or notes based system, we use Markdown.

At SUSE, they use DocBookXML and AsciiDoc. While the first is a derivative of XML for documentation purpose, I see the second as very close to Markdown at first glance. They use XSLT stylesheets for defining the layout.

In term of infrastructure, documentation requests are managed through Jira or Bugzilla, and everything is versioned in git.

What I liked is how she highlighted contribution. We evolve in an open-source environment, and contributing is one key aspect. Working on the documentation is one way to contribute, and it’s a great way to give back to the community. This can be done quite easily. You can either report an issue in the SUSE Best Practices or Technical Reference Documentation, or propose your change to the Product documentation. Once reviewed, and maybe approved, your changes will be included.

Exploring the Technology showcase

In between sessions and talks, I spent time in the technology showcase, which was a hub of activity. Booths from various exhibitors showcased cutting-edge technologies, solutions, and products. I had the chance to speak with some exhibitors, had for instance a great detailed description of what is Confidential Computing (encryption EVERYWHERE, even the memory is encrypted on the fly!). This place was also a good opportunity to meet people, whether you knew them or not.

This morning.

Tonight

L’article My first day at SUSECON 2024 est apparu en premier sur dbi Blog.

Investigative look into cloud-native hyper-converged infrastructure with Harvester (2)

DevOps — Thu, 30 May 2024 06:01:04 +0000

In the previous episode, we took a look together at an overview of Harvester. We looked at its components and explored the concepts behind Harvester, its cloud-native architecture, and the installation process. Now, let’s delve deeper into the main purpose of it: scheduling your workload. Let’s start!

Note: we expanded the Harvester cluster to three nodes. I would like to play with a bit of scheduling, so let’s add two worker nodes.

Virtual machine provisioning

Practical experience is the most effective method to grasp a concept, process, or product. I suggest that we proceed in the same manner.

As part of a larger consolidation and modernization project, imagine we just received the request to integrate a Microsoft Windows Server 2022 application into our new infrastructure. If you ever wonder how to control the quality of your data, there is, at dbi services, OMrun. OMrun is a Windows based application that includes test steps, scenarios, and of course, data adaptors. Let’s take this application as a workload example.

We are not going to automate anything here, as exploring the application is the main focus, so expect provisioning the virtual machine in a wizard based manner.

Log into Harvester web interface by entering the IP mentioned in the console of your node. Take the cluster URL, not the node’s IP.

In order to provision and run a Windows based application, you need first to add the ISO file of the operating system. I’ve already downloaded the file (using the following URL on the Microsoft web site).

For this, left-hand menu, select Images and then Create.

Upload the file and select Create.

Once uploaded, any virtual machine using the infrastructure can use the file as of right now.

It is now time to create the VM. On the left-hand side, select “Virtual Machines”. You’ll be presented with a set of fields and options. First of all, as we are dealing here with a Windows-based VM, tick the box “Use VM Template:” and select the option “harvester-public/windows-iso-image-base-template”. As stated by SUSE in their documention, it is used to add a volume, a disk so, where all the necessary optimised drivers will be stored, so you can refer to that when installing the operating system. If you already used KVM for virtualization, you are on familiar ground. We are talking about paravirtualized VirtIO drivers.

We start to see some Kubernetes references: a namespace is required in order to group the corresponding Kubernetes objects. Reminder: The Kubernetes backend will be in charge of managing your VM. For that, a new set of CRDs (CustomResourceDefinition) were added to handle that.

For instance, here is the definition of a VM: VirtualMachineInstance.
We create a new namespace; let’s call it omrun. We also set the name, as well as a short description. Based on your application requirements, specify the CPU and memory resources.

Here, I’m not going to implement a production ready instance of OMrun, so I won’t be greedy on that: 4 CPUs and 8GB of memory.

Move on to the storage. By selecting the “Volumes” item, we will be able to setup all disks and virtual CD-ROM. Since we use a VM template, we can see that Harvester already provisioned for us some storage devices:

a cdrom-disk: obviously used for acting as virtual CD-ROM drive,
Root disk: the main disk drive of your VM
virtio-container-disk: an ephemeral type of storage, used when you don’t want any kind of persistence of data. For instance, read-only data, configuration files, or binaries, this is used here for the VirtIO drivers.

The next item is “Networks”. This part deserved its own blog; for ease of comprehension, we are going to leave this as default.

A scheduling strategy?

The following two options are showing, for sure, the connection with Kubernetes. The first one, “Node Scheduling” is there to specify the scheduling strategy of the VM across the cluster. Either on any available node, specific named nodes, or Kubernetes labels assigned to nodes. We would like to preferably run the VM only on nodes with the label omrun set to true.

First, we label the node:

harv01:~ $ kubectl label nodes harv02 omrun=true 
node/harv02 labeled

Then, on Harvester interface, your inputs shoud look like this:

The next item is still related to scheduling, “VM Scheduling”. This time, you can specify, for instance, if your workload needs to run on the same node as already existing workloads (for an increase in performance, perhaps ?) or the opposite: you want to ensure that your workload is properly separated from other specific ones.

All this is managed using the same kind of affinity / anti-affinity concepts, dear to Kubernetes.

If you want to know more about that, jump here.

In the following configuration items, in Advanced Options, you’ll be ask to set the behavior of the scheduler, Kubernetes then, on the run strategy of your virtual machine. What would need to happen in case your VM crashed for instance. What should happen if you manually stop your VM. No need to carbon copying the SUSE Harvester documentation.

By the way, this is reminds me the restart policy in the Docker run command, we just discovered less than a decade ago

Once done, you can go forward, and click on the Create button at the bottom right corner.

The spin up of the VM takes a bit of time. Especially on my home lab I’m currently using

But wait.. We are running a Kubernetes cluster here.. Why not going directly on it and trying to see what’s going on? Using our lovely kubectl ?

Well, again it’s quite easy and straight forward. You remember the namespace we created initially ? I’m logged on my Harvester cluster, in SSH.

harv01:~ $ kubectl get vm -n omrun
NAME             AGE   STATUS     READY
omrun-instance   19d   Starting   False
harv01:~ $ kubectl get vmi -n omrun
NAME             AGE   PHASE       IP    NODENAME  
omrun-instance   17s   Scheduled         harv02

The virtual machine is currently being started. The VirtualMachine object has status Starting, the VirtualMachineInstance got the Scheduled status.

State of your VM is reflected directly across Kubernetes objects.

We can add the usual -o yaml flag, as we do for pods:

harv01:~ $ kubectl get vm -n omrun -o wide -o yaml
apiVersion: v1
items:
- apiVersion: kubevirt.io/v1
 kind: VirtualMachine
 metadata:
   annotations:
     field.cattle.io/description: An OMrun instance running on Harvester
     harvesterhci.io/reservedMemory: 256Mi
...
        architecture: amd64
        domain:
          cpu:
            cores: 1
          devices:
            disks:
            - bootOrder: 1
              cdrom:
                bus: sata
              name: cdrom-disk
            - bootOrder: 2
              disk:
                bus: virtio
              name: rootdisk
...

All the configuration you’ve made in the UI, is there!

The virtual machine is running now, and as specified, on the second node. The time now is to install the operating system and then, the application. And for that, as we are dealing with a Windows-based virtual machine, it’s more than ssh-ing into a Linux box.

The first try is to use what is proposed by Harvester. Based on my experience with a previous cluster I’ve build with a KubeVirt stack on it, connecting to it on Harvester is a dream. In your list of virtual machines, select the “console” button, and then “Open in WebVNC”.

Then, what is following is nothing else but a standard Microsoft Windows Server 2022 installation. The only point of attention here would be to select the paravirtualized VirtIO drivers.

With the power of this blog and some magic, I will fast-forward the installation of OMRun and show you the running application.

Strangely, I didn’t manage to get the RDP protocol available. This is probably due to the network settings I’ve left default at the creation. Something to look for in the next part of this blog series.

I haven’t installed the dashboard of OMRun for now, which is accessible from the client through a web browser.

The power of scheduling

What if we mess around with this part? What if we simulate a downtime from one of the nodes in our cluster?

Sadly, this unexpected, unfortunate event could occur on the node that runs, specifically, our Windows Server instance. How will Harvester, KubeVirt, and Kubernetes react?

We can first assign the label omrun=true to our first node.

Then, we simply power off our second node, with the VM running (naughty, I know). And we watch

First, the second node is set unavailable, cordoned using Kubernetes terms.

harv01:~ $ kubectl get nodes
NAME     STATUS     ROLES                       AGE   VERSION
harv01   Ready      control-plane,etcd,master   21d   v1.27.10+rke2r1
harv02   NotReady                         21d   v1.27.10+rke2r1
harv03   Ready                            21d   v1.27.10+rke2r1

At the same time, the virtual machine is set as not ready, but still, scheduled on harv02. On the Kubernetes side, the same.

harv01:~ $ kubectl get vm -n omrun  
NAME             AGE   STATUS    READY  
omrun-instance   20d   Running   False  
harv01:~ $ kubectl get vmi -n omrun  
NAME             AGE     PHASE     IP            NODENAME   READY  
omrun-instance   6m32s   Running   10.52.1.122   harv02     False

Let’s wait for the timeouts do their jobs and wait a bit.

harv01:~ $ kubectl get vm -n omrun
NAME             AGE   STATUS    READY
omrun-instance   20d   Running   True
harv01:~ $ kubectl get vmi -n omrun
NAME             AGE   PHASE     IP           NODENAME   READY
omrun-instance   24m   Running   10.52.0.84   harv01     True

Harvester, Kubernetes and KubeVirt in the background, have rescheduled the virtual machine to the next node that fits the constraint we asked for (label omrun=true) which was the harv01 (the master node was not the best idea I had).

Thanks to the replication on the storage backend, Longhorn had replicas for each PersistentVolumes created (defaulted to 3). That means we didn’t lose any data when the second node failed.

We can try to start the VNC console. And voila!

Our virtual machine is running, OMrun is still able to start.

I will for sure continue my exploration of Harvester, and probably, this time, explore the networking aspects of it!

L’article Investigative look into cloud-native hyper-converged infrastructure with Harvester (2) est apparu en premier sur dbi Blog.

Detect XZ Utils CVE 2024-3094 with Tetragon

DevOps — Wed, 24 Apr 2024 07:21:23 +0000

The recent discovery of the XZ Utils backdoor, classified as CVE 2024-3094, has been now well documented. Detecting it with Tetragon from Isovalent (now part of Cisco) has been explained in this blog post. I also did some research and experimented with this vulnerability. I wondered how we could leverage Tetragon capabilities to detect it before it was known. There are other vulnerabilities out there, so we need to be prepared for the unknown. For this we have to apply a security strategy called Zero Trust. I wrote another blog post on this topic with another example and another tool if you want to have a look. Let’s build an environment on which we can experiment and learn more about it. Follow along!

Setup an environment for CVE 2024-3094

We have learned that this vulnerability needs an x86 architecture to be exploited and that it targets several Linux distribution (source here). I’ve used an Ubuntu 22.04 virtual machine in Azure to setup the environment. To exploit this vulnerability, we’re going to use the GitHub resource here.

This vulnerability is related to the library liblzma.so used by the ssh daemon so let’s switch to the root user and install openssh-server along with other packages we will use later:

azureuser@Ubuntu22:~$ sudo -i

root@Ubuntu22:~# apt-get update && apt-get install -y golang-go curl openssh-server net-tools python3-pip wget vim git file bsdmainutils jq

Let’s use ssh key authentication (as this is how the vulnerable library can be exploited), start the ssh daemon and see which version of the library it uses:

root@Ubuntu22:~# which sshd
/usr/sbin/sshd
root@Ubuntu22:~# sed -E -i 's/^#?PasswordAuthentication .*/PasswordAuthentication no/' /etc/ssh/sshd_config

root@Ubuntu22:~# service ssh status
 * sshd is not running
root@Ubuntu22:~# service ssh start
 * Starting OpenBSD Secure Shell server sshd
root@Ubuntu22:~# service ssh status
 * sshd is running

root@Ubuntu22:~# ldd /usr/sbin/sshd|grep liblzma
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007ae3aac37000)

root@Ubuntu22:~# file /lib/x86_64-linux-gnu/liblzma.so.5
/lib/x86_64-linux-gnu/liblzma.so.5: symbolic link to liblzma.so.5.2.5

Here it uses version 5.2.5, sometimes it uses version 5.4.5 from the tests I did on other distributions. The vulnerable versions are 5.6.0 and 5.6.1. So by default our machine is not vulnerable. To make it so, we need to upgrade this library to one of these vulnerable versions as shown below:

root@Ubuntu22:~# wget https://snapshot.debian.org/archive/debian/20240328T025657Z/pool/main/x/xz-utils/liblzma5_5.6.1-1_amd64.deb

root@Ubuntu22:~# apt-get install --allow-downgrades --yes ./liblzma5_5.6.1-1_amd64.deb

root@Ubuntu22:~# file /lib/x86_64-linux-gnu/liblzma.so.5
/lib/x86_64-linux-gnu/liblzma.so.5: symbolic link to liblzma.so.5.6.1

We are now using the vulnerable library in version 5.6.1. Next we can use the files and xzbot tool from the GitHub project as shown below:

root@Ubuntu22:~# git clone https://github.com/amlweems/xzbot.git
root@Ubuntu22:~# cd xzbot/

To be able to exploit this vulnerability we can’t just use the vulnerable library. In fact the backdoor uses a hardcoded ED448 public key for signature and we don’t have the associated private key. To be able to trigger that backdoor, the author of the tool xzbot replaced them with their own key pair they’ve generated. We then need to replace the vulnerable library with the patched one using these keys as follows:

root@Ubuntu22:~# cp ./assets/liblzma.so.5.6.1.patch /lib/x86_64-linux-gnu/liblzma.so.5.6.1

Now everything is ready to exploit this vulnerability with the xzbot tool. We just need to compile it with the go package we installed at the beginning:

root@Ubuntu22:~# go build

root@Ubuntu22:~# ./xzbot -h
Usage of ./xzbot:
  -addr string
    	ssh server address (default "127.0.0.1:2222")
  -cmd string
    	command to run via system() (default "id > /tmp/.xz")
  -seed string
    	ed448 seed, must match xz backdoor key (default "0")

Detecting the backdoor with Tetragon

Let’s see now how we could use Tetragon to detect something by applying a Zero Trust strategy. At this stage we consider we don’t know anything about this vulnerability and we are using Tetragon as a security tool for our environment. Here we don’t use Kubernetes, we just have a Ubuntu 22.04 host but we can still use Tetragon by running it into a docker container.

We install docker in our machine by following the instructions described here:

root@Ubuntu22:~# sudo apt-get install ca-certificates curl
root@Ubuntu22:~# sudo install -m 0755 -d /etc/apt/keyrings
root@Ubuntu22:~# sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
root@Ubuntu22:~# sudo chmod a+r /etc/apt/keyrings/docker.asc

root@Ubuntu22:~# echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
root@Ubuntu22:~# sudo apt-get update

root@Ubuntu22:~# sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Then we install Tetragon in a docker container by following the instructions here:

root@Ubuntu22:~# docker run --name tetragon --rm -d \
    --pid=host --cgroupns=host --privileged \
    -v /sys/kernel:/sys/kernel \
    quay.io/cilium/tetragon:v1.0.3 \
    /usr/bin/tetragon --export-filename /var/log/tetragon/tetragon.log

Tetragon – Backdoor detection

Now everything is ready and we can trigger the backdoor and see what Tetragon can observe. We open a new shell by using the azureuser. We jump into the Tetragon container and monitor the log file for anything related to ssh as shown below:

azureuser@Ubuntu22:~$ sudo docker exec -it 76dc8c268caa bash
76dc8c268caa:/# tail -f /var/log/tetragon/tetragon.log | grep ssh

In another shell (the one with the root user), we can start the exploit by using the xzbot tool. We execute the command sleep 60 so we can observe in real time what is happening:

root@Ubuntu22:~/xzbot# ./xzbot -addr 127.0.0.1:22 -cmd "sleep 60"

This is an example of a malicious actor connecting through the backdoor to get a shell on our compromised Ubuntu machine. Below is what we can see in our Tetragon shell (the output has been copied and pasted for being parsed with jq to provide a better reading and we’ve kept only the process execution event):

{
  "process_exec": {
    "process": {
      "exec_id": "OjIwNjAyNjc1NDE0MTU2OjE1NDY0MA==",
      "pid": 154640,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/sbin/sshd",
      "arguments": "-D -R",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-23T12:03:08.447280556Z",
      "auid": 4294967295,
      "parent_exec_id": "OjE0MTYwMDAwMDAwOjc0Mg==",
      "tid": 154640
    },
    "parent": {
      "exec_id": "OjE0MTYwMDAwMDAwOjc0Mg==",
      "pid": 742,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/sbin/sshd",
      "flags": "procFS auid rootcwd",
      "start_time": "2024-04-23T06:19:59.931865800Z",
      "auid": 4294967295,
      "parent_exec_id": "OjM4MDAwMDAwMDox",
      "tid": 742
    }
  },
  "time": "2024-04-23T12:03:08.447279856Z"
}
...
{
  "process_exec": {
    "process": {
      "exec_id": "OjIwNjAyOTk4NzY3ODU0OjE1NDY0Mg==",
      "pid": 154642,
      "uid": 0,
      "cwd": "/",
      "binary": "/bin/sh",
      "arguments": "-c \"sleep 60\"",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-23T12:03:08.770634054Z",
      "auid": 4294967295,
      "parent_exec_id": "OjIwNjAyNjc1NDE0MTU2OjE1NDY0MA==",
      "tid": 154642
    },
    "parent": {
      "exec_id": "OjIwNjAyNjc1NDE0MTU2OjE1NDY0MA==",
      "pid": 154640,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/sbin/sshd",
      "arguments": "-D -R",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-23T12:03:08.447280556Z",
      "auid": 4294967295,
      "parent_exec_id": "OjE0MTYwMDAwMDAwOjc0Mg==",
      "tid": 154640
    }
  },
  "time": "2024-04-23T12:03:08.770633854Z"
}

Here we have all the interesting information about the process as well as the link to its parent process. With Tetragon Entreprise we could have a graphical view of these linked processes. As we are using the Community Edition, we can use the ps command instead here to get a more graphical view as shown below:

azureuser@Ubuntu22:~$ ps -ef --forest
root         742       1  0 06:19 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 1 of 10-100 startups
root      154640     742  2 12:03 ?        00:00:00  \_ sshd: root [priv]
sshd      154641  154640  0 12:03 ?        00:00:00      \_ sshd: root [net]
root      154642  154640  0 12:03 ?        00:00:00      \_ sh -c sleep 60
root      154643  154642  0 12:03 ?        00:00:00          \_ sleep 60

The 2 processes highlighted above are those related to the Tetragon output. Let’s now see what Tetragon displays in case of a normal ssh connection.

Tetragon – Normal ssh connection

We first need to setup a pair of keys for the root user (to better compare it with the output above):

root@Ubuntu22:~# ssh-keygen

root@Ubuntu22:~# cat ~/.ssh/id_rsa.pub > ~/.ssh/authorized_keys

root@Ubuntu22:~# ssh root@127.0.0.1
Welcome to Ubuntu 22.04.4 LTS (GNU/Linux 6.5.0-1017-azure x86_64)

For the key generation we use the default folder with no passphase. We see we can connect with the root user to the localhost by using the generated keys. We can then use the same method as above to launch Tetragon and the ps command to capture this ssh connection. Here is what we can see with Tetragon:

{
  "process_exec": {
    "process": {
      "exec_id": "OjU1ODY3OTQ0NTI0ODY6NDc1MDE=",
      "pid": 47501,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/sbin/sshd",
      "arguments": "-D -R",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-23T07:52:52.566318686Z",
      "auid": 4294967295,
      "parent_exec_id": "OjE0MTYwMDAwMDAwOjc0Mg==",
      "tid": 47501
    },
    "parent": {
      "exec_id": "OjE0MTYwMDAwMDAwOjc0Mg==",
      "pid": 742,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/sbin/sshd",
      "flags": "procFS auid rootcwd",
      "start_time": "2024-04-23T06:19:59.931865800Z",
      "auid": 4294967295,
      "parent_exec_id": "OjM4MDAwMDAwMDox",
      "tid": 742
    }
  },
  "time": "2024-04-23T07:52:52.566318386Z"
}

{
  "process_exec": {
    "process": {
      "exec_id": "OjU1ODgxMzk5MjM5NjA6NDc2MDQ=",
      "pid": 47604,
      "uid": 0,
      "cwd": "/root",
      "binary": "/bin/bash",
      "flags": "execve clone",
      "start_time": "2024-04-23T07:52:53.911790360Z",
      "auid": 0,
      "parent_exec_id": "OjU1ODY3OTQ0NTI0ODY6NDc1MDE=",
      "tid": 47604
    },
    "parent": {
      "exec_id": "OjU1ODY3OTQ0NTI0ODY6NDc1MDE=",
      "pid": 47501,
      "uid": 0,
      "cwd": "/",
      "binary": "/usr/sbin/sshd",
      "arguments": "-D -R",
      "flags": "execve rootcwd clone",
      "start_time": "2024-04-23T07:52:52.566318686Z",
      "auid": 4294967295,
      "parent_exec_id": "OjE0MTYwMDAwMDAwOjc0Mg==",
      "tid": 47501
    }
  },
  "time": "2024-04-23T07:52:53.911789660Z"
}

And the output of the corresponding ps command:

azureuser@Ubuntu22:~$ ps -ef --forest
root         742       1  0 06:19 ?        00:00:00 sshd: /usr/sbin/sshd -D [listener] 0 of 10-100 startups
root       45501     742 10 07:49 ?        00:00:00  \_ sshd: root@pts/1
root       47604   45501  0 07:49 pts/1    00:00:00      \_ -bash

You can see there is a difference but it is not easy to spot! In the normal connection it launches a bash under sshd and through the backdoor it is running a command with sh instead.

Wrap up

We have seen how we can leverage Tetragon to observe anything happening on this machine. Even for unknown threats, you get some information but you have to know first how your system is working in very details. You need to have a baseline for each running process on your machine to be able to detect any deviation. That is what we call the Zero Trust strategy and it is the only way to detect such stealthy backdoor.

It may seem tenuous and it is, however that is how Andres Freund discovered it when he noticed ssh was several milliseconds slower than it should. The famous adage says that the devil is in the detail, this backdoor discovery proves that this is especially true when it comes to security.

L’article Detect XZ Utils CVE 2024-3094 with Tetragon est apparu en premier sur dbi Blog.

Investigative look into cloud-native hyper-converged infrastructure with Harvester

DevOps — Fri, 19 Apr 2024 06:42:19 +0000

In mid-February, I had the pleasure of attending with SUSE, the Swiss Cyber Security Days. This two-day event was taking place in Bern, and I gave a 20-minute session on Harvester. Even if I think this was well received, in 20 minutes, you can’t go through every single feature of a product. And I planned it more as a global overview than a deep dive.

This time, I decided to conduct a deeper exploration of Harvester.

HCI, Harvester, what is this all about?

Nowadays, most of our applications run virtualized. From development environments to huge production clusters, virtual machines are everywhere. And to provide better scalability and flexibility, IT infrastructures evolved from traditional legacy datacenters to hyper-converged infrastructures.

What is hyper-converged infrastructure, so? An HCI is a software-defined, fully integrated system that combines compute, storage, networking, and virtualization resources into a single platform. Those are big boxes that concentrate everything here in one place. You define your workload, you setup your storage, and you design your network all by yourself.

Harvester is an open-source HCI solution that provides cloud-native technologies for managing and orchestrating virtual machines, along with storage and networks. The best is yet to come. Harvester, under the hood, is based on famous and well-known cloud-native technologies. We will discover that while playing with this platform.

Nevertheless, let’s not waste time talking about the theory. Let’s get to work!

Installing Harvester

Let’s start with the installation of Harvester. It’s, in fact, a pretty straight forward process.

ISO is provided and available on GitHub, the version 1.3.0 is the latest stable version as of this writing. Paying attention to the GitHub release page, there are regularly new versions or updates, like RCs. You can try those, but, well, it is release candidates

SUSE and the corresponding web page are pretty clear on the hardware requirements.

Remember, here we are dealing with a critical piece of software that is going to handle all of your workload. It is obvious, but there’s no point, except maybe for very basic testing, running Harvester as a VM on your personal computer. And also, don’t be shy about the specifications.

In summary:

CPU: requires hardware virtualization. All servers have those CPU extensions activated. But in order to cross check, just try to execute and trigger this command on a Linux shell prompt. In terms of resources, 8 cores are marked as a minimum, with 16 cores recommended.
Memory: 32GB is the minimum. But as always, the more, the better.
Local storage of the node requires a minimum of 250GB. It can be either one single disk or spread across multiple disks

The network stack? I would treat that, in fact, as separate, as I think it would deserve, regarding the architecture, its own blog post. Only remember that the server would obviously require network connectivity. Please note the set of network ports that you would need to open for incoming traffic.

Installation for real

As mentioned previously, the installation stack is and ISO file, which is 5.79GB. Once you boot your machine straight from it, you’ll be welcomed by … a GRUB, a quite usual welcome page for Linux enthusiasts. Be aware: mouse not required, no graphical interface here (which is nice: the KISS principle).

Once validated, you need to follow the installation process. It will ask you for the node hostname, standard network settings (IP, DNS, Proxy, etc.), VIP, manual token (useful when you add additional Harvester nodes).

One word about the VIP. This IP is different from the one you have assigned to the node. The VIP is kind of assigned to the Harvester cluster you are going to create. This is the one you’ll enter when you want to target the cluster for the Web UI of Harvester.

Talking about cluster, based on my own testing, Harvester could potentially work in single-node mode, but more as a cluster. The first node you will install will always be a management node. Starting at the fourth node, the first three ones are management nodes, remaining ones will be worker nodes.

I hadn’t talked until now about the core architecture of Harvester, and it was on purpose: the best is to discover it now

Once installed, you end up with a welcome screen with a summary of the management cluster URL and status, the current node name, IP, and status as well.

Press F12, fill up the password you’ve entered during the installation process, and you’ll reach a prompt. Warning: except if I missed a setting, the shell prompt that is presented is using the QWERTY keyboard layout. Tips found in the FAQ, you can SSH using the admin user name (password authentication for a fresh installed node).

jpc@dbi-lt-jpc:~> ssh rancher@192.168.20.121
(rancher@192.168.20.121) Password:  
Last login: Thu Apr 18 11:57:11 2024 from 192.168.20.211
rancher@harv-01:~> sudo su -
harv-01:~ #

Enter a very common command for any DevOps & Kubernetes addict:

harv-01:~ $ kubectl get nodes -o wide
NAME      STATUS   ROLES                       AGE     VERSION           INTERNAL-IP      EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION                  CONTAINER-RUNTIME
harv-01   Ready    control-plane,etcd,master   4h40m   v1.27.10+rke2r1   192.168.20.121           Harvester v1.3.0   5.14.21-150400.24.108-default   containerd://1.7.11-k3s2

If you already got some interest into Harvester, you expected that. As said in the introduction, Harvester is based on cloud native components and Kubernetes is at its core.

“To Host and Serve”

But Kubernetes is not the only one. Remember: Harvester is a HCI, and it is intended to carry out corporate workloads. And in a cloud-native way

As always, on a Kubernetes cluster, looking at namespaces and pods gives a good overview of what is going on here.

Let’s do this!

harv-01:~ $ kubectl get ns
NAME                                     STATUS   AGE
cattle-dashboards                        Active   5h27m
cattle-fleet-clusters-system             Active   5h27m
cattle-fleet-local-system                Active   5h27m
cattle-fleet-system                      Active   5h28m
cattle-logging-system                    Active   5h27m
cattle-monitoring-system                 Active   5h27m
cattle-provisioning-capi-system          Active   5h27m
cattle-system                            Active   5h28m
cluster-fleet-local-local-1a3d67d0a899   Active   5h27m
default                                  Active   5h29m
fleet-local                              Active   5h28m
harvester-public                         Active   5h26m
harvester-system                         Active   5h26m
kube-node-lease                          Active   5h29m
kube-public                              Active   5h29m
kube-system                              Active   5h29m
local                                    Active   5h28m
longhorn-system                          Active   5h26m

Let’s start with the harvester-system namespace.

harv-01:~ $ kubectl get pods -n harvester-system 
NAME                                                    READY   STATUS    RESTARTS   AGE 
harvester-667fb9cbc8-jwmtx                              1/1     Running   0          5h47m 
harvester-load-balancer-6755cb4d67-7xjw8                1/1     Running   0          5h47m 
harvester-load-balancer-webhook-6b5c6b546-bdf66         1/1     Running   0          5h47m 
harvester-network-controller-manager-5ff644ffb6-66s69   1/1     Running   0          5h47m 
harvester-network-controller-v9lhf                      1/1     Running   0          5h47m 
harvester-network-webhook-5c596bdd6c-grctr              1/1     Running   0          5h47m 
harvester-node-disk-manager-gdfzx                       1/1     Running   0          5h47m 
harvester-node-manager-mrqbs                            1/1     Running   0          5h47m 
harvester-node-manager-webhook-9cfccc84c-hxnbm          1/1     Running   0          5h47m 
harvester-webhook-79f5446494-65lvb                      1/1     Running   0          5h47m 
kube-vip-zk62c                                          1/1     Running   0          5h47m 
virt-api-77cbf85485-spnf8                               1/1     Running   0          5h46m 
virt-controller-659ccbfbcd-6xzmb                        1/1     Running   0          5h46m 
virt-controller-659ccbfbcd-tkk2b                        1/1     Running   0          5h46m 
virt-handler-9bwd5                                      1/1     Running   0          5h46m 
virt-operator-6b8b9b7578-66tps                          1/1     Running   0          5h47m

Pods virt-api, virt-controller, virt-operator. For those who know me, they know my addiction to KubeVirt. This wonderful add-on brings the management of virtual machines to Kubernetes and relies on KVM (Kerned-based Virtual Machine, released in 2007).

Add-on to Kubernetes, that also means new objects.

harv-01:~ $ kubectl api-resources |grep "kubevirt.io/v1" 
virtualmachineclones                         vmclone,vmclones                                          clone.kubevirt.io/v1alpha1                   true         VirtualMachineClone 
virtualmachineexports                        vmexport,vmexports                                        export.kubevirt.io/v1alpha1                  true         VirtualMachineExport 
virtualmachineclusterinstancetypes           vmclusterinstancetype,vmclusterinstancetypes,vmcf,vmcfs   instancetype.kubevirt.io/v1beta1             false        VirtualMachineClusterInstancetype 
virtualmachineclusterpreferences             vmcp,vmcps                                                instancetype.kubevirt.io/v1beta1             false        VirtualMachineClusterPreference 
virtualmachineinstancetypes                  vminstancetype,vminstancetypes,vmf,vmfs                   instancetype.kubevirt.io/v1beta1             true         VirtualMachineInstancetype 
virtualmachinepreferences                    vmpref,vmprefs,vmp,vmps                                   instancetype.kubevirt.io/v1beta1             true         VirtualMachinePreference 
kubevirts                                    kv,kvs                                                    kubevirt.io/v1                               true         KubeVirt 
virtualmachineinstancemigrations             vmim,vmims                                                kubevirt.io/v1                               true         VirtualMachineInstanceMigration 
virtualmachineinstancepresets                vmipreset,vmipresets                                      kubevirt.io/v1                               true         VirtualMachineInstancePreset 
virtualmachineinstancereplicasets            vmirs,vmirss                                              kubevirt.io/v1                               true         VirtualMachineInstanceReplicaSet 
virtualmachineinstances                      vmi,vmis                                                  kubevirt.io/v1                               true         VirtualMachineInstance 
virtualmachines                              vm,vms                                                    kubevirt.io/v1                               true         VirtualMachine 
migrationpolicies                                                                                      migrations.kubevirt.io/v1alpha1              false        MigrationPolicy 
virtualmachinepools                          vmpool,vmpools                                            pool.kubevirt.io/v1alpha1                    true         VirtualMachinePool 
virtualmachinerestores                       vmrestore,vmrestores                                      snapshot.kubevirt.io/v1alpha1                true         VirtualMachineRestore 
virtualmachinesnapshotcontents               vmsnapshotcontent,vmsnapshotcontents                      snapshot.kubevirt.io/v1alpha1                true         VirtualMachineSnapshotContent 
virtualmachinesnapshots                      vmsnapshot,vmsnapshots                                    snapshot.kubevirt.io/v1alpha1                true         VirtualMachineSnapshot

With KubeVirt, and so Harvester, a virtual machine will be defined as a Kubernetes object, VirtualMachine. VirtualMachineInstances will be a running instance of a VirtualMachine object. Just imagine, being able to apply VirtualMachine definitions and get your VM scheduled, by Kubernetes standard scheduler somewhere on your Harvester cluster . The best of two worlds is brought together. Harvester simplifies all this using, like Rancher is doing by easing the management of standard Kubernetes workloads.

Quickly continue to get a taste of what is running also on Harvester. We can see some cattle- namespace. Rancher is running here!

harv-01:~ $ kubectl get pods -n cattle-system
NAME                                         READY   STATUS    RESTARTS        AGE
harvester-cluster-repo-5c75f7d9fd-88g4x      1/1     Running   0               5h31m
rancher-5dbd4cf7dc-995rr                     1/1     Running   0               5h29m
rancher-webhook-5788f655d8-4p8bt             1/1     Running   0               5h31m
system-upgrade-controller-78cfb99bb7-hdslt   1/1     Running   2 (5h14m ago)   5h31m

What else do we have? Longhorn, for the storage.

harv-01:~ $ kubectl get pods -n longhorn-system
NAME                                                READY   STATUS    RESTARTS   AGE
csi-attacher-dc76666dd-jhs8l                        1/1     Running   0          5h32m
csi-attacher-dc76666dd-t96tr                        1/1     Running   0          5h32m
csi-attacher-dc76666dd-w9zwf                        1/1     Running   0          5h32m
csi-provisioner-7fc9d85c66-48n7g                    1/1     Running   0          5h32m
csi-provisioner-7fc9d85c66-89h4d                    1/1     Running   0          5h32m
csi-provisioner-7fc9d85c66-c94hb                    1/1     Running   0          5h32m
csi-resizer-67664c5755-jpl6r                        1/1     Running   0          5h32m
csi-resizer-67664c5755-r4llm                        1/1     Running   0          5h32m
csi-resizer-67664c5755-vcn94                        1/1     Running   0          5h32m
csi-snapshotter-6c9d675d9c-flkf6                    1/1     Running   0          5h32m
csi-snapshotter-6c9d675d9c-hw4mj                    1/1     Running   0          5h32m
csi-snapshotter-6c9d675d9c-sppkx                    1/1     Running   0          5h32m
engine-image-ei-acb7590c-cxtvm                      1/1     Running   0          5h32m
instance-manager-3c8c76df5f2bfa171e62a198a9ade00e   1/1     Running   0          5h32m
longhorn-csi-plugin-t7x2h                           3/3     Running   0          5h32m
longhorn-driver-deployer-67fd98774c-xd7x2           1/1     Running   0          5h32m
longhorn-loop-device-cleaner-vq2nr                  1/1     Running   0          5h32m
longhorn-manager-mn75c                              1/1     Running   0          5h32m
longhorn-ui-7f8cdfcc48-gqcs4                        1/1     Running   0          5h32m
longhorn-ui-7f8cdfcc48-v9nhc                        1/1     Running   0          5h32m

You remember the cattle-monitoring-system namespace? We didn’t talked also about the based OS, Elemental. We will go deeper into all those topics, as they are pretty dense, lots of subjects here. But by using Kubernetes for his HCI, SUSE simplifies the management of workloads for companies and reduces the time required for their delivery.

Stay connected for much more!

L’article Investigative look into cloud-native hyper-converged infrastructure with Harvester est apparu en premier sur dbi Blog.

Rancher RKE2: Rancher roles for cluster autoscaler

DevOps — Wed, 17 Apr 2024 08:12:09 +0000

The cluster autoscaler brings horizontal scaling into your cluster by deploying it into the cluster to autoscale. This is described in the following blog article https://www.dbi-services.com/blog/rancher-autoscaler-enable-rke2-node-autoscaling/. It didn’t emphasize much about the user and role configuration.

With Rancher, the cluster autoscaler uses a user’s API key. We will see how to configure minimal permissions by creating Rancher roles for cluster autoscaler.

Rancher user

First, let’s create the user that will communicate with Rancher, and whose token will be used. It will be given minimal access rights which is login access.

Go to Rancher > Users & Authentication > Users > Create.

Set a username, for example, autoscaler
Set the password
Give User-Base permissions
Create

The user is now created, let’s set Rancher roles with minimal permission for the cluster autoscaler.

Rancher roles authorization

To make the cluster autoscaler work, the user whose API key is provided needs the following roles:

Cluster role (for the cluster to autoscale)
Get/Update for clusters.provisioning.cattle.io
Update of machines.cluster.x-k8s.io
Project role (for the namespace that contains the cluster resource (fleet-default))
Get/List of machines.cluster.x-k8s.io

Go to Rancher > Users & Authentication > Role Templates > Cluster > Create.
Create the cluster role. This role will be applied to every cluster that we want to autoscale.

Then in Rancher > Users & Authentication > Role Templates > Project/Namespaces > Create.
Create the project role, it will be applied to the project of our local cluster (Rancher) that contains the namespace fleet-default.

Rancher roles assignment

The user and Rancher roles are created, let’s assign them.

Project role

First, we will set the project role, this is to be done once.
Go to the local cluster (Rancher), in Cluster > Project/Namespace.
Search for the fleet-default namespace, by default it is contained in the project System.
Edit the project System and add the user with the project permissions created precedently.

Cluster role

For each cluster where you will deploy the cluster autoscaler, you need to assign the user as a member with the cluster role.
In Rancher > Cluster Management, edit the cluster’s configuration and assign the user.

The roles assignment is done, let’s proceed to generate the token that is provided to the cluster autoscaler configuration.

Rancher API keys

Let’s create an API Key for the cluster autoscaler configuration. Note that in a recent update of Rancher, the API keys expired by default in 90 days.

If you see this limitation, you can do the following steps to have no expiration.
With the admin account, in Global settings > Settings, search for the setting auth-token-max-ttl-minutes and set it to 0.

Go back with the autoscaler user and create the API Key, name it for example, autoscaler, and select “no scope”.

You can copy the Bearer Token, and use it for the cluster autoscaler configuration.

As seen above, the token never expires.
Let’s reset the parameter auth-token-max-ttl-minutes and use the default value button or the precedent value set.

We are now done with the roles configuration.

Conclusion

This blog article covers only a part of the setup for the cluster autoscaler for RKE2 provisioning. It explained the configuration of a Rancher user and Rancher’s roles with minimal permissions to enable the cluster autoscaler. It was made to complete this blog article https://www.dbi-services.com/blog/rancher-autoscaler-enable-rke2-node-autoscaling/ which covers the whole setup and deployment of the cluster autoscaler. Therefore if you are still wondering how to deploy and make the cluster autoscaler work, check the other blog.

Links

Rancher official documentation: Rancher
RKE2 official documentation: RKE2
GitHub cluster autoscaler: https://github.com/kubernetes/autoscaler/tree/master/cluster-autoscaler

Blog – Rancher autoscaler – Enable RKE2 node autoscaling
https://www.dbi-services.com/blog/rancher-autoscaler-enable-rke2-node-autoscaling
Blog – Reestablish administrator role access to Rancher users
https://www.dbi-services.com/blog/reestablish-administrator-role-access-to-rancher-users/
Blog – Introduction and RKE2 cluster template for AWS EC2
https://www.dbi-services.com/blog/rancher-rke2-cluster-templates-for-aws-ec2
Blog – Rancher RKE2 templates – Assign members to clusters
https://www.dbi-services.com/blog/rancher-rke2-templates-assign-members-to-clusters

L’article Rancher RKE2: Rancher roles for cluster autoscaler est apparu en premier sur dbi Blog.

Learning Azure by having fun with ChatGPT

DevOps — Tue, 16 Apr 2024 07:27:16 +0000

Are you studying for an Azure certification? You may feel at some point you need a break from the dry and factual technical information. I’m in your shoes right now and I have done some experiments with ChatGPT. I’m not using it to write my text but to learn by having fun! It is very good at telling stories so why not use that for what interests me right now? Azure architecture!

In this blog post I’m going to show you how I use it, it may give you some ideas to make your learning a little more spicy!

Prompt for preparing ChatGPT with Azure

First you’ll need to prepare ChatGPT by explaining what you want. Set the framework and limit its responses otherwise it is too chatty. As I like science fiction (sci-fi), I’m interested in a futuristic story but you could choose any style you like. My first prompt and ChatGPT response is shown below:

Act as an Azure architect. You are in charge of educating me, your student, to become an Azure architect too. You will respond to my questions by using story telling to explain each concepts requested.
I want you to:
– Always tell a simple and futuristic story
– Limit the reply to 100 words

As an Azure architect, imagine Azure as a vast cosmic city. Each service is a towering skyscraper, and we, architects, design the blueprints. Let’s start with Virtual Networks. They’re like the city’s infrastructure, connecting buildings. Subnets divide areas within each building, like floors. Next, VMs are the inhabitants, residing in these skyscrapers, each serving a unique purpose. Azure Storage, a network of secure vaults, safeguards their belongings. With Azure Load Balancer as traffic controllers, the city thrives, ensuring smooth navigation. As architects of this digital metropolis, we sculpt a future where innovation soars amidst the azure skyline.

Not bad! It gives me some key components of an Azure infrastructure and makes some analogies I can easily picture in my head. With this basis, I can now be the hero of this story by asking more questions.

Build up that Azure cosmic city with ChatGPT

You can then ask questions around your area of study (mine is Azure architecture) and stay immersed into that story. For example:

What are the best practice to govern this Azure cosmic city?
In our cosmic city, governance is paramount. Azure Policy acts as the city’s constitution, ensuring compliance with regulations. Role-Based Access Control (RBAC) assigns roles to citizens, granting appropriate permissions. Azure Resource Manager templates are like city blueprints, ensuring consistency in construction. Tags label buildings for easy identification, while Azure Monitor keeps a watchful eye on city activities. With these best practices, our cosmic city thrives, ensuring order, security, and harmony amidst the azure skyline.

It gives you the key elements of Azure governance by continuing its story. You can go on like this back and forth by throwing in any chapter of your course. You can focus on a specific use case as well and see how it integrates with this story:

How can I migrate my SQL Server database on-premises to this city?
Imagine your SQL Server database as a cozy cottage in the countryside. To migrate to our cosmic city, Azure, we’ll use Azure Database Migration Service, like a futuristic moving company. It carefully transports your data to Azure SQL Database, a sleek skyscraper in our city. With minimal downtime, your database seamlessly transitions to its new home, ready to thrive amidst the azure skyline. Just sit back, relax, and watch as your data becomes part of the bustling digital metropolis.

How do I ensure my database is highly resilient in this cosmic city?
In our cosmic city, resilience is key. Azure offers a robust solution: Azure SQL Database with built-in redundancy, akin to a fortress with multiple layers of defense. Automatic backups act as a safety net, ensuring data is never lost. Geo-replication creates duplicates across regions, safeguarding against disasters. With Azure SQL Database’s high availability and disaster recovery features, your database thrives amidst the azure skyline, resilient against any digital storm.

I don’t know if it works for you but I find it fascinating I can make ChatGPT talk endlessly and respond to any questions I have about Azure in a sci-fi story!

Dive into an Azure component

As usual with ChatGPT you need to drive the discussion to get the most of it. Let’s see what it has to say about Azure AD (I use the legacy terminology as ChatGPT v3.5 doesn’t know yet Microsoft Entra ID):

What and where would be the Azure AD in this city?
In our bustling city, Azure AD is the central hub, akin to City Hall. It’s where identities are registered, like citizens in our digital society. Located at the heart of the city, Azure AD ensures secure access to every digital service and resource, much like City Hall grants access to essential services. From authentication to authorization, Azure AD governs the city’s digital identities, ensuring a seamless and secure experience for all inhabitants amidst the azure skyline.

How would Azure AD conditional access be implemented in this city?
In our city, Azure AD Conditional Access acts as the city’s security checkpoints. Picture futuristic gates at key entry points, only granting access to authorized individuals based on specific conditions. Want to enter the financial district skyscraper? You’ll need more than just a pass; perhaps a biometric scan during business hours. Conditional Access ensures only the right people, with the right permissions, access sensitive areas of our digital city. It’s our shield against cyber threats amidst the azure skyline, ensuring a safe and secure digital environment for all inhabitants.

How would I connect from my Active Directory in the countryside to this Azure AD in the cloud?
In the countryside, your Active Directory (AD) is like a quaint village square, bustling with local identities. To connect to Azure AD in our cosmic cloud city, envision a digital bridge spanning the distance between village and skyline. Azure AD Connect is your trusty guide, seamlessly synchronizing identities from countryside AD to Azure AD. With this bridge in place, villagers can access digital services in the cloud using their familiar countryside credentials. Just like that, the countryside becomes an integral part of our digital metropolis, united under the azure skyline.

You can then piece by piece for each component create a complete city in the countryside (OnPrem) and in the cloud and illustrate their interaction. We could completely gamify this! SimCity anyone?

Limitations

Some general concepts are more difficult to put into a visual story as for example the Microsoft Cloud Adoption Framework:

What would be the Microsoft Cloud Adoption Framework for Azure in this city?
In our dynamic metropolis, the Microsoft Cloud Adoption Framework for Azure is our roadmap to success. Like a futuristic GPS, it guides us through the twists and turns of digital transformation. Envisioning helps us set our destination, planning ensures we choose the best route, and readiness prepares us for the journey ahead. Migration helps us seamlessly transition, while innovation propels us forward. With governance as our compass and optimization as our fuel, we navigate the ever-changing landscape, building a future where our digital city thrives amidst the azure skyline. Ready to embark on this transformative journey?

I don’t find it fantastic as we recognize the empty blabbering of ChatGPT. We would need to be more specific to get something more interesting.

Wrap up

You can see how by relaxing from your studies, you can still continue to learn by having fun in an imaginary world. You could totally convert all this into visual notes that will help you when you renew your certifications. That is something I’m starting to explore.

This is just a glimpse of how you could use ChatGPT in your journey to learn Azure or anything else. Brainstorm any concept, service or component you are learning and see how it integrates into a visual story to get a high-level picture. Let me know if your are using ChatGPT that way for learning and what is the world you are building for it!

L’article Learning Azure by having fun with ChatGPT est apparu en premier sur dbi Blog.

DevOps Best Practice – Backup and Share your work with GitHub

DevOps — Tue, 09 Apr 2024 06:17:19 +0000

With my mate Chay Te (our DevOps champion in all categories and the mastermind of this best practice) we worked on scripts for our new Kubernetes security talk. These scripts where stored in our EC2 instance but this should not be their permanent location. First the EC2 instance could be deleted and we would lose everything. Then we need to version these files and keep track of the changes between us two. It was time to apply DevOps best practice for our scripts and we decided to use GitHub for this purpose. Read on to learn how to backup and share your work with GitHub in this step-by-step guide!

GitHub

The first step is to sign up for a GitHub account if you don’t already have one.

Then you can create your first repository (also called repo for short) by giving it a name. You can select a Private repo if the files you share are private (it was in our case). So far so good, nothing complicated here!

Now you want to connect from your EC2 instance (in our case but it could be any type of machine) to this repo and push your scripts. Before you can do that, there is some configuration to do in GitHub. You have to create a Personal Access Token (PAT) to allow this connection. Click on your profile in the top right corner and select Settings. Then choose Developer Settings and you will reach the PAT menu. Here there are 2 choices between a fine-grained and a classic token. The first one is in Beta and allow you to choose which access you want to give to each element of your repo. You give it a name and the token will be generated for you. It has an expiration date and you have to keep it somewhere safe like a password as you will not be able to retrieve it later.

You can now use your GitHub account name and this token to synchronize your scripts or files between EC2 and this repo.

The last thing to configure in GitHub is to invite your collaborators to access your repo. Click on Add people and enter the email address of your collaborator. She/He will receive an invite to accept to join you in this repo.

Creating a repo and collaborating in it is part of DevOps best practice!

Git commands in EC2

Your GitHub repo is now ready so let’s use it and backup your scripts in it. Another DevOps best practice is to use Git as the CLI tool in our machine.

On the EC2 instance, the easiest way to proceed is to clone your GitHub repo (we give it the name MyNewRepo) with Git as follows:

$ git clone https://github.com/mygithubaccount/MyNewRepo.git

You will be asked to authenticate with your GitHub account name (here mygithubaccount) and use the PAT you have created above as password. In your EC2 instance you now have a new folder called MyNewRepo. At this stage it is empty. Go into it and set the Git configuration:

$ cd MyNewRepo

$ git config --global user.email "benoit.entzmann@dbi-services.com"
$ git config --global user.name "Benoit Entzmann"
$ git branch -M main
$ git remote add origin https://github.com/mygithubaccount/MyNewRepo.git

You set the global email and username you will use with Git. By default there is one Git branch that is called Master. Rename it as main. Finally set up a connection between your local Git repository and your remote repository.

Next copy or move all of your script files into this folder as shown in the example below:

$ cp -Rp ~/MyScripts/* ./

Now all of your script files are in right folder and you just need to add them to the local Git repo and push them to your repo in GitHub:

$ git add .
$ git commit -m "My scripts"
$ git push -u origin main

And this is it! You can just check in GitHub that all of your script files are now in the repo called MyNewRepo.

Wrap up

In a few steps we have seen how to backup your script files by using a repository in GitHub. You have not only backup your files, you have also setup the GitHub environment to collaborate in this repo. This is a DevOps best practice!

Now in case of a failure or accidental deletion of your EC2 (yes Instance state -> Terminate instance can happen!), you will be able to clone again your repo from GitHub and quickly get back on track with your scripts!

L’article DevOps Best Practice – Backup and Share your work with GitHub est apparu en premier sur dbi Blog.

Enhance Containers Security – Prevent Encrypted Data Exfiltration with NeuVector

DevOps — Tue, 02 Apr 2024 06:07:22 +0000

In my previous blog post we have seen how NeuVector from SUSE can detect and prevent data exfiltration. We used the DLP (Data Loss Prevention) feature of NeuVector to recognize patterns in our HTTP packet. That was great but what could you do when the traffic is not in clear text but encrypted with HTTPS instead? I ended my previous blog saying that we would then need to apply a different security strategy. Let’s find out what we can do and how NeuVector can help with that.

Application Baseline

Before deploying a new containerized application in production, you have to assess it first. From the security point of view it means you have to learn what processes are running in this container and what are the network connections to and from it.

This observability phase will help you define what is the normal behaviour of your application. That will be your baseline.

A good practice is to deploy first your application in a dev or test environment. Here you can do your assessment in a controlled environment. NeuVector can easily help you with this task as we can see in the picture below:

We can see all the processes that are currently running in the application’s container.

To exfiltrate data you will also need a connection to an external website or server that is under the control of the attacker. With NeuVector we can see all the connections related to this container as shown below:

We can see this container connects to the coredns pod of our cluster and has one external connection with a server outside of our cluster.

We can get more information on this external server by looking at the network map and click on that connection:

All these information are key to your security strategy again encrypted data exfiltration. There are several ways to exfiltrate data but all involve running a process in that container. If it is not a malware, it could be a simple SSH or a curl command. As a security good practice, these tools shouldn’t be available in your application’s container. You have to reduce the possibility of attacks and exploitation to the minimum.

Also data exfiltration requires a connection to an external server. As the traffic is encrypted, you will not get an alert and can’t use DLP. However, you’ll see an abnormal external connection for your application.

It is then paramount to create a baseline of your application and investigate everything that is a drift from it.

Zero Trust Architecture for encrypted data exfiltration

Basically zero trust means you don’t trust anything or anybody. In our topic about encrypted data exfiltration, it means we don’t trust any behaviour that is not part of the normal behaviour of our application (our baseline).

NeuVector can help us with that too. Once we are confident, we have learned all the normal behaviour of our application, we are ready to move it to production. Here we can monitor any drift from our baseline as shown below:

We switch the mode for this container from Discover to Monitor. By default Zero drift is set which means we will now log any new behaviour that is unknown. We can see this new mode for our container below:

Let’s now see how we could detect an encrypted data exfiltration by looking at the “Network Activity” map:

In addition to our normal external connection, we see another one using port 443 (HTTPS). This is a drift from our baseline and you’ll have to investigate it. We can check the security events to learn more about it:

We can first see an alert about a curl process that is not part of the processes we have identified as normal in our baseline. NeuVector logs it as a process profile rule violation. We then see another violation, this one is related to our networking rules. There is an implicit deny rule for any traffic that is not what has been discovered by NeuVector (in our baseline). Setting our container in monitoring mode will not stop that traffic, it will log these drifts as security events and we have then to investigate.

With these 2 informations we have a high probability that our container has been compromised. We can’t say if it is data exfiltration or something else, you can just see a process in your container is connecting to an unknown external server. Note that even if it connects just once, it will be captured by NeuVector. So even stealthy connections will be detected.

At this stage, you have to investigate to discard a false positive alert. Maybe somebody did some tests with that container to check connectivity for example. Once you are confident this is abnormal, you can take some actions to stop this abnormal behavior. To do so we switch our container into “Protect” mode to block any drift. You can do that directly from the map as shown below:

From this point NeuVector will block any network traffic or process that is not part of our baseline.

Wrap up

Congratulations! We have detected and protected our container and indirectly our cluster by applying a zero trust security strategy. As the traffic is encrypted we can’t see what it is and can’t tell it is precisely data exfiltration. However we have identified its operating pattern and were able to block it. From there you can investigate deeper how this container has been compromised by checking the logs, reviewing the accesses and the roles in your cluster.

Note that this zero trust strategy allows you to defeat not only encrypted data exfiltration but any unknown attack as well. It is a very effective strategy and we recommend you to deploy it in your Kubernetes cluster as a best practice. Stay safe!

L’article Enhance Containers Security – Prevent Encrypted Data Exfiltration with NeuVector est apparu en premier sur dbi Blog.

Cloud Native Storage: Identify your storage

DevOps — Wed, 27 Mar 2024 14:25:35 +0000

Welcome back in this series of blogs regarding Cloud Native Storage. Check my previous on Cloud Native Storage: Overview for the introduction.

In this one, I will discuss about the process involved in choosing a cloud native storage product. If you remember my previous blog, I pasted the exhaustive (big!) list of products. Of course, if you are familiar with Kubernetes you’ll probably know that we can create multiple storage classes, and you are right. The point here is more about choosing a product that will fit a specific workload.
Workload can be of several kinds.

Databases
Stateless/stateful application
Monitoring stack
Web applications
Big datatest
Microservices architectures
e-commerce
Healthcare sensitive data
Machine learning

First approach – Know your constraints

It’s mandatory to know what are the constraints when thinking about your workload. Let’s try to list some of them and determine which ones are relevant to your use case

Reliability and durability
Scalability
Performance
Security
Cloud/On premise
Storage type (S3, nvme, …)
Cost
Lifecycle management
Observability
Ease of use
Vendor support
Popularity

There is so much to say regarding each constraints, that’s why I mentioned to do a short list of main constraints and secondly to weight them. This will help you focus on essential expectations. Let’s take an example with cost. You may remember that products with a white background are open source, it not only means the product is free to use but also that you’ll be able to compare different products and why not also compare a proprietary product with trials that are often offered.

We can also take the performance constraint. This one is essential with relational databases workload. If it’s your case and you’re new with the topic, you’ve probably chosen the local PV storage to maximize latency and throughput, but with more experience you’ll find that products like

Portworx that allows you to control IOPS or throughput at the storage layer here.
Linbit that has impressive IOPS performance here

Adoption

Another constraint example I would like to talk about is popularity. Most of the products should be either supported by vendor or adopted by majority of the community so it guarantees you (a certain degree) of confidence to use it. My opinion reflect of course a “majority adopter” posture. In case your posture is “early”, it means you probably contribute to open source, then I just want to say “Thank you!” and keep going. I hope I will also be able to contribute in a near future. In case your posture is “laggards”, then continue your analysis with PoC to gain confidence.
You’ll find below the Innovation model lifecycle from Wikipedia.

All kinds of adopters have their pros and cons and contribute to the community by giving feedbacks from their usage.

This brings me to the next point I wanted to mention in this blog. How can we discuss with contributors, users and vendor.
Let’s take the easy point with vendor. If you have something to discuss (issue, remarks, usage, feedback) the official vendor communication channels (e-mail, slack, sales, ticket, etc …) will be the best.
Now, regarding open-source products, there are severals way to discuss points, you have official vendor communication channels (e-mail, slack, sales, ticket, etc …) and also what you can find from people usage (stackoverflow, reddit, …).
Of course, said like that open-source has more possibility to get you answer. But the main difference is SLA with a product you paid for, vendor will have to give you an answer in a defined timeframe regarding the priority given to your ticket.

Conclusion

All discussed points brings us to the final words that

There is no silver bullet solution
You have to know and weight your constraints
You need to know where you stand regarding adoption

We, at dbi services, can provide support to help you choose and also accompany you on your journey to understand the CNCF. Don’t hesitate to post comments or contact our sales team for support.

In the next blog, I’ll go deeper with a concrete example of a database workload that will leverage on a cloud native storage.

L’article Cloud Native Storage: Identify your storage est apparu en premier sur dbi Blog.