You want to deploy Ansible AWX on a Kubernetes cluster using awx-operator, but your security policy requires you to use a local registry to store the necessary container images ?
Okay, let’s see how to do this using Helm.
Default deployment
By reading the awx-operator documentation, you should have a values file similar to this one (with spec values according to your needs):
---
AWX:
enabled: true
name: awx-demo
spec:
service_type: ClusterIP
Then add awx-operator repository to Helm and deploy it:
$ helm repo add awx-operator https://ansible.github.io/awx-operator/
"awx-operator" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "awx-operator" chart repository
...Successfully got an update from the "harbor" chart repository
...Successfully got an update from the "komodorio" chart repository
Update Complete. ⎈Happy Helming!⎈
$ helm install -n awx --create-namespace awx-demo awx-operator/awx-operator -f awx.yaml
Release "awx-demo" does not exist. Installing it now.
NAME: awx-demo
LAST DEPLOYED: Fri Jun 9 15:23:27 2023
NAMESPACE: awx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWX Operator installed with Helm Chart version 2.2.1
After a bit of time, AWX should be well deployed:
$ kubectl get pods -n awx
NAME READY STATUS RESTARTS AGE
awx-operator-controller-manager-7695c6c799-9sh6x 2/2 Running 0 41m
awx-demo-postgres-13-0 1/1 Running 0 40m
awx-demo-task-57ccc9c6d-c6qx7 4/4 Running 0 40m
awx-demo-web-5f486b545-6vfm2 3/3 Running 0 40m
Set components images
We just deployed our AWX with the default images, which one are in use?
$ kubectl get pods -n awx -o jsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}'
gcr.io/kubebuilder/kube-rbac-proxy:v0.13.0
quay.io/ansible/awx-operator:2.2.1
postgres:13
docker.io/redis:7
quay.io/ansible/awx:22.3.0
quay.io/ansible/awx-ee:latest
quay.io/ansible/awx:22.3.0
docker.io/redis:7
quay.io/ansible/awx:22.3.0
quay.io/ansible/awx:22.3.0
Now we want to use images stored on a local registry, for example here, we use Harbor. Harbor is deployed on Kubernetes cluster and works as a remote registry replication.
In the awx-operator documentation, we can find values to set in the “spec” dictionary for various image locations. Here the new “awx.yaml” values file with these changes:
---
AWX:
enabled: true
name: "awx-demo"
spec:
hostname: awx-demo.dwi.local
service_type: ClusterIP
image: harbor.dwi.local/awx-demo/awx
image_version: latest
postgres_image: harbor.dwi.local/awx-demo/postgres
postgres_image_version: "13"
ee_images:
- name: AWX Demo EE
image: harbor.dwi.local/awx-demo/awx-ee:latest
control_plane_ee_image: harbor.dwi.local/awx-demo/awx-ee:latest
redis_image: harbor.dwi.local/awx-demo/redis
redis_image_version: "7"
Now let’s delete and reinstall awx-operator completely to see what happens:
$ helm delete -n awx awx-demo
release "awx-demo" uninstalled
$ helm install -n awx --create-namespace awx-demo awx-operator/awx-operator -f awx.yaml
NAME: awx-demo
LAST DEPLOYED: Fri Jun 9 15:45:18 2023
NAMESPACE: awx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWX Operator installed with Helm Chart version 2.2.1
$ kubectl get pods -n awx -o jsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}'
gcr.io/kubebuilder/kube-rbac-proxy:v0.13.0
quay.io/ansible/awx-operator:2.2.1
harbor.dwi.local/awx-demo/postgres:13
harbor.dwi.local/awx-demo/redis:7
harbor.dwi.local/awx-demo/awx:latest
harbor.dwi.local/awx-demo/awx-ee:latest
harbor.dwi.local/awx-demo/awx:latest
harbor.dwi.local/awx-demo/redis:7
harbor.dwi.local/awx-demo/awx:latest
harbor.dwi.local/awx-demo/awx:latest
Now, most pods are using images from our local registry, with the exception of kube-rbac-proxy and awx-operator. Unfortunately, there are no values in the Helm to change them… So we’re going to rebuild the Helm chart 💪
Rebuild Helm chart
First we will clone the awx-operator project in /tmp/awx-operator
$ git clone https://github.com/ansible/awx-operator.git /tmp/awx-operator
Cloning into '/tmp/awx-operator'...
remote: Enumerating objects: 8796, done.
remote: Counting objects: 100% (1432/1432), done.
remote: Compressing objects: 100% (224/224), done.
remote: Total 8796 (delta 1251), reused 1306 (delta 1203), pack-reused 7364
Receiving objects: 100% (8796/8796), 2.35 MiB | 3.53 MiB/s, done.
Resolving deltas: 100% (5062/5062), done.
$ cd /tmp/awx-operator
The Helm chart is not directly sourced in the project, so we will use the Makefile that transforms the Kustomize into a Helm chart, and here’s some good news: we have two variables to define image and version of the awx-operator (IMAGE_TAG_BASE and VERSION).
$ IMAGE_TAG_BASE=harbor.dwi.local/awx-demo/awx-operator VERSION=2.2.1 make helm-chart
mkdir -p charts
== KUSTOMIZE: Set image and chart label ==
cd config/manager && /tmp/awx-operator/bin/kustomize edit set image controller=harbor.dwi.local/awx-demo/awx-operator:latest
cd config/manager && /tmp/awx-operator/bin/kustomize edit set label helm.sh/chart:awx-operator
cd config/default && /tmp/awx-operator/bin/kustomize edit set label helm.sh/chart:awx-operator
== Gather Helm Chart Metadata ==
[...]
Helm chart successfully configured for awx-operator version 2.2.1
Once this has been done, we will manually modify the template that uses the kube-rbac-proxy image. So, using vi (or another editor), edit the file /tmp/awx-operator/charts/awx-operator/templates/deployment-awx-operator-controller-manager.yaml
Modify line:
image: gcr.io/kubebuilder/kube-rbac-proxy:v0.13.0
To match the image on your local registry
image: harbor.dwi.local/awx-demo/kube-rbac-proxy:v0.13.0
We can finally delete the actual Helm release and deploy our custom awx-operator Helm chart:
$ helm delete -n awx awx-demo
release "awx-demo" uninstalled
$ helm install -n awx --create-namespace awx-demo /tmp/awx-operator/charts/awx-operator -f awx.yaml
NAME: awx-demo
LAST DEPLOYED: Fri Jun 23 01:46:27 2023
NAMESPACE: awx
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
AWX Operator installed with Helm Chart version 2.2.1
Wait a few seconds for all the pods to start up correctly, then you can list the images used:
$ kubectl get pods -n awx -o jsonpath='{range .items[*].spec.containers[*]}{.image}{"\n"}{end}'
harbor.dwi.local/awx-demo/kube-rbac-proxy:v0.13.0
harbor.dwi.local/awx-demo/awx-operator:2.2.1
harbor.dwi.local/awx-demo/postgres:13
harbor.dwi.local/awx-demo/redis:7
harbor.dwi.local/awx-demo/awx:latest
harbor.dwi.local/awx-demo/awx-ee:latest
harbor.dwi.local/awx-demo/awx:latest
harbor.dwi.local/awx-demo/redis:7
harbor.dwi.local/awx-demo/awx:latest
harbor.dwi.local/awx-demo/awx:latest
Congratulations ! You now have an AWX installed on your Kubernetes cluster that using only local images 🎉
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/DWI_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/HER_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/GRE_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/DWE_web-min-scaled.jpg)
elgreco
05.02.2024With Operator 2.10.0 and AWX 23.6.0 sourced from a private Container Registry backed by STEP-CA, which I also added to my Minikube image (running rootless), it seems to work at first, but the awx-task resource has an error on the ReplicaSet, where the override does not work, which you do not notice at first ... only when you do a kubectl describe after while to find it uses the default (quay.io/ansible/awx-ee:latest) as defined in https://github.com/ansible/awx-operator/blob/devel/roles/installer/defaults/main.yml. Other than that, your post is very complete.