Kubernetes SSO using Azure AD

I’ve worked on this project to authenticate Kubernetes users against Azure AD and so have Single Sign-On (SSO). That was a long journey on a rocky road and it’s worth summarising in this blog all the steps required to make it done. I didn’t find the Kubernetes and Azure documentation helpful enough to make it work straight away. I had to collect chunks of information here and there to build a working solution and you will find here information I didn’t see anywhere else on the Web. If you too need to configure SSO with Azure AD, you’re in the right place and I’ll share everything I’ve learned to help you set it up!

The starting point

Our cluster is an OnPrem Kubernetes cluster (so we are not using AKS – the Azure Kubernetes solution – on which most documentation relates) and the Azure configuration part is done in our case by a dedicated team.

In Azure you’ll need to register an application, set a Redirect URI (web page and path used for authentication) and take note of the Application ID, Client ID and Client Secret generated by this new registration. You’ll need those when configuring Kubernetes to interact with Azure AD. Of course you’ll also need an Azure AD account which is an email address and a password.

On the Kubernetes side, we do the configuration on the Master node of this cluster and there are 2 places where you will configure this interaction with Azure AD: The /etc/kubernetes/manifests/kube-apiserver.yaml file and the .kube/config file used by kubectl. Sounds easy right? Buckle up we are entering a zone of turbulence!

The deprecated method

This method is the first I’ve found and tried. In Azure you have to register 2 Applications in Azure AD: One for Kubernetes and one for kubectl.

On the Kubernetes side the configuration is as follows:

kube-apiserver.yaml uses the Azure Kubernetes Application registration information:

    - --oidc-client-id=...
    - --oidc-issuer-url=https://sts.windows.net/.../
    - --oidc-username-claim=...

For the kubectl command fill the file .kube/config using the Azure kubectl Application registration information:

$ kubectl config set-credentials <user_name> \
--auth-provider=azure \
--auth-provider-arg=environment=AzurePublicCloud \
--auth-provider-arg=client-id=... \
--auth-provider-arg=client-secret=... \
--auth-provider-arg=tenant-id=... \
--auth-provider-arg=apiserver-id=...

$ kubectl config set-context <context_name> \
--cluster=<cluster_name> \
--namespace=default \
--user=<user_name>

That looks pretty straightforward, so I’ve switched to this new context and got the following:

$ kubectl config use-context <context_name>
$ kubectl get pods -A

W1118 10:00:38.620058 3523839 azure.go:92] WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https://github.com/Azure/kubelogin instead.
To learn more, consult https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins
E1118 10:00:38.773424 3523839 azure.go:162] Failed to acquire a token: failed acquiring new token: initialing the device code authentication: autorest/adal/devicetoken: Error occurred while handling response from the Device Endpoint: Error HTTP status != 200
Unable to connect to the server: acquiring a token for authorization header: failed acquiring new token: initialing the device code authentication: autorest/adal/devicetoken: Error occurred while handling response from the Device Endpoint: Error HTTP status != 200

I would need to dig deeper to make this work but what annoyed me was that this method is deprecated and so for a future proof solution I’ve started to look at kubelogin as recommended in the output above.

The future proof method – Authentication

With this method you only need to register one Application in Azure and take note of the same information as in the previous method.

On the Kubernetes side, first I’ve installed krew the package manager for kubectl in order to install the oidc-login plugin:

$ curl -fsSLO https://github.com/kubernetes-sigs/krew/releases/latest/download/krew-linux_amd64.tar.gz
$ tar zxvf krew-linux_amd64.tar.gz
$ ./krew-linux_amd64 install krew
$ kubectl krew update
$ kubectl krew install oidc-login

Then I’ve configured Kubernetes as follows:

kube-apiserver.yaml uses the Azure Application registration information:

    - --oidc-client-id=...
    - --oidc-issuer-url=https://sts.windows.net/.../
    - --oidc-username-claim=...

This is the same config as with the deprecated method, however the kubectl parameters to fill in the .kube/config file is now different:

kubectl config set-credentials <user_name> \
--exec-api-version=client.authentication.k8s.io/v1beta1 \
--exec-command=kubectl \
--exec-arg=kubectl-oidc_login \
--exec-arg=get-token \
--exec-arg=--oidc-issuer-url="https://sts.windows.net/.../" \
--exec-arg=--oidc-client-id=... \
--exec-arg=--oidc-client-secret=...

Be careful to not put ” ” around your secret value otherwise it will not work as I will show you a bit further below.

Now we are ready to initiate the authentication:

$ kubectl-oidc_login get-token \
--oidc-issuer-url="https://sts.windows.net/.../" \
--oidc-client-id=... \
--oidc-client-secret="..."

Be careful here too, here you have to put ” ” around your secret value! Yes I’ve told you I’ll share everything with you and at this point with those information you will already save yourself severals days of headaches. But we are not done yet. Hang on tight!

The result of the command above is as follows:

error: could not open the browser: exec: "xdg-open,x-www-browser,www-browser": executable file not found in $PATH
Please visit the following URL in your browser manually: http://localhost:8000

First if you only have command line access to this server, you can avoid the error above and configure to not try to open the web browser automatically by adding the parameter –grant-type=authcode-keyboard in the file .kube/config

At this stage you need to open the URL http://localhost:8000 in a web browser on the Master node to enter your Azure credentials. In our configuration, the cluster nodes have no web browser as those nodes are only using a command line interface. This is where you need to be creative and for example configure Putty with a ssh tunnel in order for your Web browser on your office machine to connect directly to the cluster node on this port (L8000 localhost:8000). This is what we did for opening the URL in the web browser of our office machine. The first time you need to authenticate with your Azure AD credentials and then if succeeded, you’ll automatically get a token in your Master node terminal for a period of time as shown below:

{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1beta1","spec":{"interactive":false},"status":{"expirationTimestamp":"2022-11-25T10:20:43Z","token":"eyJ0eXAiOi..."}}

The URL you see in your terminal has to match the Redirect URI parameter of your application registration in Azure. You could also configure this URL in kubelogin and Redirect URI in your Azure application with a path that would be reachable from your office machine in order to perform this authentication (and avoid using an ssh tunneling from the Office machine to the Master node). Note that this path needs to use https:// (because http is only allowed by Azure for localhost). Keep a note of that as I’ll come back to this in the limitations section below.

At this stage you are now authenticated.

Before moving on, let’s take a look at this error you may encounter when trying to authenticate:

$ kubectl get pods
error: could not open the browser: exec: "xdg-open,x-www-browser,www-browser": executable file not found in $PATH

Please visit the following URL in your browser manually: http://localhost:8000
error: get-token: authentication error: authcode-browser error: authentication error: authorization code flow error: oauth2 error: could not exchange the code and token: oauth2: cannot fetch token: 401 Unauthorized
Response: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '...'.\r\nTrace ID: ...\r\nCorrelation ID: ...\r\nTimestamp: 2022-11-25 07:46:29Z","error_codes":[7000215],"timestamp":"2022-11-25 07:46:29Z","trace_id":"d...","correlation_id":"...","error_uri":"https://login.windows.net/error?code=7000215"}

If you get this ugly error and you’re pretty sure you’ve set the client secret value properly then as written previously, remove the ” ” around this secret in the .kube/config file and you’ll be good to go.

The future proof method – RBAC

You can then enter your kubectl commands:

$ kubectl get pods
Error from server (Forbidden): pods is forbidden: User "<user_name>" cannot list resource "pods" in API group "" in the namespace "default"

Yes that is also an error but a beautiful one! This means you’ve passed the authentication stage for running kubectl commands! You just need to define some Role-Based Access Control (RBAC) for your user in order to specify what he can do and see in this cluster (authorization stage).

For example you can configure a clusterrole and clusterrolebinding as follows in a yaml file:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: test-sso-clusterrolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: test-sso-clusterrole
subjects:
- apiGroup: rbac.authorization.k8s.io
  kind: User
  name: "<user_name>"

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: test-sso-clusterrole
rules:
- apiGroups: [""] # "" indicates the core API group
  resources: ["nodes", "pods"]
  verbs: ["get", "watch", "list"]

Apply this file and you’ll now see the results below:

$ kubectl get secrets
Error from server (Forbidden): secrets is forbidden: User "<user_name>" cannot list resource "secrets" in API group "" in the namespace "default"

$ kubectl get po
No resources found in default namespace.

$ kubectl get no
NAME    STATUS   ROLES           AGE   VERSION
node1   Ready    control-plane   9d    v1.24.4
node2   Ready    <none>          9d    v1.24.4
node3   Ready    <none>          9d    v1.24.4
node4   Ready    <none>          9d    v1.24.4
node5   Ready    <none>          9d    v1.24.4
node6   Ready    <none>          9d    v1.24.4

As expected, you can access only what is defined in your clusterrole/clusterrolebinding.

Hold on! Are you frustrated because maybe you couldn’t apply that yaml file right? No problem I’ll not let you down and let’s clear that up!

If you are still using the context that uses SSO you don’t have access to the resources clusterrole and clusterrolebinding. If you’ve followed along this blog you should have 2 contexts: One that is accessing your cluster without SSO and one with it. You’ll then need first to switch to the context without SSO:

$ kubectl config get-contexts
$ kubectl config use-context <context_without_SSO>

Then apply your yaml file and switch back to your context with SSO in order to test those RBAC in action.

It comes without saying that when your cluster will be fully configured you’ll need to remove that context without SSO otherwise your cluster security is deficient and all those efforts were for nothing!

Limitations & Conclusion

I’ve talked about a limitation regarding the Redirect URI that supports only https in Azure. Unfortunately the oidc plugin of kubectl seems to support only http… So here only the localhost URI as used in this blog can be used.

If you don’t want to use the default localhost as the Redirect URL in kubectl you can play with the following parameters of the .kube/config file:

 - --listen-address=0.0.0.0:8000
 - --oidc-redirect-url-hostname=<http url>

By default the kubectl-oid process listen on the local IP address 127.0.0.1 so you can change it (and have to do it if you don’t want to use locahost as URL) with the listen-address parameter. You can check it afterwards as follows (you need to enter a kubectl command first in another terminal in order for the Redirect URL address to appear and the port to be put in the LISTEN state):

$ netstat -anp|grep 8000
tcp        0      0 0.0.0.0:8000            0.0.0.0:*               LISTEN      3457640/kubectl-oid

With this method you can then change the Redirect URL with something that is reachable from your office machine for example. However this URL must start with http as this can not be changed in the parameters of kubelogin. So we can’t use it with Azure as we can only use https and so there is a mismatch here.

Another promising method we’ve tried is to use the following parameter in .kube/config:

--grant-type=authcode-keyboard

You then get the following Redirect URL:

$ kubectl get pods
Please visit the following URL in your browser: https://login.windows.net/9b52ed8b...
Enter code:

So we could authenticate from any computer in order to get this code and this is an https URL! But for this to work we need to add the following URL urn:ietf:wg:oauth:2.0:oob as Redirect URI in Azure. Unfortunately Azure doesn’t support it either! We must set an URI starting with https:// in Azure so we are stuck again.

So the only working method is by authenticating using the URL http://localhost:… as this suits both kubelogin and Azure. To use something else we would need to insert kind of a proxy in the middle that could accomodate both sides.

I’ll be happy if based on this blog you can follow along and make your configuration work right away. It took me several days to make it work, sort out what is needed and what is not, reading the experience of others and building up on the insightful ideas of my fantastic DevOps Team in dbi services.

We just had a breakthrough (thank you Arnaud Berbier, our Mbappé in our DevOps Team!) and made this work using https with kubelogin! Stay tuned as I’ll share the details in a next blog!