I’ve worked on this project to authenticate Kubernetes users against Azure AD and so have Single Sign-On (SSO). That was a long journey on a rocky road and it’s worth summarising in this blog all the steps required to make it done. I didn’t find the Kubernetes and Azure documentation helpful enough to make it work straight away. I had to collect chunks of information here and there to build a working solution and you will find here information I didn’t see anywhere else on the Web. If you too need to configure SSO with Azure AD, you’re in the right place and I’ll share everything I’ve learned to help you set it up!<\/p>\n\n\n\n
Our cluster is an OnPrem Kubernetes cluster (so we are not using AKS – the Azure Kubernetes solution – on which most documentation relates) and the Azure configuration part is done in our case by a dedicated team.<\/p>\n\n\n\n
In Azure you’ll need to register an application, set a Redirect URI (web page and path used for authentication) and take note of the Application ID, Client ID and Client Secret generated by this new registration. You’ll need those when configuring Kubernetes to interact with Azure AD. Of course you’ll also need an Azure AD account which is an email address and a password.<\/p>\n\n\n\n
On the Kubernetes side, we do the configuration on the Master node of this cluster and there are 2 places where you will configure this interaction with Azure AD: The \/etc\/kubernetes\/manifests\/kube-apiserver.yaml<\/strong> file and the .kube\/config<\/strong> file used by kubectl. Sounds easy right? Buckle up we are entering a zone of turbulence!<\/p>\n\n\n\n This method is the first I’ve found and tried. In Azure you have to register 2 Applications in Azure AD: One for Kubernetes and one for kubectl.<\/p>\n\n\n\n On the Kubernetes side the configuration is as follows:<\/p>\n\n\n\n kube-apiserver.yaml<\/strong> uses the Azure Kubernetes Application registration information:<\/p>\n\n\n For the kubectl command fill the file .kube\/config<\/strong> using the Azure kubectl Application registration information:<\/p>\n\n\n That looks pretty straightforward, so I’ve switched to this new context and got the following:<\/p>\n\n\n I would need to dig deeper to make this work but what annoyed me was that this method is deprecated and so for a future proof solution I’ve started to look at kubelogin as recommended in the output above.<\/p>\n\n\n\n With this method you only need to register one Application in Azure and take note of the same information as in the previous method.<\/p>\n\n\n\n On the Kubernetes side, first I’ve installed krew<\/strong> the package manager for kubectl in order to install the oidc-login plugin:<\/p>\n\n\n Then I’ve configured Kubernetes as follows:<\/p>\n\n\n\n kube-apiserver.yaml<\/strong> uses the Azure Application registration information:<\/p>\n\n\n This is the same config as with the deprecated method, however the kubectl parameters to fill in the .kube\/config<\/strong> file is now different:<\/p>\n\n\n Be careful to not put ” ” around your secret value otherwise it will not work as I will show you a bit further below.<\/p>\n\n\n\n Now we are ready to initiate the authentication:<\/p>\n\n\n Be careful here too, here you have to put ” ” around your secret value! Yes I’ve told you I’ll share everything with you and at this point with those information you will already save yourself severals days of headaches. But we are not done yet. Hang on tight!<\/p>\n\n\n\n The result of the command above is as follows:<\/p>\n\n\n First if you only have command line access to this server, you can avoid the error above and configure to not try to open the web browser automatically by adding the parameter –grant-type=authcode-keyboard in the file .kube\/config<\/strong><\/p>\n\n\n\n At this stage you need to open the URL http:\/\/localhost:8000 in a web browser on the Master node to enter your Azure credentials. In our configuration, the cluster nodes have no web browser as those nodes are only using a command line interface. This is where you need to be creative and for example configure Putty with a ssh tunnel in order for your Web browser on your office machine to connect directly to the cluster node on this port (L8000 localhost:8000). This is what we did for opening the URL in the web browser of our office machine. The first time you need to authenticate with your Azure AD credentials and then if succeeded, you’ll automatically get a token in your Master node terminal for a period of time as shown below:<\/p>\n\n\n The URL you see in your terminal has to match the Redirect URI parameter of your application registration in Azure. You could also configure this URL in kubelogin and Redirect URI in your Azure application with a path that would be reachable from your office machine in order to perform this authentication (and avoid using an ssh tunneling from the Office machine to the Master node). Note that this path needs to use https:\/\/ (because http is only allowed by Azure for localhost). Keep a note of that as I’ll come back to this in the limitations section below.<\/p>\n\n\n\n At this stage you are now authenticated.<\/p>\n\n\n\n Before moving on, let’s take a look at this error you may encounter when trying to authenticate:<\/p>\n\n\n If you get this ugly error and you’re pretty sure you’ve set the client secret value properly then as written previously, remove the ” ” around this secret in the .kube\/config<\/strong> file and you’ll be good to go.<\/p>\n\n\n\n You can then enter your kubectl commands:<\/p>\n\n\n Yes that is also an error but a beautiful one! This means you’ve passed the authentication stage for running kubectl commands! You just need to define some Role-Based Access Control (RBAC) for your user in order to specify what he can do and see in this cluster (authorization stage).<\/p>\n\n\n\n For example you can configure a clusterrole and clusterrolebinding as follows in a yaml file:<\/p>\n\n\n Apply this file and you’ll now see the results below:<\/p>\n\n\n As expected, you can access only what is defined in your clusterrole\/clusterrolebinding.<\/p>\n\n\n\n Hold on! Are you frustrated because maybe you couldn’t apply that yaml file right? No problem I’ll not let you down and let’s clear that up!<\/p>\n\n\n\n If you are still using the context that uses SSO you don’t have access to the resources clusterrole and clusterrolebinding. If you’ve followed along this blog you should have 2 contexts: One that is accessing your cluster without SSO and one with it. You’ll then need first to switch to the context without SSO:<\/p>\n\n\n Then apply your yaml file and switch back to your context with SSO in order to test those RBAC in action.<\/p>\n\n\n\n It comes without saying that when your cluster will be fully configured you’ll need to remove that context without SSO otherwise your cluster security is deficient and all those efforts were for nothing!<\/p>\n\n\n\n I’ve talked about a limitation regarding the Redirect URI that supports only https in Azure. Unfortunately the oidc plugin of kubectl seems to support only http… So here only the localhost URI as used in this blog can be used.<\/p>\n\n\n\n If you don’t want to use the default localhost as the Redirect URL in kubectl you can play with the following parameters of the .kube\/config<\/strong> file:<\/p>\n\n\n By default the kubectl-oid process listen on the local IP address 127.0.0.1 so you can change it (and have to do it if you don’t want to use locahost as URL) with the listen-address parameter. You can check it afterwards as follows (you need to enter a kubectl command first in another terminal in order for the Redirect URL address to appear and the port to be put in the LISTEN state):<\/p>\n\n\n With this method you can then change the Redirect URL with something that is reachable from your office machine for example. However this URL must start with http as this can not be changed in the parameters of kubelogin. So we can’t use it with Azure as we can only use https and so there is a mismatch here.<\/p>\n\n\n\n Another promising method we’ve tried is to use the following parameter in .kube\/config<\/strong>:<\/p>\n\n\n You then get the following Redirect URL:<\/p>\n\n\nThe deprecated method<\/h2>\n\n\n\n
\n - --oidc-client-id=...\n - --oidc-issuer-url=https:\/\/sts.windows.net\/...\/\n - --oidc-username-claim=...\n<\/pre><\/div>\n\n\n
\n$ kubectl config set-credentials <user_name> \\\n--auth-provider=azure \\\n--auth-provider-arg=environment=AzurePublicCloud \\\n--auth-provider-arg=client-id=... \\\n--auth-provider-arg=client-secret=... \\\n--auth-provider-arg=tenant-id=... \\\n--auth-provider-arg=apiserver-id=...\n\n$ kubectl config set-context <context_name> \\\n--cluster=<cluster_name> \\\n--namespace=default \\\n--user=<user_name>\n<\/pre><\/div>\n\n\n
\n$ kubectl config use-context <context_name>\n$ kubectl get pods -A\n\nW1118 10:00:38.620058 3523839 azure.go:92] WARNING: the azure auth plugin is deprecated in v1.22+, unavailable in v1.25+; use https:\/\/github.com\/Azure\/kubelogin instead.\nTo learn more, consult https:\/\/kubernetes.io\/docs\/reference\/access-authn-authz\/authentication\/#client-go-credential-plugins\nE1118 10:00:38.773424 3523839 azure.go:162] Failed to acquire a token: failed acquiring new token: initialing the device code authentication: autorest\/adal\/devicetoken: Error occurred while handling response from the Device Endpoint: Error HTTP status != 200\nUnable to connect to the server: acquiring a token for authorization header: failed acquiring new token: initialing the device code authentication: autorest\/adal\/devicetoken: Error occurred while handling response from the Device Endpoint: Error HTTP status != 200\n<\/pre><\/div>\n\n\n
The future proof method – Authentication<\/h2>\n\n\n\n
\n$ curl -fsSLO https:\/\/github.com\/kubernetes-sigs\/krew\/releases\/latest\/download\/krew-linux_amd64.tar.gz\n$ tar zxvf krew-linux_amd64.tar.gz\n$ .\/krew-linux_amd64 install krew\n$ kubectl krew update\n$ kubectl krew install oidc-login\n<\/pre><\/div>\n\n\n
\n - --oidc-client-id=...\n - --oidc-issuer-url=https:\/\/sts.windows.net\/...\/\n - --oidc-username-claim=...\n<\/pre><\/div>\n\n\n
\nkubectl config set-credentials <user_name> \\\n--exec-api-version=client.authentication.k8s.io\/v1beta1 \\\n--exec-command=kubectl \\\n--exec-arg=kubectl-oidc_login \\\n--exec-arg=get-token \\\n--exec-arg=--oidc-issuer-url="https:\/\/sts.windows.net\/...\/" \\\n--exec-arg=--oidc-client-id=... \\\n--exec-arg=--oidc-client-secret=...\n<\/pre><\/div>\n\n\n
\n$ kubectl-oidc_login get-token \\\n--oidc-issuer-url="https:\/\/sts.windows.net\/...\/" \\\n--oidc-client-id=... \\\n--oidc-client-secret="..."\n<\/pre><\/div>\n\n\n
\nerror: could not open the browser: exec: "xdg-open,x-www-browser,www-browser": executable file not found in $PATH\nPlease visit the following URL in your browser manually: http:\/\/localhost:8000\n<\/pre><\/div>\n\n\n
\n{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io\/v1beta1","spec":{"interactive":false},"status":{"expirationTimestamp":"2022-11-25T10:20:43Z","token":"eyJ0eXAiOi..."}}\n<\/pre><\/div>\n\n\n\n$ kubectl get pods\nerror: could not open the browser: exec: "xdg-open,x-www-browser,www-browser": executable file not found in $PATH\n\nPlease visit the following URL in your browser manually: http:\/\/localhost:8000\nerror: get-token: authentication error: authcode-browser error: authentication error: authorization code flow error: oauth2 error: could not exchange the code and token: oauth2: cannot fetch token: 401 Unauthorized\nResponse: {"error":"invalid_client","error_description":"AADSTS7000215: Invalid client secret provided. Ensure the secret being sent in the request is the client secret value, not the client secret ID, for a secret added to app '...'.\\r\\nTrace ID: ...\\r\\nCorrelation ID: ...\\r\\nTimestamp: 2022-11-25 07:46:29Z","error_codes":[7000215],"timestamp":"2022-11-25 07:46:29Z","trace_id":"d...","correlation_id":"...","error_uri":"https:\/\/login.windows.net\/error?code=7000215"}\n<\/pre><\/div>\n\n\nThe future proof method – RBAC<\/h2>\n\n\n\n
\n$ kubectl get pods\nError from server (Forbidden): pods is forbidden: User "<user_name>" cannot list resource "pods" in API group "" in the namespace "default"\n<\/pre><\/div>\n\n\n
\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRoleBinding\nmetadata:\n name: test-sso-clusterrolebinding\nroleRef:\n apiGroup: rbac.authorization.k8s.io\n kind: ClusterRole\n name: test-sso-clusterrole\nsubjects:\n- apiGroup: rbac.authorization.k8s.io\n kind: User\n name: "<user_name>"\n\n---\napiVersion: rbac.authorization.k8s.io\/v1\nkind: ClusterRole\nmetadata:\n name: test-sso-clusterrole\nrules:\n- apiGroups: [""] # "" indicates the core API group\n resources: ["nodes", "pods"]\n verbs: ["get", "watch", "list"]\n<\/pre><\/div>\n\n\n
\n$ kubectl get secrets\nError from server (Forbidden): secrets is forbidden: User "<user_name>" cannot list resource "secrets" in API group "" in the namespace "default"\n\n$ kubectl get po\nNo resources found in default namespace.\n\n$ kubectl get no\nNAME STATUS ROLES AGE VERSION\nnode1 Ready control-plane 9d v1.24.4\nnode2 Ready <none> 9d v1.24.4\nnode3 Ready <none> 9d v1.24.4\nnode4 Ready <none> 9d v1.24.4\nnode5 Ready <none> 9d v1.24.4\nnode6 Ready <none> 9d v1.24.4\n<\/pre><\/div>\n\n\n
\n$ kubectl config get-contexts\n$ kubectl config use-context <context_without_SSO>\n<\/pre><\/div>\n\n\n
Limitations & Conclusion<\/h2>\n\n\n\n
\n - --listen-address=0.0.0.0:8000\n - --oidc-redirect-url-hostname=<http url>\n<\/pre><\/div>\n\n\n
\n$ netstat -anp|grep 8000\ntcp 0 0 0.0.0.0:8000 0.0.0.0:* LISTEN 3457640\/kubectl-oid\n<\/pre><\/div>\n\n\n
\n--grant-type=authcode-keyboard\n<\/pre><\/div>\n\n\n