I had an interesting case at a customer where the Single Sign On (using SAML2) on a WebLogic Server stopped working after the renewal of the WebLogic SSL Certificate. The WebLogic Server itself and its applications were still accessible (by bypassing the SSO) and working fine but not the SSO itself. That’s definitively a strange behavior and not something I would expect to see broken after just changing a certificate that has, apparently, no link to it at all but that’s what happened.
What I mean by the SSO stopped working is that WebLogic returned its usual 403 Forbidden (e.g.: WebLogic – SSO/Atn/Atz – 403 Forbidden, a first issue or WebLogic – SSO/Atn/Atz – 403 Forbidden, another issue). As always, the first step to try to debug what is really happening is to enable the SAML2/Atn/Atz logging (WebLogic – SSO/Atn/Atz – How to enable debug logs) since it usually gives good insight into the real issue and it was again the case:
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <SAML2Filter: Processing request on URI '/D2/X3_Portal.jsp'>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <getServiceTypeFromURI(): request URI is '/D2/X3_Portal.jsp'>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <getServiceTypeFromURI(): request URI is not a service URI>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <getServiceTypeFromURI(): returning service type 'SPinitiator'>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <SP initiating authn request: processing>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <SP initiating authn request: partner id is null>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <SP initiating authn request: use partner binding HTTP/Redirect>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <signature algorithm of saml object: null>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <URL encoded saml message:fZBRa4MwF...Rd%2F%2FgU%3D>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <URL encoded relay state:null>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <QueryString without signature_SAMLRequest=fZBRa4MwF...Rd%2F%2FgU%3D>
####<Jan 24, 2022 7:04:57> <SecuritySAML2Service> <BEA-000000> <URL:https://sso.domain.net/oamfed/idp/samlv20?SAMLRequest=fZBRa4MwF...Rd%2F%2FgU%3D>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <SAML2Servlet: Initialized logger service>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <SAML2Servlet: Initialized SAML2 service>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <SAML2Servlet: setConfigKey called with key 'default'>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <SAML2Servlet: Processing request on URI '/saml2/sp/acs/post'>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <getServiceTypeFromURI(): request URI is '/saml2/sp/acs/post'>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <getServiceTypeFromURI(): service URI is '/sp/acs/post'>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <getServiceTypeFromURI(): returning service type 'ACS'>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <Assertion consumer service: processing>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <get SAMLResponse from http request:PHNhbWxwO...
...
...
...bnNlPg==
>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <BASE64 decoded saml message:<samlp:Response xmlns_samlp="urn:oasis:names:tc:SAML:2.0:protocol" ...
...
...
...</saml:Assertion></samlp:Response>>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Service> <BEA-000000> <<samlp:Response> is signed.>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.IdentityAssertionServiceImpl.assertIdentity(SAML2.Assertion.DOM)>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.IdentityAssertionTokenServiceImpl.assertIdentity(SAML2.Assertion.DOM)>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Atn> <BEA-000000> <SAML2IdentityAsserterProvider: start assert SAML2 token>
...
...
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.LoginModuleWrapper.initialize ClassLoader=java.net.URLClassLoader@5dec31be>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.LoginModuleWrapper.initialize created delegate login module>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <LDAP ATN LoginModule initialized>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.LoginModuleWrapper.initialize delegated>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.LoginModuleWrapper.login>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <LDAP Atn Login>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.CallbackHandlerWrapper.handle>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.CallbackHandlerWrapper.handle callbcacks[0] will be delegated>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.CallbackHandlerWrapper.handle callbcacks[0] will use NameCallback to retrieve name>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.CallbackHandlerWrapper.handle will delegate all callbacks>
####<Jan 24, 2022 7:04:58> <SecuritySAML2Atn> <BEA-000000> <SAMLIACallbackHandler: callback[0]: NameCallback: setName(PatouMorgan01)>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.CallbackHandlerWrapper.handle delegated callbacks>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.CallbackHandlerWrapper.handle got username from callbacks[0], UserName=PatouMorgan01>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <LDAP Atn Login username: PatouMorgan01>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <getUserDNName? user:PatouMorgan01>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <getDNForUser search("ou=people,ou=intranet,dc=company,dc=com", "(&(uid=PatouMorgan01)(objectclass=companyperson))", base DN & below)>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <new LDAP connection to host ldaps.domain.net port 636 use local connection is false>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <created new LDAP connection LDAPConnection { ldapVersion:2 bindDN:""}>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <Connecting to host=ldaps.domain.net, ssl port=636>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <Successfully connected to host=ldaps.domain.net, ssl port=636>
####<Jan 24, 2022 7:04:58> <Security> <BEA-099117> <The LDAP authentication provider named "LDAPSAuthenticator" failed to make connection to ldap server at ldaps://ldaps.domain.net:636, the error cause is: Fatal Alert received: Certificate Unknown.>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <connection failed netscape.ldap.LDAPException: Fatal Alert received: Certificate Unknown (91); Cannot connect to the LDAP server>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <weblogic.security.providers.authentication.LoginServerUnavailableException: Fatal Alert received: Certificate Unknown
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.handleLDAPAtnDelegateException(LDAPAtnLoginModuleImpl.java:679)
at weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl.login(LDAPAtnLoginModuleImpl.java:219)
at com.bea.common.security.internal.service.LoginModuleWrapper$1.run(LoginModuleWrapper.java:110)
at java.security.AccessController.doPrivileged(Native Method)
at com.bea.common.security.internal.service.LoginModuleWrapper.login(LoginModuleWrapper.java:106)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
...
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1632)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:256)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)
>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.LoginModuleWrapper.initialize LoginModuleClassName=weblogic.security.providers.authentication.LDAPAtnLoginModuleImpl>
####<Jan 24, 2022 7:04:58> <SecurityAtn> <BEA-000000> <com.bea.common.security.internal.service.LoginModuleWrapper.initialize ClassLoader=java.net.URLClassLoader@5dec31be>
...
As you can see above, the SSO itself starts properly, the SAML2 exchange is done successfully but then WebLogic tries to check if the SAML2 user is valid by trying to check the user details inside the LDAP but that fails with “Fatal Alert received: Certificate Unknown“. Because WebLogic isn’t able to contact the LDAP, it checks the local users but, in this case, all users are on the LDAP so the SSO is rejected, and the 403 Forbidden page is displayed.
As part of the update process for the WebLogic SSL Certificate, the following is done:
- Create a new SSL Certificate (JKS) for WebLogic
- Create a new Trust Store containing the needed CA and the new WebLogic specific SSL Certificate
- Update the Java cacerts with the new WebLogic specific SSL Certificate
- Update the WebLogic cacerts with the new WebLogic specific SSL Certificate
All these steps were verified and there were no big differences before and after the SSL Certificate renewal. The only small one I could see was regarding the SSL Certificate itself and more specifically regarding the “ExtendedKeyUsages” of it. On the old/previous SSL Certificate, this property contained two options: serverAuth and clientAuth. However, the new SSL Certificate only contained the serverAuth and not the clientAuth:
[weblogic@wls01 certs]$ keytool -v -list -keystore new/identity.jks
Enter keystore password:
Keystore type: jks
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: wlscert
Creation date: Jan 24, 2022
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
Owner: CN=wls01.domain.net, OU=it, O=company, L=city, ST=state, C=CH
Issuer: CN=Int CA
Serial number: ...9773
Valid from: Mon Jan 24 06:30:09 UTC 2022 until: Tue Jan 24 06:30:09 UTC 2023
Certificate fingerprints:
MD5: ...
SHA1: ...
SHA256: ...
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:
...
#7: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
serverAuth
]
#8: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_Encipherment
]
...
Certificate[2]:
Owner: CN=Int CA
Issuer: CN=Root CA
...
Certificate[3]:
Owner: CN=Root CA
Issuer: CN=Root CA
...
*******************************************
*******************************************
[weblogic@wls01 certs]$
[weblogic@wls01 certs]$
[weblogic@wls01 certs]$ keytool -v -list -keystore old/identity.jks
Enter keystore password:
Keystore type: jks
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: wlscert
Creation date: Jan 29, 2020
Entry type: PrivateKeyEntry
Certificate chain length: 3
Certificate[1]:
Owner: CN=wls01.domain.net, OU=it, O=company, L=city, ST=state, C=CH
Issuer: CN=Int CA
Serial number: ...3e41
Valid from: Wed Jan 29 08:27:25 UTC 2020 until: Fri Jan 28 08:27:25 UTC 2022
Certificate fingerprints:
MD5: ...
SHA1: ...
SHA256: ...
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:
...
#7: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
serverAuth
]
#8: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_Encipherment
]
...
Certificate[2]:
Owner: CN=Int CA
Issuer: CN=Root CA
...
Certificate[3]:
Owner: CN=Root CA
Issuer: CN=Root CA
...
*******************************************
*******************************************
[weblogic@wls01 certs]$
After some digging, it appeared that the process of generating the SSL Certificate, which is managed by another team at this customer, changed for security reasons. Therefore, newly generated SSL Certificates do not have the clientAuth anymore, which means that this SSL Certificate cannot be used as a “client” certificate. It is understandable that they would want to remove that by default since it could be used by malicious people, but it also means that it would break any 2-way-SSL or similar behavior where SSL Certificates of a “client” and “server” are exchanged.
In this case, the WebLogic Server is acting as the “client” of the LDAP Server (which must be requesting the client certificate I assume) and the communication isn’t proceeding because the SSL Certificate of the WebLogic Server doesn’t contain the clientAuth. Therefore, the only solution is to re-generate an SSL Certificate with the clientAuth this time. After doing so, the communication with the LDAPS was back and the SSO as well. If you ever face a similar situation, try to compare the previous JKS with the new one using the keytool utility for example and make sure there are no differences between them!
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/OLS_web-min-scaled.jpg)