When working with WebLogic, there will probably be a moment when you will ask yourself: damn, what is wrong? Nowadays, software are so complex that it is kind of easy to introduce bugs\/non-wanted behaviors in them… In this blog, I will present a small thing that just blew my mind when I faced it: I had a fully working WLS on which I executed a WLST script in order to configure the SSL Ciphers for the Admin Server as well as Managed Servers. After this, the WLS wasn\u2019t able to start anymore but this had nothing to do with the WLST script since I was sure it didn\u2019t contain any error and it did what I expected\u2026 To stay generic, to update the Ciphers of all Managed Servers\/Domains of a single server and because we manage more than one hundred WebLogic Servers at this customer, an automated deployment was really necessary.<\/p>\n
On all these WLS, we have a very few of them that contain two domains on the same server (with different IPs\/DNS Aliases of course). To handle this case, we\u00a0had to use a variable that point to the Admin Server in order to connect to this server and execute the WLST script against it. The name of the variable we\u00a0defined is, of course, “ADMIN_URL”. This is the name used by WebLogic in the configuration files to point to the Admin Server. So what better name than this? Well actually, anything would have been better since this is what caused this issue…<\/p>\n
<\/p>\n
So let’s demonstrate this. First, I’m just trying to start the Admin Server without the variable defined:<\/p>\n
[weblogic@weblogic_server_01 ~]$ cd $DOMAIN_HOME\/bin\n[weblogic@weblogic_server_01 bin]$ \n[weblogic@weblogic_server_01 bin]$ .\/startWebLogic.sh\n.\n.\nJAVA Memory arguments: -Xms256m -Xmx512m -XX:MaxPermSize=256m\n.\nCLASSPATH=$ORACLE_HOME\/wlserver\/server\/lib\/jcmFIPS.jar:$ORACLE_HOME\/wlserver\/server\/lib\/sslj.jar:$ORACLE_HOME\/wlserver\/server\/lib\/cryptoj.jar::$JAVA_HOME\/lib\/tools.jar:$ORACLE_HOME\/wlserver\/server\/lib\/weblogic_sp.jar:$ORACLE_HOME\/wlserver\/server\/lib\/weblogic.jar:$ORACLE_HOME\/wlserver\/..\/oracle_common\/modules\/net.sf.antcontrib_1.1.0.0_1-0b3\/lib\/ant-contrib.jar:$ORACLE_HOME\/wlserver\/modules\/features\/oracle.wls.common.nodemanager_2.0.0.0.jar:$ORACLE_HOME\/wlserver\/..\/oracle_common\/modules\/com.oracle.cie.config-wls-online_8.1.0.0.jar:$ORACLE_HOME\/wlserver\/common\/derby\/lib\/derbyclient.jar:$ORACLE_HOME\/wlserver\/common\/derby\/lib\/derby.jar:$ORACLE_HOME\/wlserver\/server\/lib\/xqrl.jar:$DOMAIN_HOME\/lib\/LB.jar:$DOMAIN_HOME\/lib\/LBJNI.jar:\n.\nPATH=$ORACLE_HOME\/wlserver\/server\/bin:$ORACLE_HOME\/wlserver\/..\/oracle_common\/modules\/org.apache.ant_1.9.2\/bin:$JAVA_HOME\/jre\/bin:$JAVA_HOME\/bin:$DOMAIN_HOME\/D2\/lockbox:$DOMAIN_HOME\/D2\/lockbox\/lib\/native\/linux_gcc34_x64:$JAVA_HOME\/bin:\/usr\/lib64\/qt-3.3\/bin:\/usr\/local\/bin:\/bin:\/usr\/bin:\/usr\/local\/sbin:\/usr\/sbin:\/sbin:\/home\/weblogic\/bin\n.\n***************************************************\n* To start WebLogic Server, use a username and *\n* password assigned to an admin-level user. For *\n* server administration, use the WebLogic Server *\n* console at http:\/\/hostname:port\/console *\n***************************************************\nstarting weblogic with Java version:\njava version \"1.8.0_102\"\nJava(TM) SE Runtime Environment (build 1.8.0_102-b14)\nJava HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)\nStarting WLS with line:\n$JAVA_HOME\/bin\/java -server -Xms256m -Xmx512m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=$ORACLE_HOME\/wlserver\/server\/lib\/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Ddomain.home=$DOMAIN_HOME -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.2 -Dweblogic.security.disableNullCipher=true -Djava.security.egd=file:\/\/\/dev\/.\/urandom -Dweblogic.security.allowCryptoJDefaultJCEVerification=true -Dweblogic.nodemanager.ServiceEnabled=true -Djava.endorsed.dirs=$JAVA_HOME\/jre\/lib\/endorsed:$ORACLE_HOME\/wlserver\/..\/oracle_common\/modules\/endorsed -da -Dwls.home=$ORACLE_HOME\/wlserver\/server -Dweblogic.home=$ORACLE_HOME\/wlserver\/server -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true weblogic.Server\nJava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0\n<Aug 5, 2017 1:37:30 PM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>\n<Aug 5, 2017 1:37:30 PM UTC> <Notice> <WebLogicServer> <BEA-000395> <The following extensions directory contents added to the end of the classpath:\n$DOMAIN_HOME\/lib\/LB.jar:$DOMAIN_HOME\/lib\/LBJNI.jar.>\n<Aug 5, 2017 1:37:30 PM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 25.102-b14 from Oracle Corporation.>\n<Aug 5, 2017 1:37:31 PM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.3.0.0 Wed May 21 18:53:34 PDT 2014 1604337 >\n<Aug 5, 2017 1:37:32 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>\n<Aug 5, 2017 1:37:32 PM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>\n<Aug 5, 2017 1:37:32 PM UTC> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>\n<Aug 5, 2017 1:37:33 PM UTC> <Notice> <Log Management> <BEA-170019> <The server log file $DOMAIN_HOME\/servers\/AdminServer\/logs\/AdminServer.log is opened. All server side log events will be written to this file.>\n<Aug 5, 2017 1:37:34 PM UTC> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>\n<Aug 5, 2017 1:37:36 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STANDBY.>\n<Aug 5, 2017 1:37:36 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>\nAug 5, 2017 1:37:36 PM weblogic.wsee.WseeCoreMessages logWseeServiceStarting\nINFO: The Wsee Service is starting\n<Aug 5, 2017 1:37:37 PM UTC> <Warning> <Munger> <BEA-2156227> <The \"META-INF\/application.xml\" deployment descriptor for the \"consoleapp\" module is in DTD format. Overrides from the deployment plan will NOT be applied since only deployment descriptors in XML Schema format are supported.>\n<Aug 5, 2017 1:37:37 PM UTC> <Warning> <Munger> <BEA-2156227> <The \"WEB-INF\/weblogic.xml\" deployment descriptor for the \"consolehelp\" module is in DTD format. Overrides from the deployment plan will NOT be applied since only deployment descriptors in XML Schema format are supported.>\n<Aug 5, 2017 1:37:38 PM UTC> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias mycert from the JKS keystore file $DOMAIN_HOME\/certs\/identity.jks.>\n<Aug 5, 2017 1:37:38 PM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the JKS keystore file $DOMAIN_HOME\/certs\/trust.jks.>\n<Aug 5, 2017 1:37:39 PM UTC> <Notice> <Log Management> <BEA-170027> <The server has successfully established a connection with the Domain level Diagnostic Service.>\n<Aug 5, 2017 1:37:40 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>\n<Aug 5, 2017 1:37:40 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RESUMING.>\n<Aug 5, 2017 1:37:40 PM UTC> <Notice> <Server> <BEA-002613> <Channel \"DefaultSecure\" is now listening on weblogic_server_01:8443 for protocols iiops, t3s, ldaps, https.>\n<Aug 5, 2017 1:37:40 PM UTC> <Notice> <WebLogicServer> <BEA-000329> <Started the WebLogic Server Administration Server \"AdminServer\" for domain \"DOMAIN\" running in production mode.>\n<Aug 5, 2017 1:37:40 PM UTC> <Notice> <WebLogicServer> <BEA-000360> <The server started in RUNNING mode.>\n<Aug 5, 2017 1:37:40 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to RUNNING.>\n\n^C\n<Aug 5, 2017 1:37:55 PM UTC> <Notice> <WebLogicServer> <BEA-000388> <JVM called the WebLogic Server shutdown hook. The server will force shutdown now.>\n<Aug 5, 2017 1:37:55 PM UTC> <Notice> <WebLogicServer> <BEA-000396> <Server shutdown has been requested by <WLS Kernel>.>\n<Aug 5, 2017 1:37:55 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SUSPENDING.>\n<Aug 5, 2017 1:37:55 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to ADMIN.>\n<Aug 5, 2017 1:37:55 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN.>\n<Aug 5, 2017 1:37:55 PM UTC> <Notice> <Server> <BEA-002607> <Channel \"DefaultSecure\", listening on weblogic_server_01:8443, was shut down.>\n[weblogic@weblogic_server_01 bin]$<\/pre>\n<\/p>\n
As you can see above, the Admin Server is starting properly. Once in RUNNING state, I just stopped it (CTRL+C) to continue the demo.<\/p>\n
The next step is to find the current value of “ADMIN_URL”. This variable is used in the files stopWebLogic.sh, stopManagedWebLogic.sh and startManagedWebLogic.sh. To be able to stop these components, the address of the Admin Server is needed. The funny thing is that in two of these three files, it is using the “t3s” protocol and on the third one, it is using “https” (or t3\/http if not in SSL-enabled). Once you have this value, you define your environment variable using this exact same value (manual setup or more automatic setup):<\/p>\n
[weblogic@weblogic_server_01 bin]$ echo $ADMIN_URL\n\n[weblogic@weblogic_server_01 bin]$\n[weblogic@weblogic_server_01 bin]$\n[weblogic@weblogic_server_01 bin]$ grep -E 'ADMIN_URL=\"[th]' *.sh\nstartManagedWebLogic.sh:\tADMIN_URL=\"https:\/\/weblogic_server_01:8443\"\nstopManagedWebLogic.sh:\t\tADMIN_URL=\"t3s:\/\/weblogic_server_01:8443\"\nstopWebLogic.sh:\t\t\tADMIN_URL=\"t3s:\/\/weblogic_server_01:8443\"\n[weblogic@weblogic_server_01 bin]$\n[weblogic@weblogic_server_01 bin]$ export ADMIN_URL=\"https:\/\/weblogic_server_01:8443\"\n[weblogic@weblogic_server_01 bin]$ # or\n[weblogic@weblogic_server_01 bin]$ export ADMIN_URL=`grep -E 'ADMIN_URL=\"[th]' stopWebLogic.sh | sed 's,ADMIN_URL=\"([^\"]*)\",1,'`\n[weblogic@weblogic_server_01 bin]$\n[weblogic@weblogic_server_01 bin]$ echo $ADMIN_URL\nt3s:\/\/weblogic_server_01:8443\n[weblogic@weblogic_server_01 bin]$<\/pre>\n<\/p>\n
At this point, I defined the ADMIN_URL variable using the T3S protocol. We can think that this wouldn’t affect the start of our domain. I mean we are just defining a variable that exist in the shell scripts of WebLogic with the exact same value… But note that this variable isn’t defined in the file “startWebLogic.sh”… Once this is done, we can reproduce the issue. For that, simply try to start the Admin Server again:<\/p>\n
[weblogic@weblogic_server_01 bin]$ .\/startWebLogic.sh\n.\n.\nJAVA Memory arguments: -Xms256m -Xmx512m -XX:MaxPermSize=256m\n.\nCLASSPATH=$ORACLE_HOME\/wlserver\/server\/lib\/jcmFIPS.jar:$ORACLE_HOME\/wlserver\/server\/lib\/sslj.jar:$ORACLE_HOME\/wlserver\/server\/lib\/cryptoj.jar::$JAVA_HOME\/lib\/tools.jar:$ORACLE_HOME\/wlserver\/server\/lib\/weblogic_sp.jar:$ORACLE_HOME\/wlserver\/server\/lib\/weblogic.jar:$ORACLE_HOME\/wlserver\/..\/oracle_common\/modules\/net.sf.antcontrib_1.1.0.0_1-0b3\/lib\/ant-contrib.jar:$ORACLE_HOME\/wlserver\/modules\/features\/oracle.wls.common.nodemanager_2.0.0.0.jar:$ORACLE_HOME\/wlserver\/..\/oracle_common\/modules\/com.oracle.cie.config-wls-online_8.1.0.0.jar:$ORACLE_HOME\/wlserver\/common\/derby\/lib\/derbyclient.jar:$ORACLE_HOME\/wlserver\/common\/derby\/lib\/derby.jar:$ORACLE_HOME\/wlserver\/server\/lib\/xqrl.jar:$DOMAIN_HOME\/lib\/LB.jar:$DOMAIN_HOME\/lib\/LBJNI.jar:\n.\nPATH=$ORACLE_HOME\/wlserver\/server\/bin:$ORACLE_HOME\/wlserver\/..\/oracle_common\/modules\/org.apache.ant_1.9.2\/bin:$JAVA_HOME\/jre\/bin:$JAVA_HOME\/bin:$DOMAIN_HOME\/D2\/lockbox:$DOMAIN_HOME\/D2\/lockbox\/lib\/native\/linux_gcc34_x64:$JAVA_HOME\/bin:\/usr\/lib64\/qt-3.3\/bin:\/usr\/local\/bin:\/bin:\/usr\/bin:\/usr\/local\/sbin:\/usr\/sbin:\/sbin:\/home\/weblogic\/bin\n.\n***************************************************\n* To start WebLogic Server, use a username and *\n* password assigned to an admin-level user. For *\n* server administration, use the WebLogic Server *\n* console at http:\/\/hostname:port\/console *\n***************************************************\nstarting weblogic with Java version:\njava version \"1.8.0_102\"\nJava(TM) SE Runtime Environment (build 1.8.0_102-b14)\nJava HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode)\nStarting WLS with line:\n$JAVA_HOME\/bin\/java -server -Xms256m -Xmx512m -XX:MaxPermSize=256m -Dweblogic.Name=AdminServer -Djava.security.policy=$ORACLE_HOME\/wlserver\/server\/lib\/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Ddomain.home=$DOMAIN_HOME -Dweblogic.nodemanager.ServiceEnabled=true -Dweblogic.security.SSL.minimumProtocolVersion=TLSv1.2 -Dweblogic.security.disableNullCipher=true -Djava.security.egd=file:\/\/\/dev\/.\/urandom -Dweblogic.security.allowCryptoJDefaultJCEVerification=true -Dweblogic.nodemanager.ServiceEnabled=true -Djava.endorsed.dirs=$JAVA_HOME\/jre\/lib\/endorsed:$ORACLE_HOME\/wlserver\/..\/oracle_common\/modules\/endorsed -da -Dwls.home=$ORACLE_HOME\/wlserver\/server -Dweblogic.home=$ORACLE_HOME\/wlserver\/server -Dweblogic.management.server=t3s:\/\/weblogic_server_01:8443 -Dweblogic.utils.cmm.lowertier.ServiceDisabled=true weblogic.Server\nJava HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0\n<Aug 5, 2017 1:40:04 PM UTC> <Info> <Security> <BEA-090906> <Changing the default Random Number Generator in RSA CryptoJ from ECDRBG128 to FIPS186PRNG. To disable this change, specify -Dweblogic.security.allowCryptoJDefaultPRNG=true.>\n<Aug 5, 2017 1:40:04 PM UTC> <Notice> <WebLogicServer> <BEA-000395> <The following extensions directory contents added to the end of the classpath:\n$DOMAIN_HOME\/lib\/LB.jar:$DOMAIN_HOME\/lib\/LBJNI.jar.>\n<Aug 5, 2017 1:40:04 PM UTC> <Info> <WebLogicServer> <BEA-000377> <Starting WebLogic Server with Java HotSpot(TM) 64-Bit Server VM Version 25.102-b14 from Oracle Corporation.>\n<Aug 5, 2017 1:40:05 PM UTC> <Warning> <Security> <BEA-090924> <JSSE has been selected by default, since the SSLMBean is not available.>\n<Aug 5, 2017 1:40:05 PM UTC> <Info> <Security> <BEA-090908> <Using the default WebLogic SSL Hostname Verifier implementation.>\n<Aug 5, 2017 1:40:05 PM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file $ORACLE_HOME\/wlserver\/server\/lib\/DemoTrust.jks.>\n<Aug 5, 2017 1:40:06 PM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file $JAVA_HOME\/jre\/lib\/security\/cacerts.>\n<Aug 5, 2017 1:40:06 PM UTC> <Info> <Management> <BEA-141298> <Could not register with the Administration Server: java.rmi.RemoteException: [Deployer:149147]Could not reach the Administration Server through any of its URLs: \"https:\/\/weblogic_server_01:8443\".>\n<Aug 5, 2017 1:40:06 PM UTC> <Info> <Management> <BEA-141107> <Version: WebLogic Server 12.1.3.0.0 Wed May 21 18:53:34 PDT 2014 1604337 >\n<Aug 5, 2017 1:40:07 PM UTC> <Info> <Security> <BEA-090908> <Using the default WebLogic SSL Hostname Verifier implementation.>\n<Aug 5, 2017 1:40:07 PM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file $ORACLE_HOME\/wlserver\/server\/lib\/DemoTrust.jks.>\n<Aug 5, 2017 1:40:07 PM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the jks keystore file $JAVA_HOME\/jre\/lib\/security\/cacerts.>\n<Aug 5, 2017 1:40:07 PM UTC> <Alert> <Management> <BEA-141151> <The Administration Server could not be reached at https:\/\/weblogic_server_01:8443.>\n<Aug 5, 2017 1:40:07 PM UTC> <Info> <Configuration Management> <BEA-150018> <This server is being started in Managed Server independence mode in the absence of the Administration Server.>\n<Aug 5, 2017 1:40:07 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to STARTING.>\n<Aug 5, 2017 1:40:07 PM UTC> <Info> <WorkManager> <BEA-002900> <Initializing self-tuning thread pool.>\n<Aug 5, 2017 1:40:07 PM UTC> <Info> <WorkManager> <BEA-002942> <CMM memory level becomes 0. Setting standby thread pool size to 256.>\n<Aug 5, 2017 1:40:07 PM UTC> <Notice> <Log Management> <BEA-170019> <The server log file $DOMAIN_HOME\/servers\/AdminServer\/logs\/AdminServer.log is opened. All server side log events will be written to this file.>\n<Aug 5, 2017 1:40:09 PM UTC> <Notice> <Security> <BEA-090082> <Security initializing using security realm myrealm.>\n<Aug 5, 2017 1:40:10 PM UTC> <Error> <Configuration Management> <BEA-150000> <An error occurred while establishing a connection back to the Adminstration Server t3s:\/\/weblogic_server_01:8443 during startup. Since bootstrap succeeded, check that t3s:\/\/weblogic_server_01:8443 uniquely identifies the Administration Server.\njavax.naming.ConfigurationException: Cannot register for disconnect events on local server\n at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:234)\n at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:190)\n at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:54)\n at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:188)\n at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:183)\n Truncated. see log file for complete stacktrace\n>\n<Aug 5, 2017 1:40:10 PM UTC> <Notice> <Security> <BEA-090171> <Loading the identity certificate and private key stored under the alias myacert from the JKS keystore file $DOMAIN_HOME\/certs\/identity.jks.>\n<Aug 5, 2017 1:40:10 PM UTC> <Notice> <Security> <BEA-090169> <Loading trusted certificates from the JKS keystore file $DOMAIN_HOME\/certs\/trust.jks.>\n<Aug 5, 2017 1:40:12 PM UTC> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:\n\n There are 1 nested errors:\n\njavax.naming.ConfigurationException: Cannot register for disconnect events on local server\n at weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:234)\n at weblogic.server.channels.RemoteChannelServiceImpl.registerForever(RemoteChannelServiceImpl.java:190)\n at weblogic.protocol.ConnectMonitorFactory.registerForever(ConnectMonitorFactory.java:54)\n at weblogic.management.provider.MSIService.registerForReconnectToAdminServer(MSIService.java:188)\n at weblogic.management.mbeanservers.compatibility.internal.CompatibilityMBeanServerService.start(CompatibilityMBeanServerService.java:183)\n at weblogic.server.AbstractServerService.postConstruct(AbstractServerService.java:78)\n at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)\n at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n at java.lang.reflect.Method.invoke(Method.java:498)\n at org.glassfish.hk2.utilities.reflection.ReflectionHelper.invoke(ReflectionHelper.java:1017)\n at org.jvnet.hk2.internal.ClazzCreator.postConstructMe(ClazzCreator.java:388)\n at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:430)\n at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:456)\n at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:225)\n at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:82)\n at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2488)\n at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:98)\n at org.jvnet.hk2.internal.ServiceHandleImpl.getService(ServiceHandleImpl.java:87)\n at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.oneJob(CurrentTaskFuture.java:1162)\n at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$QueueRunner.run(CurrentTaskFuture.java:1147)\n at org.glassfish.hk2.runlevel.internal.CurrentTaskFuture$UpOneLevel.run(CurrentTaskFuture.java:753)\n at weblogic.work.SelfTuningWorkManagerImpl$WorkAdapterImpl.run(SelfTuningWorkManagerImpl.java:553)\n at weblogic.work.ExecuteThread.execute(ExecuteThread.java:311)\n at weblogic.work.ExecuteThread.run(ExecuteThread.java:263)\n\n>\n<Aug 5, 2017 1:40:12 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED.>\n<Aug 5, 2017 1:40:12 PM UTC> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down.>\n<Aug 5, 2017 1:40:12 PM UTC> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN.>\n[weblogic@weblogic_server_01 bin]$<\/pre>\n<\/p>\n
So what is happening exactly that can cause this simple variable definition to prevent you to start your Admin Server? If you take a look at the script “startWebLogic.sh”, you will not see any use of this variable so you should be good, right? Well not really because this script is actually loading its environment by using the well-known “setDomainEnv.sh” (in the same folder). The interesting part is in this second file… If you are checking the usage of “ADMIN_URL” in this setDomainEnv, you will see an “if-then-else” section:<\/p>\n
[weblogic@weblogic_server_01 bin]$ grep -C2 \"ADMIN_URL\" setDomainEnv.sh\n# Clustering support (edit for your cluster!)\n\nif [ \"${ADMIN_URL}\" = \"\" ] ; then\n # The then part of this block is telling us we are either starting an admin server OR we are non-clustered\n CLUSTER_PROPERTIES=\"\"\n export CLUSTER_PROPERTIES\nelse\n CLUSTER_PROPERTIES=\"-Dweblogic.management.server=${ADMIN_URL}\"\n export CLUSTER_PROPERTIES\nfi\n[weblogic@weblogic_server_01 bin]$<\/pre>\n<\/p>\n
With this small portion of code, you can actually understand the issue:<\/p>\n
<\/p>\n
Therefore when defining the ADMIN_URL in the environment, we are actually – unintentionally – saying to WebLogic that this is a cluster and that the Management Server can be found at $ADMIN_URL… But this URL is the local Admin Server which we are trying to start. Thus the failure to start…<\/p>\n
If you carefully read the logs above, you could actually saw this small difference… When reproducing the error, if you take a look at the end of the line 21, you will see that the JVM parameter “-Dweblogic.management.server=t3s:\/\/weblogic_server_01:8443” is present and that’s the reason of this issue.<\/p>\n
<\/p>\n","protected":false},"excerpt":{"rendered":"
When working with WebLogic, there will probably be a moment when you will ask yourself: damn, what is wrong? Nowadays, software are so complex that it is kind of easy to introduce bugs\/non-wanted behaviors in them… In this blog, I will present a small thing that just blew my mind when I faced it: I […]<\/p>\n","protected":false},"author":20,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[197],"tags":[60,1163,1164,38,96,723],"type_dbi":[],"class_list":["post-10436","post","type-post","status-publish","format-standard","hentry","category-application-integration-middleware","tag-12c","tag-admin-server","tag-admin_url","tag-cluster","tag-oracle","tag-weblogic-server"],"acf":[],"yoast_head":"\n