After creating a new WebLogic Domain that is having multiple WebLogic Servers on multiple hosts, remote WebLogic Servers do not start with the following error reported

<Nov 18, 2022 11:20:34,091 AM CET> <Critical> <WebLogicServer> <BEA-000386> <Server subsystem failed. Reason: A MultiException has 2 exceptions.  They are:
1. java.lang.AssertionError: java.lang.reflect.InvocationTargetException
2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.server.channels.ChannelService

A MultiException has 2 exceptions.  They are:
1. java.lang.AssertionError: java.lang.reflect.InvocationTargetException
2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.server.channels.ChannelService

        at org.jvnet.hk2.internal.ClazzCreator.create(ClazzCreator.java:392)
        at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:487)
        at org.glassfish.hk2.runlevel.internal.AsyncRunLevelContext.findOrCreate(AsyncRunLevelContext.java:305)
        at org.glassfish.hk2.runlevel.RunLevelContext.findOrCreate(RunLevelContext.java:85)
        at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2126)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.AssertionError: java.lang.reflect.InvocationTargetException
        at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:180)
        at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy.decrypt(DescriptorManager.java:197)
        at weblogic.descriptor.DescriptorManager$SecurityServiceImpl.decrypt(DescriptorManager.java:119)
        at weblogic.descriptor.internal.AbstractDescriptorBean._decrypt(AbstractDescriptorBean.java:1314)
        at weblogic.management.configuration.SecurityConfigurationMBeanImpl.getCredential(SecurityConfigurationMBeanImpl.java:1052)
        Truncated. see log file for complete stacktrace
Caused By: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at weblogic.descriptor.DescriptorManager$SecurityServiceImpl$SecurityProxy._invokeServiceMethod(DescriptorManager.java:178)
        Truncated. see log file for complete stacktrace
Caused By: weblogic.security.internal.encryption.EncryptionServiceException: com.rsa.jsafe.JSAFE_PaddingException: Invalid padding.
        at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptBytes(JSafeEncryptionServiceImpl.java:144)
        at weblogic.security.internal.encryption.JSafeEncryptionServiceImpl.decryptString(JSafeEncryptionServiceImpl.java:192)
        at weblogic.security.internal.encryption.ClearOrEncryptedService.decrypt(ClearOrEncryptedService.java:99)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

The highlighted line above shows that the error is related to some encryption. The only encryption done is for the password. To create the WebLogic Domain, I’m using a WLST script that after creating the domain, generates a boot.properties file for the Administration Server as shown below:

def createBootFile(domainPathDir,directoryName, fileName, UserName, Password):
    secDirectory = java.io.File(directoryName);
    secFile = java.io.File(directoryName + '/' + fileName);

    writer = None;
    try:
        secDirectory.mkdirs();
        secFile.createNewFile();
        secUserName = encrypt(UserName,domainPathDir);
        secPassword =  encrypt(Password,domainPathDir);
        writer = java.io.FileWriter(secFile);
        writer.write("username=" +  secUserName);
        writer.write("\npassword=" +  secPassword);
    finally:
        try:
            if writer != None:
                writer.flush();
                writer.close();
        except java.io.IOException, e:
            e.printStackTrace();
            exit()

The boot.properties file is generated and written in the $DOMAIN_HOME/servers/AdminServer/security folder. The username and password are directly encrypted before writing the file. This is the reason for the remote WebLogic Servers not to start after installing them using unpack.

Note that the Admin Server and the WebLogic Servers located on the first host are starting and running fine without any failures or warning.

If the WebLogic Domain is having JDBC connections created using the same python script, the following exception will display during the unpack.sh run:

<< read template from "/u01/app/weblogic/local/etc/myDomain/myDomain.jar"
>>  succeed: read template from "/u01/app/weblogic/local/etc/myDomain/myDomain.jar"
<< set config option DomainName to "myDomain"
>>  succeed: set config option DomainName to "myDomain"
>>  validateConfig "KeyStorePasswords"
>>  succeed: validateConfig "KeyStorePasswords"
<< write Domain to "/u02/config/domains/myDomain"
>> warning:write Domain to "/u02/config/domains/myDomain"
>> 40318: Invalid or missing JDBC datasource connection parameters.
40318: The JDBC datasource required parameters are invalid or missing.
40318: Correct the datasource connection parameters.
..................................................
>>  succeed: write Domain to "/u02/config/domains/myDomain"
<< close template
>>  succeed: close template

Checking the JDBC description file in the WebLogic domain files on the remote host, it appears that the line containing the encrypted password is missing.

A possible workaround is to browse to the WebLogic domain console and do some changes in the domain. After those changes, rerun the pack.sh and unpack.sh commands and this time, the remote WebLogic Servers will start.

Even if the workaround above might avoid you to drop the domain and recreate it new, it’s better to not encrypt the username and password in the Admin Server boot.properties using WLST at WebLogic Domain creation time. Keep it is clear text, it will be encrypted at first start.

Post Views: 3,720
Thumbnail [60x60]
by
Pascal Brand