In a previous blog (click here), I described how to setup the DSearch and IndexAgent(s) in HTTPS using the Groovy script provided by EMC in newer versions of Documentum. This script is quite cool because it will allow you to automatically do some stuff like updating the xml configuration files, put the java keystore in the right location and configure JBoss to use it, aso… It also allows you to quickly apply/revert changes to do some tests for example. Several months ago when I first used this script, I faced some issues. I will present in this blog some of the errors I’ve seen and what was the reason behind that.

 

Actually I have a “funny” story related to these errors: several months after seeing these errors for the first time (and telling EMC about it with complete description of the issue and according solution), I worked on another new issue related to the Full Text Server with the help of EMC. Our EMC contact was trying to replicate our new issue and for that purpose, he had to setup the IndexAgent in SSL and he faced an error with the SSL setup… So he opened a ticket with the EMC Engineering Team to ask them to check why this was happening. In the meantime, I also asked him to share the error he was facing with me because I was curious… Of course, it turned out to be one of the errors I saw and shared with EMC several months before that. Therefore I provided him the solution to correct this bug and he was able to install the IndexAgent in SSL. I found that funny how sometimes you can actually be the one helping EMC with their own products ^^.

 

So let’s start with some of the errors I saw:

1. While configuring a DSearch in HTTPS

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IS 
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" 
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" 
    -isname PrimaryDsearch

Component: IS
====== Configure Index Server: PrimaryDsearch ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/my.keystore
Copy: /app/xPlore/config/indexserverconfig.xml to: /app/xPlore/config/indexserverconfig.xml.bakHttp
Primary instance: PrimaryDsearch
Host name: xplore_server_01
Http port: 9300, https port: 9302
Updated: /app/xPlore/config/indexserverconfig.xml
Copy: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_PrimaryDsearch/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IndexOutOfBoundsException: No group 6
        at java.util.regex.Matcher.start(Matcher.java:374)
        at java.util.regex.Matcher.appendReplacement(Matcher.java:831)
        at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
        at java.lang.String.replaceFirst(String.java:2119)
        at java_lang_String$replaceFirst.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at ConfigSSL.configIS(ConfigSSL.groovy:624)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:361)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.callCurrent(PogoMetaClassSite.java:66)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at ConfigSSL$_run_closure1.doCall(ConfigSSL.groovy:333)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
        at org.codehaus.groovy.runtime.metaclass.ClosureMetaClass.invokeMethod(ClosureMetaClass.java:272)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:877)
        at groovy.lang.Closure.call(Closure.java:412)
        at groovy.lang.Closure.call(Closure.java:425)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1376)
        at org.codehaus.groovy.runtime.DefaultGroovyMethods.each(DefaultGroovyMethods.java:1348)
        at org.codehaus.groovy.runtime.dgm$162.invoke(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite$PojoMetaMethodSiteNoUnwrapNoCoerce.invoke(PojoMetaMethodSite.java:271)
        at org.codehaus.groovy.runtime.callsite.PojoMetaMethodSite.call(PojoMetaMethodSite.java:53)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
        at ConfigSSL.run(ConfigSSL.groovy:331)
        at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
        at groovy.lang.GroovyShell.run(GroovyShell.java:229)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

 

2. While configuring an IndexAgent in HTTPS

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA 
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" 
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" 
    -ianame Indexagent -iaport 9200

Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IndexOutOfBoundsException: No group 6
        at java.util.regex.Matcher.start(Matcher.java:374)
        at java.util.regex.Matcher.appendReplacement(Matcher.java:831)
        at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
        at java.lang.String.replaceFirst(String.java:2119)
        at java_lang_String$replaceFirst.call(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
        at ConfigSSL.configIA(ConfigSSL.groovy:888)
        at ConfigSSL$configIA.callCurrent(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
        at ConfigSSL.run(ConfigSSL.groovy:338)
        at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
        at groovy.lang.GroovyShell.run(GroovyShell.java:229)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
        at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

 

3. While configuring an IndexAgent in HTTPS: another one

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA 
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" 
    -storepass Test4Pass+word$ -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" 
    -ianame Indexagent -iaport 9200
    
Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Exception in thread "main" java.lang.IllegalArgumentException: Illegal group reference
      at java.util.regex.Matcher.appendReplacement(Matcher.java:808)
      at java.util.regex.Matcher.replaceFirst(Matcher.java:955)
      at java.lang.String.replaceFirst(String.java:2119)
      at java_lang_String$replaceFirst.call(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:120)
      at ConfigSSL.configIA(ConfigSSL.groovy:888)
      at ConfigSSL$configIA.callCurrent(Unknown Source)
      at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
      at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
      at ConfigSSL.run(ConfigSSL.groovy:338)
      at groovy.lang.GroovyShell.runScriptOrMainOrTestOrRunnable(GroovyShell.java:266)
      at groovy.lang.GroovyShell.run(GroovyShell.java:229)
      at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.callEntrance(DSearchAdminCLI.java:83)
      at com.emc.documentum.core.fulltext.client.admin.cli.DSearchAdminCLI.main(DSearchAdminCLI.java:22)

 

4. The solution?

You might have noticed above that each time the exception is thrown just after the backup of the file standalone.xml. So what is the Groovy script doing in this file exactly which can cause these errors? Well when configuring JBoss in HTTPS, there is only one thing to do in this file: add a new connector to enable the HTTPS and to point to the correct keystore/password. Based on this information, it is really easy to find the reason for all these exceptions. In addition to that, you might also have noticed that in the two first points above, the password of the keystore was “$JKS_Password” while in the last command, the password is in clear text (Test4Pass+word$). So what is that environment variable $JKS_Password? Well we had to use IQs to install all these environments with logging tools to review and validate the installation and since these logging tools are storing everything displayed to the console, we were using environment variable to store our password so it is not displayed on the console or on the history of the “xplore” user. This is done as follow:

[xplore@xplore_server_01 ~]$ stty -echo; read JKS_Password; stty echo
                                                         => Enter the password here, it will be invisible
[xplore@xplore_server_01 ~]$
[xplore@xplore_server_01 ~]$ echo $JKS_Password
Test4Pass+word$
[xplore@xplore_server_01 ~]$

 

Basically the first command will ask you to type the password and when you press Enter, the prompt is returned and the environment variable $JKS_Password contains your password.

 

Now that this is clear, what is the solution to correct the errors mentioned above? Well the solution is the same for all errors above and this solution is quite simple: don’t use any annoying special characters in your passwords (like “$” or “”). The problem with these characters is that the Groovy script isn’t able to manage them at the moment and therefore creating a password that contains one of these characters might result is errors and/or strange results as shown above.

 

Even if annoying special characters should be avoided to prevent these errors, you can continue to use some other special characters if you want to use a strong password like “+”, “-“, “?” and you can also continue to use the environment variable if you want. This is an example of the command that is working:

[xplore@xplore_server_01 ~]$ echo $JKS_Password
Test4Pass+word
[xplore@xplore_server_01 ~]$ 
[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA 
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" 
    -storepass $JKS_Password -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" 
    -ianame Indexagent -iaport 9200

Component: IA
====== Configure Index Agent: Indexagent ======
Copy: /app/xPlore/jboss7.1.1/certs/xplore_server_01.jks to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/my.keystore
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml.bakHttp
Updated: ./../../jboss7.1.1/server/DctmServer_Indexagent/configuration/standalone.xml
Copy: ./../../watchdog/config/dsearch-watchdog-config.xml to: ./../../watchdog/config/dsearch-watchdog-config.xml.bakHttp
Updated: ./../../watchdog/config/dsearch-watchdog-config.xml
Copy: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml to: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml.bakHttp
Updated: ./../../jboss7.1.1/server/DctmServer_Indexagent/deployments/IndexAgent.war/WEB-INF/classes/indexagent.xml
SSL is enabled, clear jboss cache and start server to verify result.

 

Once this execution completed successfully, you can just continue the SSL setup as described in this blog linked at the top and you shouldn’t face any other issue.

 

As soon as you put a “$” in your password and use this password in the command line (or use the environment variable), you will in the end face an error that will depend on whether or not you used the environment variable to hide it. As an additional note, when I was working on these errors, I did a lot of tests and I already saw the script working (or so it seemed) even with a password containing a “$” (depending on how it is written, with single quotes, double quotes, escaped, aso…) when put directly in the command line. But when I tried to start the IndexAgent, it wasn’t able to start… Therefore I took a look at the file standalone.xml and it was corrupted! The Groovy script screwed up my standalone.xml file and I had to revert a backup to be able to restart it. Hopefully the script is smart enough to backup all files before doing anything…

 

In case you absolutely want a “$” in your password, then it is possible but you will have to do one step manually. First you need to execute the script with a DUMMY password without any special character:

[xplore@xplore_server_01 admin]$ ./xplore.sh -f scripts/ConfigSSL.groovy -enable -component IA 
    -alias ft_alias -keystore "/app/xPlore/jboss7.1.1/certs/xplore_server_01.jks" 
    -storepass DUMMY_PASSWORD -indexserverconfig "/app/xPlore/config/indexserverconfig.xml" 
    -ianame Indexagent -iaport 9200
...

 

Once done, you can simply open the file standalone.xml and replace “DUMMY_PASSWORD” with the real password that contains a “$”. Restart the IndexAgent and this time, it should work. 🙂