During an upgrade project of xPlore from 16.4 to 20.2 at a customer, I faced a quite interesting issue with the xPlore CPS (Content Processing Services) that couldn’t start properly and were, therefore, shown as “UNREACHABLE“. The funny thing is that the Java/WildFly processes of the PrimaryDsearch and CPS were properly running, and it was possible to login to the Dsearch Admin interface, but the CPS part wouldn’t work at all:

Dsearch Admin UI - Content Processing Service

This environment had 2 xPlore Federations, each with 1 PrimaryDsearch + 2 CPS. The first Federation was working properly post upgrade but not the second one. On the upgrade logs, there was no errors. Everything appeared to be successful during upgrade for both Federations and the logs were the same on both side but somehow, one of the two couldn’t start its CPS anymore.

At runtime, the error on the CPS (PrimaryDsearch + CPS Only) was the following:

2022-12-09 06:26:28,967 ERROR [Daemon0(2622)-Core-(140492346279744)] Err occured in cpsPluginLibrary::Open $XPLORE_HOME/dsearch/cps/cps_daemon/bin/liblinguistic_processor.so with dlopen, err-msg: libbtrlpcore.so.7.17: cannot open shared object file: No such file or directory.
2022-12-09 06:26:28,967 ERROR [Daemon0(2622)-Core-(140492346279744)] bin/liblinguistic_processor.so loading failed
2022-12-09 06:26:28,967 FATAL [Daemon0(2622)-Core-(140492346279744)] Err occurred while initializing LP manager

On the Dsearch side, there was also errors of course, when checking if CPS are available and when executing Searches:

2022-12-09 06:04:42,008 DEBUG [Search-Thread-10] c.e.d.core.fulltext.indexserver.cps.CPSSubmitter - Begin to connect to CPS at (local)  with connection 6
2022-12-09 06:04:42,851 DEBUG [Search-Thread-10] c.e.d.core.fulltext.indexserver.cps.CPSSubmitter - Begin to connect to CPS at (https://cps2-1_hostname:9302/cps/ContentProcessingService?wsdl)  with connection 6
2022-12-09 06:04:43,733 DEBUG [Search-Thread-10] c.e.d.core.fulltext.indexserver.cps.CPSSubmitter - Begin to connect to CPS at (https://cps2-2_hostname:9302/cps/ContentProcessingService?wsdl)  with connection 6
2022-12-09 06:04:44,606 WARN [Search-Thread-10] c.e.d.core.fulltext.indexserver.cps.CPSRouter - NO CPS available
2022-12-09 06:04:44,606 WARN [Search-Thread-10] c.e.d.c.f.i.core.index.xhive.ESSBaseAnalyzer - Failed to detect the language for string(mySearch) - in query (PrimaryDsearch$1c7c173b-4485-70ce-9418-1bcf652fd4be)
com.emc.documentum.core.fulltext.common.exception.CPSUnavailableException: CPS_ERR_CONNECT
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.detectLanguage(CPSRouter.java:237)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.detectLanguage(ESSBaseAnalyzer.java:560)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.rlpTokenStreams(ESSBaseAnalyzer.java:239)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.tokenStreams(ESSBaseAnalyzer.java:201)
	...
...
2022-12-09 06:04:47,202 WARN [Search-Thread-10] c.e.d.core.fulltext.indexserver.cps.CPSRouter - NO CPS available
2022-12-09 06:04:47,203 WARN [Search-Thread-10] c.e.d.c.fulltext.indexserver.search.SearchSession - Unknown exception ocurred during query PrimaryDsearch$1c7c173b-4485-70ce-9418-1bcf652fd4be
java.lang.RuntimeException: com.emc.documentum.core.fulltext.common.exception.CPSUnavailableException: CPS_ERR_CONNECT
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.rlpTokenStreams(ESSBaseAnalyzer.java:256)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.tokenStreams(ESSBaseAnalyzer.java:201)
	...
Caused by: com.emc.documentum.core.fulltext.common.exception.CPSUnavailableException: CPS_ERR_CONNECT
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.sendRequestsWithRetry(CPSRouter.java:397)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.submit(CPSRouter.java:167)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.createCPSTokenStream(ESSBaseAnalyzer.java:284)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.rlpTokenStreams(ESSBaseAnalyzer.java:252)
	... 51 common frames omitted
2022-12-09 06:04:47,212 ERROR [RMI TCP Connection(2)-172.20.132.153] c.e.d.c.f.i.admin.mbean.ESSAdminSearchManagement - Query fail for domain REPO with query ( let $j:= for $i score $s in /dmftdoc[. ftcontains 'mySearch' with stemming using stop words default] order by $s descending return <d> {$i/dmftmetadata//r_object_id}  { $i/dmftmetadata//object_name } { $i/dmftmetadata//r_modifier }</d> return (for $k in subsequence($j,1,200) return $k) )
com.emc.documentum.core.fulltext.common.exception.FtSearchException: java.lang.RuntimeException: com.emc.documentum.core.fulltext.common.exception.CPSUnavailableException: CPS_ERR_CONNECT
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.search.SearchSession.handleException(SearchSession.java:978)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.search.threads.QueryXHive.call(QueryXHive.java:292)
	...
Caused by: java.lang.RuntimeException: com.emc.documentum.core.fulltext.common.exception.CPSUnavailableException: CPS_ERR_CONNECT
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.rlpTokenStreams(ESSBaseAnalyzer.java:256)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.tokenStreams(ESSBaseAnalyzer.java:201)
	...
Caused by: com.emc.documentum.core.fulltext.common.exception.CPSUnavailableException: CPS_ERR_CONNECT
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.sendRequestsWithRetry(CPSRouter.java:397)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.cps.CPSRouter.submit(CPSRouter.java:167)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.createCPSTokenStream(ESSBaseAnalyzer.java:284)
	at deployment.dsearch.war//com.emc.documentum.core.fulltext.indexserver.core.index.xhive.ESSBaseAnalyzer.rlpTokenStreams(ESSBaseAnalyzer.java:252)
	... 51 common frames omitted

Debug logs were enabled on both the Dsearch Admin UI as well as inside the logback.xml file. Unfortunately, it didn’t give more information on the error, it only added messages when it was trying to contact the different CPS (as you can see above) but nothing more related to why the CPS failed to start. Since the CPS startup error included a shared object library (“liblinguistic_processor.so with dlopen, err-msg: libbtrlpcore.so.7.17: cannot open shared object file: No such file or directory“), I tried to look into that:

[xplore@ds2-0 ~]$ ldd $XPLORE_HOME/dsearch/cps/cps_daemon/bin/liblinguistic_processor.so
        linux-vdso.so.1 =>  (0x00007ffdc2fc8000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6a3703f000)
        libbtrlpcore.so.7.17 => not found
        libbtutils.so.7.17 => not found
        libbtunicode.so => not found
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f6a36d37000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f6a36a35000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6a3681f000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f6a36451000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f6a37485000)
[xplore@ds2-0 ~]$

The ldd command output confirms that this file isn’t found (“libbtrlpcore.so.7.17 => not found“) but it gave the same result inside the working and non-working xPlore Federations. Therefore, I searched for the specific shared object library file, and I was able to find it:

[xplore@ds2-0 ~]$ find $XPLORE_HOME/ -name libbtrlpcore.so.7.17 -ls
8391369 624 -rwxr-xr-x 1 xplore xplore 637946 Dec 7 16:09 $XPLORE_HOME/dsearch/cps/cps_daemon/shared_libraries/rlp/lib/amd64-glibc212-gcc44/libbtrlpcore.so.7.17
[xplore@ds2-0 ~]$

Since this file exists but the ldd and xPlore complains about not being able to find it when it starts, then it most probably means that the LD_LIBRARY_PATH is the problem. You might have seen it before but when xPlore gets installed, it will create a file named “cpsenv.sh” that is being used to set the environment for the CPS, as the name suggest. Knowing that we have issues on the CPS and that it appears to be linked to the environment… That was obviously the next stop in my investigation and indeed, it was the reason for this issue. On the non-working Federation, I saw the following content:

[xplore@ds2-0 ~]$ cat $XPLORE_HOME/dsearch/cps/cps_daemon/bin/cpsenv.sh
#!/bin/bash

#
# (c) EMC Corporation 2007. All rights reserved.
#

# Place the CPS specific paths first
export CPS_DAEMON=$XPLORE_HOME/dsearch/cps/cps_daemon
export LD_LIBRARY_PATH=$CPS_DAEMON/bin:.:$CPS_DAEMON/shared_libraries/stellent_text_extractor:$CPS_DAEMON/shared_libraries/isys_text_extractor:$CPS_DAEMON/shared_libraries/rlp/lib/amd64-glibc25-gcc42:$LD_LIBRARY_PATH
export PATH=.:$PATH
[xplore@ds2-0 ~]$

The fact that the comment at the beginning contains “EMC Corporation 2007” didn’t sound right to me, because I remembered that an xPlore 20.2 environment should have a reference to “OpenText” instead, with a more recent date like “2018” or something. Therefore, I compared the cpsenv.sh file with the one from the working Federation:

[xplore@ds1-0 ~]$ cat $XPLORE_HOME/dsearch/cps/cps_daemon/bin/cpsenv.sh
#!/bin/bash

#
# Copyright (c) 2018-2019 OpenText.
# All Rights Reserved.
#

# Place the CPS specific paths first
export CPS_DAEMON=$XPLORE_HOME/dsearch/cps/cps_daemon
export LD_LIBRARY_PATH=$CPS_DAEMON/bin:.:$CPS_DAEMON/shared_libraries/stellent_text_extractor:$CPS_DAEMON/shared_libraries/rlp/lib/amd64-glibc212-gcc44:$LD_LIBRARY_PATH
export PATH=.:$PATH
[xplore@ds1-0 ~]$

As shown above, the path of the “rlp” libraries is different between the two (“/rlp/lib/amd64-glibc25-gcc42” for xPlore 16.4 versus “/rlp/lib/amd64-glibc212-gcc44” for xPlore 20.2). Because of that, the shared object library files aren’t loaded into the environment when the CPS starts and therefore the CPS failed to start, which explains this issue. After correcting the content of the cpsenv.sh, the environment was fixed, and everything could start again without issue.

After some more investigation on why this happened, it was found that it was actually a backup of the file in version xPlore 16.4 that was wrongly restored after the upgrade to xPlore 20.2. Therefore, xPlore upgrade had no errors as the file was properly updated to the 20.2 version but then a restore of this file was executed from a backup, which caused this issue.

Post Views: 546
Thumbnail [60x60]
by
Morgan Patou