Receiving an incident is not always a pleasure, but it is always the case when we share the solution!
A few days ago, I received an incident regarding WebConsumer on a production environment, saying that documents are not transferred as expected to WebConsumer.
The issue didn’t happened for all documents, that’s why I directly suspect the High Availability configuration on this environment. Moreover, I know that the IDS is installed only on CS1 (as designed). So I checked the JMS logs on :
CS1 : No errors found there.
CS2 : Errors found :
2019-02-11 04:05:39,097 UTC INFO [stdout] (default task-60) [DEBUG] - c.e.d.a.c.m.lifecycle.D2LifecycleConfig : D2LifecycleConfig::applyMethod start 'WCPublishDocumentMethod' 2019-02-11 04:05:39,141 UTC INFO [stdout] (default task-60) [DEBUG] - c.e.d.a.c.m.lifecycle.D2LifecycleConfig : D2LifecycleConfig::applyMethod before session apply 'WCPublishDocumentMethod' time: 0.044s 2019-02-11 04:05:39,773 UTC INFO [stdout] (default task-89) 2019-02-11 04:05:39,773 UTC ERROR [com.domain.repository1.dctm.methods.WCPublishDoc] (default task-89) DfException:: THREAD: default task-89; MSG: [DM_METHOD_E_JMS_APP_SERVER_NAME_NOTFOUND]error: "The app_server_name/servlet_name 'WebCache' is not specified in dm_server_config/dm_jms_config."; ERRORCODE: 100; NEXT: null
To cross check:
On CS1:
[dmadmin@CONTENT_SERVER1 ~]$ cd $DOCUMENTUM/shared/wildfly9.0.1/server/DctmServer_MethodServer/log [dmadmin@CONTENT_SERVER1 log]$ grep DM_METHOD_E_JMS_APP_SERVER_NAME_NOTFOUND server.log | wc -l 0
On CS2:
[dmadmin@CONTENT_SERVER2 ~]$ cd $DOCUMENTUM/shared/wildfly9.0.1/server/DctmServer_MethodServer/log [dmadmin@CONTENT_SERVER2 log]$ grep DM_METHOD_E_JMS_APP_SERVER_NAME_NOTFOUND server.log | wc -l 60
So I listed all dm_server_config:
API> ?,c,select r_object_id,object_name from dm_server_config; r_object_id object_name ---------------- ----------------------------- 3d01e24080000102 repository1 3d01e240800062be CONTENT_SERVER2_repository1
Then, I checked the app servers list configured:
On CS1:
API> dump,c,3d01e24080000102
...
USER ATTRIBUTES
object_name : repository1
...
app_server_name [0]: do_method
[1]: do_mail
[2]: FULLTEXT_SERVER1_PORT_IndexAgent
[3]: WebCache
[4]: FULLTEXT_SERVER2_PORT_IndexAgent
app_server_uri [0]: https://CONTENT_SERVER1:9082/DmMethods/servlet/DoMethod
[1]: https://CONTENT_SERVER1:9082/DmMail/servlet/DoMail
[2]: https://FULLTEXT_SERVER1:PORT/IndexAgent/servlet/IndexAgent
[3]: https://CONTENT_SERVER1:6679/services/scs/publish
[4]: https://FULLTEXT_SERVER2:PORT/IndexAgent/servlet/IndexAgent
...
Good, WebCache is configured here.
On CS2:
API> dump,c,3d01e240800062be
...
USER ATTRIBUTES
object_name : CONTENT_SERVER2_repository1
...
app_server_name [0]: do_method
[1]: do_mail
[2]: FULLTEXT_SERVER1_PORT_IndexAgent
[3]: FULLTEXT_SERVER2_PORT_IndexAgent
app_server_uri [0]: https://CONTENT_SERVER2:9082/DmMethods/servlet/DoMethod
[1]: https://CONTENT_SERVER2:9082/DmMail/servlet/DoMail
[2]: https://FULLTEXT_SERVER1:PORT/IndexAgent/servlet/IndexAgent
[3]: https://FULLTEXT_SERVER2:PORT/IndexAgent/servlet/IndexAgent
...
Ok! The root cause of this error is clear now.
The concerned method is WCPublishDocumentMethod, but applied when? by who?
I noticed that in the log above:
D2LifecycleConfig::applyMethod start 'WCPublishDocumentMethod'
So, WCPublishDocumentMethod applied by the D2LifecycleConfig, which is applied when? by who?
Sought in the server.log file and found:
2019-02-11 04:05:04,490 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : User : repository1 2019-02-11 04:05:04,490 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : New session manager creation. 2019-02-11 04:05:04,491 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Session manager set identity. 2019-02-11 04:05:04,491 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Session manager get session. 2019-02-11 04:05:06,006 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Workitem ID: 4a01e2408002bd3d 2019-02-11 04:05:06,023 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Searching workflow tracker... 2019-02-11 04:05:06,031 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Searching workflow config... 2019-02-11 04:05:06,032 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Get packaged documents... 2019-02-11 04:05:06,067 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Apply on masters... 2019-02-11 04:05:06,068 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Workitem acquire... 2019-02-11 04:05:06,098 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Applying lifecycle (Target state : On Approved / Transition :promote 2019-02-11 04:05:06,098 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : No workflow properties 2019-02-11 04:05:06,098 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Searching target state name and/or transition type. 2019-02-11 04:05:06,099 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Target state name :On Approved 2019-02-11 04:05:06,099 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Target transition type :promote 2019-02-11 04:05:06,099 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Performing D2 lifecycle on :FRM-8003970 (0901e240800311cd) 2019-02-11 04:05:06,099 UTC INFO [stdout] (default task-79) [INFO ] - com.emc.d2.api.methods.D2WFLifeCycleMethod : Searching associated D2 lifecycle... 2019-02-11 04:05:06,099 UTC INFO [stdout] (default task-79) [DEBUG] - c.e.d.a.c.m.lifecycle.D2LifecycleConfig : D2LifecycleConfig::getInstancesForObject start time 0.000s ... 2019-02-11 04:05:39,097 UTC INFO [stdout] (default task-60) [DEBUG] - c.e.d.a.c.m.lifecycle.D2LifecycleConfig : D2LifecycleConfig::applyMethod start 'WCPublishDocumentMethod' ...
Hummmm, the D2WFLifeCycleMethod is applied by the job D2JobLifecycleBatch. I checked the target server of this job:
API> ?,c,SELECT target_server FROM dm_job WHERE object_name='D2JobLifecycleBatch'; target_server ------------- (1 row affected)
As I suspected, no target server defined! That’s mean that the job can be executed on “Any Running Server”, that’s why this method has been executed on CS2… While CS2 is not configured to do so.
Now, two solutions are possible:
1. Change the target_server to use only CS1:
API> ?,c,UPDATE dm_job OBJECTS SET target_server='repository1.repository1@CONTENT_SERVER1' WHERE object_name='D2JobLifecycleBatch';
2. Add the app server WebCache to CS2, pointing to CS1:
API> fetch,c,3d01e240800062be API> append,c,l,app_server_name SET> WebCache API> append,c,l,app_server_uri SET> https://CONTENT_SERVER1:6679/services/scs/publish API> save,c,l
Check after update:
API> dump,c,3d01e240800062be
...
USER ATTRIBUTES
object_name : CONTENT_SERVER2_repository1
...
app_server_name [0]: do_method
[1]: do_mail
[2]: FULLTEXT_SERVER1_PORT_IndexAgent
[3]: FULLTEXT_SERVER2_PORT_IndexAgent
[4]: WebCache
app_server_uri [0]: https://CONTENT_SERVER2:9082/DmMethods/servlet/DoMethod
[1]: https://CONTENT_SERVER2:9082/DmMail/servlet/DoMail
[2]: https://FULLTEXT_SERVER1:PORT/IndexAgent/servlet/IndexAgent
[3]: https://FULLTEXT_SERVER2:PORT/IndexAgent/servlet/IndexAgent
[4]: https://CONTENT_SERVER1:6679/services/scs/publish
...
We choose the second option, because:
– The job is handled by the application team,
– Modifying the job to run only on CS1 will resolve this case, but if the method is applied by another job or manually on CS2, we will get again the same error.
After this update no error has been recorded in the log file:
...
2019-02-12 04:06:10,948 UTC INFO [stdout] (default task-81) [DEBUG] - c.e.d.a.c.m.lifecycle.D2LifecycleConfig : D2LifecycleConfig::applyMethod start 'WCPublishDocumentMethod'
2019-02-12 04:06:10,955 UTC INFO [stdout] (default task-81) [DEBUG] - c.e.d.a.c.m.lifecycle.D2LifecycleConfig : D2LifecycleConfig::applyMethod before session apply 'WCPublishDocumentMethod' time: 0.007s
2019-02-12 04:06:10,955 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.methods.D2Method : No ARG_RETURN_ID in mapArguments
2019-02-12 04:06:10,956 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.methods.D2Method : newObject created, user session used: 0801e2408023f714
2019-02-12 04:06:10,956 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.D2SysObject : getFolderIdFromCache: got folder: /System/D2/Data/c6_method_return, object id: 0b01e2408000256b, docbase: repository1
2019-02-12 04:06:11,016 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.methods.D2Method : mapArguments: {-method_return_id=0801e2408023f714}
2019-02-12 04:06:11,016 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.methods.D2Method : origArguments: {-id=0901e24080122a59}
2019-02-12 04:06:11,017 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.methods.D2Method : methodName: WCPublishDocumentMethod
2019-02-12 04:06:11,017 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.methods.D2Method : methodParams: -id 0901e24080122a59 -user_name dmadmin -docbase_name repository1
2019-02-12 04:06:11,017 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.methods.D2Method : Start WCPublishDocumentMethod method with JMS (Java Method Services) runLocally hint set is false
2019-02-12 04:06:11,017 UTC INFO [stdout] (default task-81) [DEBUG] - com.emc.d2.api.methods.D2Method : key: -method_return_id, and value: 0801e2408023f714
...
I hope this blog will help you to quickly resolve this kind of incident.
![Thumbnail [60x60]](https://www.dbi-services.com/blog/wp-content/uploads/2022/09/DDI_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/OLS_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/GME_web-min-scaled.jpg)
![Thumbnail [90x90]](https://www.dbi-services.com/blog/wp-content/uploads/2022/08/MOP_web-min-scaled.jpg)