{"id":12760,"date":"2019-08-31T08:00:44","date_gmt":"2019-08-31T06:00:44","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/documentum-encryption-decryption-of-webtop-6-8-passwords-rejected-with-recent-jdk\/"},"modified":"2019-08-31T08:00:44","modified_gmt":"2019-08-31T06:00:44","slug":"documentum-encryption-decryption-of-webtop-6-8-passwords-rejected-with-recent-jdk","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/documentum-encryption-decryption-of-webtop-6-8-passwords-rejected-with-recent-jdk\/","title":{"rendered":"Documentum – Encryption\/Decryption of WebTop 6.8 passwords ‘REJECTED’ with recent JDK"},"content":{"rendered":"

Recently, we had a project to modernize a little bit a pretty old Documentum installation. As part of this project, there were a refresh of the Application Server hosting a WebTop 6.8. In this blog, I will be talking about an issue that we faced in encryption & decryption of passwords in the refresh environment. This new environment was using WebLogic 12.1.3 with the latest PSU in conjunction with the JDK 1.8u192. Since WebTop 6.8 P08, the JDK 1.8u111 is supported so a newer version of the JDK8 should mostly<\/em> be working without much trouble.<\/p>\n

To properly deploy a WebTop application, you will need to encrypt some passwords like the Preferences or Preset passwords. Doing so in the new environment unfortunately failed:<\/p>\n

[weblogic@wls_01 ~]$ work_dir=\/tmp\/work\n[weblogic@wls_01 ~]$ cd ${work_dir}\/\n[weblogic@wls_01 work]$\n[weblogic@wls_01 work]$ jar -xf webtop_6.8_P27.war WEB-INF\/classes WEB-INF\/lib\n[weblogic@wls_01 work]$\n[weblogic@wls_01 work]$ kc=\"${work_dir}\/WEB-INF\/classes\/com\/documentum\/web\/formext\/session\/KeystoreCredentials.properties\"\n[weblogic@wls_01 work]$\n[weblogic@wls_01 work]$ sed -i \"s,use_dfc_config_dir=[^$]*,use_dfc_config_dir=false,\" ${kc}\n[weblogic@wls_01 work]$\n[weblogic@wls_01 work]$ sed -i \"s,keystore.file.location=[^$]*,keystore.file.location=${work_dir},\" ${kc}\n[weblogic@wls_01 work]$\n[weblogic@wls_01 work]$ grep -E \"^use_dfc_config_dir|^keystore.file.location\" ${kc}\nuse_dfc_config_dir=false\nkeystore.file.location=\/tmp\/work\n[weblogic@wls_01 work]$\n[weblogic@wls_01 work]$ enc_classpath=\"${work_dir}\/WEB-INF\/classes:${work_dir}\/WEB-INF\/lib\/*\"\n[weblogic@wls_01 work]$\n[weblogic@wls_01 work]$ java -classpath \"${enc_classpath}\" com.documentum.web.formext.session.TrustedAuthenticatorTool \"MyP4ssw0rd\"\nAug 27, 2019 11:02:23 AM java.io.ObjectInputStream filterCheck\nINFO: ObjectInputFilter REJECTED: class com.rsa.cryptoj.o.nc, array length: -1, nRefs: 1, depth: 1, bytes: 72, ex: n\/a\njava.security.UnrecoverableKeyException: Rejected by the jceks.key.serialFilter or jdk.serialFilter property\n        at com.sun.crypto.provider.KeyProtector.unseal(KeyProtector.java:352)\n        at com.sun.crypto.provider.JceKeyStore.engineGetKey(JceKeyStore.java:136)\n        at java.security.KeyStoreSpi.engineGetEntry(KeyStoreSpi.java:473)\n        at java.security.KeyStore.getEntry(KeyStore.java:1521)\n        at com.documentum.web.formext.session.TrustedAuthenticatorUtils.getSecretKey(Unknown Source)\n        at com.documentum.web.formext.session.TrustedAuthenticatorUtils.decryptByDES(Unknown Source)\n        at com.documentum.web.formext.session.TrustedAuthenticatorTool.main(TrustedAuthenticatorTool.java:64)\n[weblogic@wls_01 work]$<\/pre>\n
 <\/p>\n
As you can see above, the encryption of password is failing with some error. The issue is that starting with the JDK 1.8u171, Oracle introduced some new restrictions. From the Oracle release note (JDK-8189997<\/a>):<\/p>\n
New Features<\/strong>
\nsecurity-libs\/javax.crypto
\nEnhanced KeyStore Mechanisms<\/strong>
\nA new security property named jceks.key.serialFilter has been introduced. If this filter is configured, the JCEKS KeyStore uses it during the deserialization of the encrypted Key object stored inside a SecretKeyEntry. If it is not configured or if the filter result is UNDECIDED (for example, none of the patterns match), then the filter configured by jdk.serialFilter is consulted.<\/p>\n
If the system property jceks.key.serialFilter is also supplied, it supersedes the security property value defined here.<\/p>\n
The filter pattern uses the same format as jdk.serialFilter. The default pattern allows java.lang.Enum, java.security.KeyRep, java.security.KeyRep$Type, and javax.crypto.spec.SecretKeySpec but rejects all the others.<\/p>\n
Customers storing a SecretKey that does not serialize to the above types must modify the filter to make the key extractable.<\/p><\/blockquote>\n
 <\/p>\n
On recent versions of Documentum Administrator for example, there is no issue because it complies but for WebTop 6.8, it doesn’t and therefore to be able to encrypt\/decrypt the password, you will have to modify the filter. There are several solutions to our current problem:<\/p>\n