Oracle ZDM Migration – java.security.InvalidKeyException: invalid key format

ZDM tool migration requires SSH Passwordless Login without passphrase between ZDM Host, the source and the target. Configuring appropriate keys might still result in a java security exception on this one. In this blog I will tell you how to deal with such a problem. I faced this problem implementing ZDM to migrate On-Premise Database to new ExaCC at one of our customer.

Setting up SSH Passwordless Login

First of all we need to create the private and public key on the ZDM Host.

From the ZDM host, with zdmuser, go in the ~/.ssh folder and run ssh-keygen.

[zdmuser@zdmhost .ssh]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/zdmuser/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/zdmuser/.ssh/id_rsa.
Your public key has been saved in /home/zdmuser/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:8uTp************************ziw zdmuser@zdmhost
The key's randomart image is:
+---[RSA 3072]----+
|   oo+==.        |
...
...
...
|    o.+..        |
+----[SHA256]-----+

This will create 2 keys, one private (id_rsa) and one public (id_rsa_pub).

Get the content of the public key.

[zdmuser@zdmhost .ssh]$ cat id_rsa.pub
ssh-rsa AAAA************************vaU= zdmuser@zdmhost

And add the content of the public RSA key to the authorized_keys file from both ExaCC Cluster VMs (target ExaCC-cl01n1 and ExaCC-cl01n2) opc user and the on-premises VM (source vmonpr) oracle user.

[opc@ExaCC-cl01n1 .ssh]$ echo "ssh-rsa AAAA************************vaU= zdmuser@zdmhost" >> authorized_keys

[opc@ExaCC-cl01n2 .ssh]$ echo "ssh-rsa AAAA************************vaU= zdmuser@zdmhost" >> authorized_keys

oracle@vmonpr:/home/oracle/.ssh/ [ONPR] echo "ssh-rsa AAAA************************vaU= zdmuser@zdmhost" >> authorized_keys

We will then test SSH connection to the 3 VMs and ensure no password are requested. Example:

[zdmuser@zdmhost migration]$ ssh opc@ExaCC-cl01n1
Last login: Fri Feb  2 16:58:04 2024 from 10.160.52.122
[opc@ExaCC-cl01n1 ~]$

Check ZDM migration

Checking ZDM migration with zdmcli and -eval option might get failed:

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli migrate database -sourcesid ONPR -rsp /home/zdmuser/migration/zdm_ONPR_physical_online.rsp -sourcenode vmonpr -srcauth zdmauth -srcarg1 user:oracle -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode ExaCC-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -eval
zdmhost.domain.com: Audit ID: 50
Enter source database ONPR SYS password:
zdmhost: 2024-02-02T16:30:19.487Z : Processing response file ...
Operation "zdmcli migrate database" scheduled with the job ID "11".

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli query job -jobid 11
zdmhost.domain.com: Audit ID: 52
Job ID: 11
User: zdmuser
Client: zdmhost
Job Type: "EVAL"
Scheduled job command: "zdmcli migrate database -sourcesid ONPR -rsp /home/zdmuser/migration/zdm_ONPR_physical_online.rsp -sourcenode vmonpr -srcauth zdmauth -srcarg1 user:oracle -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode ExaCC-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -eval"
Scheduled job execution start time: 2024-02-02T17:30:19+01. Equivalent local time: 2024-02-02 17:30:19
Current status: FAILED
Result file path: "/u01/app/oracle/chkbase/scheduled/job-11-2024-02-02-17:30:48.log"
Metrics file path: "/u01/app/oracle/chkbase/scheduled/job-11-2024-02-02-17:30:48.json"
Job execution start time: 2024-02-02 17:30:48
Job execution end time: 2024-02-02 17:30:48
Job execution elapsed time: 0 seconds

Result file "/u01/app/oracle/chkbase/scheduled/job-11-2024-02-02-17:30:48.log" contents:
zdmhost: 2024-02-02T16:30:48.591Z : Processing response file ...
zdmhost: 2024-02-02T16:30:48.595Z : Processing response file ...
PRCZ-4002 : failed to execute command "/bin/cp" using the privileged execution plugin "zdmauth" on nodes "ExaCC-cl01n1"
java.security.InvalidKeyException: invalid key format

Error of failed execution is :

java.security.InvalidKeyException: invalid key format

Solution

The problem is due to the fact that ZDM only supports RSA key and the generated key was an OPENSSH key.

Checking current key, we can see that the key is an openssh key:

[zdmuser@zdmhost .ssh]$ head -n1 id_rsa
-----BEGIN OPENSSH PRIVATE KEY-----

[zdmuser@zdmhost .ssh]$ tail -n 1 id_rsa
-----END OPENSSH PRIVATE KEY-----

We need to convert the private key to PEM format.

[zdmuser@zdmhost .ssh]$ ssh-keygen -p -m PEM -f ~/.ssh/id_rsa
Key has comment 'zdmuser@zdmhost'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

The new key looks now like.

[zdmuser@zdmhost .ssh]$ head -n1 id_rsa
-----BEGIN RSA PRIVATE KEY-----

[zdmuser@zdmhost .ssh]$ tail -n 1 id_rsa
-----END RSA PRIVATE KEY-----

And, now, zdmcli eval command is succeeding.

[zdmuser@zdmhost migration]$ /u01/app/oracle/product/zdm/bin/zdmcli query job -jobid 39
zdmhost.domain.com: Audit ID: 434
Job ID: 39
User: zdmuser
Client: zdmhost
Job Type: "EVAL"
Scheduled job command: "zdmcli migrate database -sourcesid ONPR -rsp /home/zdmuser/migration/zdm_ONPR_physical_online.rsp -sourcenode vmonpr -srcauth zdmauth -srcarg1 user:oracle -srcarg2 identity_file:/home/zdmuser/.ssh/id_rsa -srcarg3 sudo_location:/usr/bin/sudo -targetnode ExaCC-cl01n1 -tgtauth zdmauth -tgtarg1 user:opc -tgtarg2 identity_file:/home/zdmuser/.ssh/id_rsa -tgtarg3 sudo_location:/usr/bin/sudo -tdekeystorepasswd -tgttdekeystorepasswd -eval"
Scheduled job execution start time: 2024-02-14T14:18:19+01. Equivalent local time: 2024-02-14 14:18:19
Current status: SUCCEEDED
Result file path: "/u01/app/oracle/chkbase/scheduled/job-39-2024-02-14-14:18:29.log"
Metrics file path: "/u01/app/oracle/chkbase/scheduled/job-39-2024-02-14-14:18:29.json"
Job execution start time: 2024-02-14 14:18:29
Job execution end time: 2024-02-14 14:21:18
Job execution elapsed time: 2 minutes 48 seconds
ZDM_GET_SRC_INFO ........... PRECHECK_PASSED
ZDM_GET_TGT_INFO ........... PRECHECK_PASSED
ZDM_PRECHECKS_SRC .......... PRECHECK_PASSED
ZDM_PRECHECKS_TGT .......... PRECHECK_PASSED
ZDM_SETUP_SRC .............. PRECHECK_PASSED
ZDM_SETUP_TGT .............. PRECHECK_PASSED
ZDM_PREUSERACTIONS ......... PRECHECK_PASSED
ZDM_PREUSERACTIONS_TGT ..... PRECHECK_PASSED
ZDM_VALIDATE_SRC ........... PRECHECK_PASSED
ZDM_VALIDATE_TGT ........... PRECHECK_PASSED
ZDM_POSTUSERACTIONS ........ PRECHECK_PASSED
ZDM_POSTUSERACTIONS_TGT .... PRECHECK_PASSED
ZDM_CLEANUP_SRC ............ PRECHECK_PASSED
ZDM_CLEANUP_TGT ............ PRECHECK_PASSED