{"id":25499,"date":"2023-06-01T23:17:29","date_gmt":"2023-06-01T21:17:29","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/?p=25499"},"modified":"2023-06-12T10:06:19","modified_gmt":"2023-06-12T08:06:19","slug":"troubleshoot-cloud-unit-error-invalid-cloud-config-schema-user-data-oracle-cloud-infrastructure","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/troubleshoot-cloud-unit-error-invalid-cloud-config-schema-user-data-oracle-cloud-infrastructure\/","title":{"rendered":"Troubleshoot cloud-unit “Error: Invalid cloud-config schema: user-data” @\u00a0Oracle Cloud Infrastructure"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

While applying what seemed to be a minor update, we managed to “break” our cloud-init automated OCI compute instance deployment process. This blog describes how to quickly troubleshoot & debug your #cloud-config on Oracle Cloud Infrastructure (not so different from other hyperscalers)<\/p>\n\n\n\n

safe harbor statement <\/strong><\/p>\n\n\n\n

The below troubleshooting scenario has been simplified to ease the understanding of the root cause analysis<\/p>\n\n\n\n

Step 1, Identify the error message<\/h2>\n\n\n\n

Let’s start our investigation at the roots, with logfiles. <\/p>\n\n\n\n

Python modules log: \/var\/log\/cloud-init.log<\/h3>\n\n\n\n

By default, all python module vent are logged with the standard hierarchy levels : WARNING, DEBUG, aso ..<\/p>\n\n\n\n

[opc@bastion-vpn-server ~]$ sudo tail -f -n2 \/var\/log\/cloud-init.log\n2023-05-31 20:15:44,198 - util.py[DEBUG]: cloud-init mode 'modules' took 8.091 seconds\n2023-05-31 20:15:44,198 - handlers.py[DEBUG]: finish: modules-final: FAIL: running modules for final<\/code><\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
Hum, interesting! last line shows we ran until final stage (aka. our user-defined #cloud-config). But it’s still unclear what really caused the failure.<\/p>\n\n\n\n
Let’s move forward to the next log.<\/p>\n\n\n\n
<\/p>\n\n\n\n
cloud-init configuration output: \/var\/log\/cloud-init-output.log<\/h3>\n\n\n\n
Now, we just moved to the “main” logfile which protocols all #cloud-config configuration output (aka. STDOUT and STDERR)<\/p>\n\n\n\n
[opc@bastion-vpn-server ~]$ sudo tail -f  \/var\/log\/cloud-init-output.log<\/strong>\n...\n... <snippet truncated>\n...\n\/var\/lib\/cloud\/instance\/scripts\/runcmd: line 2: \/usr\/bin: Is a directory\n<\/mark><\/code>...\n... <snippet truncated>\n...\nNo match for argument: htop\nError: Unable to find a match: htop<\/mark><\/code>\n...\n... <snippet truncated>\n...\n2023-05-31 20:15:44,107 - cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in \/var\/lib\/cloud\/instance\/scripts)\n2023-05-31 20:15:44,107 - util.py[WARNING]: Running module scripts-user () failed<\/mark><\/code><\/pre>\n\n\n\n

Hum, we get closer! Looks like a command was not interpreted correctly but do we have still access to the commands executed? Yes, of course check the generated scripts in \/var\/lib\/cloud\/instance\/scripts:<\/p>\n\n\n\n
[opc@bastion-vpn-server ~]$ sudo cat \/var\/lib\/cloud\/instance\/scripts\/runcmd<\/strong>\n#!\/bin\/sh\n'\/usr\/bin' 'dnf' '-y' 'install' 'https:\/\/dl.fedoraproject.org\/pub\/epel\/epel-release-latest-8.noarch.rpm'\n'\/usr\/bin\/sleep' '5'\n'\/usr\/bin\/dnf' '-y' 'install' 'htop'<\/code><\/pre>\n\n\n\n
From now on, a Linux expert will immediate recognize a typo ! Are we sure we have only a single error? Let’s move forward and try to validate the syntax<\/p>\n\n\n\n
Step 2, Validate cloud-init user-data deployed on the system<\/h2>\n\n\n\n
Now, here we (almost) cheated ! Unfortunately, on OCI cloud-init version isn’t up2date (Release 22.1 GA date Feb. 2022).  And no, Oracle is not lagging behind others large hyperscalers (at least compared to the one with the smile). Indeed, cloud-init evolves very quickly and the documentation is not very easy to follow especially across releases. <\/p>\n\n\n\n
<\/p>\n\n\n\n
Validate the syntax as follows (At the time of writing this post the cloud-init release 22.1 is available on OCI):<\/p>\n\n\n\n
[opc@bastion-vpn-server ~]# sudo cloud-init --version<\/strong>\n\/usr\/bin\/cloud-init 22.1-6.0.4.el8_7.2\n[opc@bastion-vpn-server ~]$ sudo cloud-init devel schema --system<\/strong>\nError:\nCloud config schema errors: runcmd.1: ['\/usr\/bin\/sleep', 5] is not valid under any of the given schemas<\/code><\/pre>\n\n\n\n
<\/p>\n\n\n\n
Since cloud-init release 22.2 (GA date May 2022) the command to validate cloud-init user-data evolved to:<\/p>\n\n\n\n
[opc@bastion-vpn-server opt]$ sudo cloud-init --version<\/strong>\n$\/bin\/cloud-init 23.2\n[opc@bastion-vpn-server opt]$ sudo cloud-init schema --system<\/strong>\nInvalid cloud-config \/var\/lib\/cloud\/instances\/ocid1.instance.oc1.eu-zurich-1.an5heljr4p5qcaac5uh7vhwsfmycazwru55ywyfrpljox76mn6edhyvfyfza\/cloud-config.txt\nError: Cloud config schema errors: runcmd.1: ['\/usr\/bin\/sleep', 5] is not valid under any of the given schemas\nError: Invalid cloud-config schema: user-data<\/code><\/pre>\n\n\n\n
The last snippet is an example after latest cloud-init release was installed from source code (https:\/\/github.com\/canonical\/cloud-init\/releases<\/a>)<\/p>\n\n\n\n
<\/p>\n\n\n\n

But wait a minute? this is not really the problem identified earlier, isn’t it?<\/p>\n\n\n\n
\/var\/lib\/cloud\/instance\/scripts\/runcmd: line 2: \/usr\/bin: Is a directory\n<\/pre>\n\n\n\n
Indeed, we have introduced not only one typo but also a second issue (syntax inconsistency).<\/p>\n\n\n\n
Let’s move forward and retrieve the “raw\/original” #cloud-config<\/p>\n\n\n\n
<\/p>\n\n\n\n
Step3, retrieve (and validate) raw user-data #cloud-config<\/h2>\n\n\n\n
The cloud-init user-datas is passed to the compute instance through RestAPIs. On OCI you can retrieve the #cloud-config user data YAML file as follows:<\/p>\n\n\n\n
[opc@bastion-vpn-server ~]$ curl --silent -H \"Authorization: Bearer Oracle\" -L http:\/\/169.254.169.254\/opc\/v2\/instance\/metadata\/user_data -o - | base64 --decode > \/tmp\/cloud-init_user-data.yml<\/strong><\/code><\/pre>\n\n\n\n
Now, we can use the aforementioned syntax validation commands to get rid of the error(s):<\/p>\n\n\n\n
[opc@bastion-vpn-server opt]$ sudo cloud-init devel schema --config-file<\/strong> \/tmp\/cloud-init_user-data.yml<\/strong>\nInvalid cloud-config \/tmp\/cloud-init_user-data.yml\nError: Cloud config schema errors: runcmd.1: ['\/usr\/bin\/sleep', 5] is not valid under any of the given schemas<\/mark>\nError: Invalid cloud-config schema: user-data\n[opc@bastion-vpn-server opt]$ vi \/tmp\/cloud-init_user-data.yml<\/strong>\n...\n<fix the typo>\n...\n[opc@bastion-vpn-server opt]$ sudo cloud-init devel schema --config-file <\/strong>\/tmp\/cloud-init_user-data.yml\nValid cloud-config: \/tmp\/cloud-init_user-data.yml<\/mark><\/code><\/pre>\n\n\n\n
Sounds good? Might be too easy to be true but who knows \ud83d\ude09
Now, we identified, fixed error and tested the fixes by re-running the whole provisioning process.<\/p>\n\n\n\n
Step 4, if needed re-run #cloud-config stages\/modules<\/h2>\n\n\n\n
Working with cloud-init can sometimes be really annoying as it’s a very complex tool with orchestrated stages, modules (ie: runcmd, ansible, write_files, aso…) which are customizable and customized by each hyperscaler differently. <\/p>\n\n\n\n
Follows, the cheat sheet to get through if further troubleshooting is needed<\/p>\n\n\n\n
# Remove logs and artifacts so cloud-init can re-run\nsudo cloud-init clean --logs\n\n# Detect local #cloud-config source (ie: cloud platform settings, IPs, routes)\nsudo cloud-init init\nsudo cloud-init init --local\n\n# Run #cloud-config modules defined in \/etc\/cloud\/cloud.cfg (ie: timezone)\nsudo cloud-init modules --mode=config \n\n# Re-run user accustomed #cloud-config\nsudo cloud-init modules --mode=final\n\n# Alternatively, re-run only a single user accustomed #cloud-config module\nsudo cloud-init single --name runcmd<\/code><\/pre>\n\n\n\n
Well, depending on \/etc\/cloud\/cloud.cfg settings most of the modules if executed won’t be re-executed without tweaking the config.file if they already ran (whatever the outcome). <\/p>\n\n\n\n
[opc@bastion-vpn-server ~]$ sudo tail -f  \/var\/log\/cloud-init.log\n...\n... <snippet truncated>\n...\n2023-06-01 21:15:44,968 - helpers.py[DEBUG]: config-runcmd already ran (freq=once-per-instance)<\/code>\n<\/pre>\n\n\n\n
Please refer to the documentation to<\/p>\n\n\n\n