{"id":11079,"date":"2018-04-10T10:13:33","date_gmt":"2018-04-10T08:13:33","guid":{"rendered":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/"},"modified":"2018-04-10T10:13:33","modified_gmt":"2018-04-10T08:13:33","slug":"deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1","status":"publish","type":"post","link":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/","title":{"rendered":"Deploy a Cloudera cluster with Terraform and Ansible in Azure &#8211; part 1"},"content":{"rendered":"<p>Deploying a Cloudera distribution of Hadoop automatically is very interesting in terms of time-saving. Infrastructure as Code tools such as Ansible, Puppet, Chef, Terraform, allow now to provision, manage and deploy configuration for large clusters.<\/p>\n<p>In this blog posts series, we will see how to deploy and install a CDH cluster with Terraform and Ansible in the Azure cloud.<\/p>\n<p>The first part consists of provisioning the environment with Terraform in Azure. Terraform features an extension to interact with cloud providers such as Azure and AWS. You can find <a href=\"https:\/\/www.terraform.io\/docs\/providers\/azure\/index.html\">here<\/a> the Terraform documentation for the Azure module.<\/p>\n<h3>Desired architecture<\/h3>\n<p>&nbsp;<\/p>\n<p><a href=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-22471 aligncenter\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png\" alt=\"Azure_architecture\" width=\"613\" height=\"390\" \/><\/a><\/p>\n<p>Above a representation of the wished architecture for our CDH environment. 5 nodes for a testing infrastructure, including a Cloudera manager node, a second master node for the Hadoop Secondary NameNode and 3 workers.<\/p>\n<h3>\u00a0Prerequisites<\/h3>\n<p>Terraform must be installed on your system. https:\/\/docs.microsoft.com\/en-us\/azure\/virtual-machines\/linux\/terraform-install-configure#install-terraform<\/p>\n<p>Generate a Client ID and a Client Secret from Azure CLI to authenticate in Azure with Terraform.<\/p>\n<p>1. Sign in to administer your Azure subscription:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">[root@centos Terraform]# az login<\/pre>\n<p>2. Get the subscription ID and tenant ID:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">[root@centos Terraform]# az account show --query \"{subscriptionId:id, tenantId:tenantId}\"<\/pre>\n<p>3. Create separate credentials for TF:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">[root@centos Terraform]# az ad sp create-for-rbac --role=\"Contributor\" --scopes=\"\/subscriptions\/${SUBSCRIPTION_ID}\"<\/pre>\n<p>4. Save the following information:<\/p>\n<ul>\n<li>subscription_id<\/li>\n<li>client_id<\/li>\n<li>client_secret<\/li>\n<li>tenant_id<\/li>\n<\/ul>\n<p>Now we are ready to start using Terraform with AzureRM API.<\/p>\n<h3>Build your cluster<\/h3>\n<p>With Terraform, we will provision the following resources in Azure:<\/p>\n<ul>\n<li>A resource group &#8211; &#8220;Cloudera-Cluster&#8221;<\/li>\n<li>1 virtual network &#8211; &#8220;cdh_vnet&#8221;<\/li>\n<li>1 network security group &#8211; &#8220;cdh-nsg&#8221;<\/li>\n<li>1 storage account &#8211; &#8220;dbistorage&#8221;<\/li>\n<li>5 network interfaces &#8211; &#8220;instance_name_network_interface&#8221;<\/li>\n<li>5 Public \/ Private IP &#8211; &#8220;cdh-pip1-4&#8221;<\/li>\n<\/ul>\n<p>First, we will create a variable file, which contains all variables needed without specific values.\u00a0 The values are specified in the var_values.tfvars file.<\/p>\n<h4>\u00a0variables.tf<\/h4>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">\/* Configure Azure Provider and declare all the Variables that will be used in Terraform configurations *\/\nprovider \"azurerm\" {\n  subscription_id \t= \"${var.subscription_id}\"\n  client_id \t\t= \"${var.client_id}\"\n  client_secret \t= \"${var.client_secret}\"\n  tenant_id \t\t= \"${var.tenant_id}\"\n}\n\nvariable \"subscription_id\" {\n  description = \"Enter Subscription ID for provisioning resources in Azure\"\n}\n\nvariable \"client_id\" {\n  description = \"Enter Client ID for Application created in Azure AD\"\n}\n\nvariable \"client_secret\" {\n  description = \"Enter Client secret for Application in Azure AD\"\n}\n\nvariable \"tenant_id\" {\n  description = \"Enter Tenant ID \/ Directory ID of your Azure AD. Run Get-AzureSubscription to know your Tenant ID\"\n}\n\nvariable \"location\" {\n  description = \"The default Azure region for the resource provisioning\"\n}\n\nvariable \"resource_group_name\" {\n  description = \"Resource group name that will contain various resources\"\n}\n\nvariable \"vnet_cidr\" {\n  description = \"CIDR block for Virtual Network\"\n}\n\nvariable \"subnet1_cidr\" {\n  description = \"CIDR block for Subnet within a Virtual Network\"\n}\n\nvariable \"subnet2_cidr\" {\n  description = \"CIDR block for Subnet within a Virtual Network\"\n}\n\nvariable \"vm_username_manager\" {\n  description = \"Enter admin username to SSH into Linux VM\"\n}\n\nvariable \"vm_username_master\" {\n  description = \"Enter admin username to SSH into Linux VM\"\n}\n\nvariable \"vm_username_worker1\" {\n  description = \"Enter admin username to SSH into Linux VM\"\n}\n\nvariable \"vm_username_worker2\" {\n  description = \"Enter admin username to SSH into Linux VM\"\n}\n\nvariable \"vm_username_worker3\" {\n  description = \"Enter admin username to SSH into Linux VM\"\n}\n\nvariable \"vm_password\" {\n  description = \"Enter admin password to SSH into VM\"\n}<\/pre>\n<p>&nbsp;<\/p>\n<h4>var_values.tfvars<\/h4>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">subscription_id \t= \"xxxxxxx\"\nclient_id \t\t= \"xxxxxxx\"\nclient_secret \t\t= \"xxxxxxx\"\ntenant_id \t\t= \"xxxxxxx\"\nlocation \t\t= \"YOUR-LOCATION\"\nresource_group_name     = \"Cloudera-Cluster\"\nvnet_cidr \t\t= \"192.168.0.0\/16\"\nsubnet1_cidr \t\t= \"192.168.1.0\/24\"\nsubnet2_cidr\t\t= \"192.168.2.0\/24\"\nvm_username_manager \t\t= \"dbi\"\nvm_username_master \t\t= \"dbi\"\nvm_username_worker1 \t\t= \"dbi\"\nvm_username_worker2 \t\t= \"dbi\"\nvm_username_worker3 \t\t= \"dbi\"\nvm_password \t\t= \"YOUR-PASSWORD\"<\/pre>\n<p>&nbsp;<\/p>\n<p>Next, we will configure the virtual network with 1 subnet for all hosts.<\/p>\n<h4>network.tf<\/h4>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">resource \"azurerm_resource_group\" \"terraform_rg\" {\n  name \t\t= \"${var.resource_group_name}\"\n  location \t= \"${var.location}\"\n}\n\nresource \"azurerm_virtual_network\" \"vnet\" {\n  name \t\t\t= \"cdh-vnet\"\n  address_space \t= [\"${var.vnet_cidr}\"]\n  location \t\t= \"${var.location}\"\n  resource_group_name   = \"${azurerm_resource_group.terraform_rg.name}\"\n\n  tags {\n\tgroup = \"Cloudera-Cluster\"\n  }\n}\n\nresource \"azurerm_subnet\" \"subnet_1\" {\n  name \t\t\t= \"Subnet-1\"\n  address_prefix \t= \"${var.subnet1_cidr}\"\n  virtual_network_name \t= \"${azurerm_virtual_network.vnet.name}\"\n  resource_group_name \t= \"${azurerm_resource_group.terraform_rg.name}\"\n}\n\n<\/pre>\n<p class=\"brush: bash; gutter: true; first-line: 1\">Next, we can create the storage account related to the resource group with a container.<\/p>\n<h4 class=\"brush: bash; gutter: true; first-line: 1\">storage.tf<\/h4>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">resource \"azurerm_storage_account\" \"storage\" {\n  name \t\t\t= \"dbistorage1\"\n  resource_group_name \t= \"${azurerm_resource_group.terraform_rg.name}\"\n  location \t\t= \"${var.location}\"\n  account_tier    = \"Standard\"\n  account_replication_type = \"LRS\"\n\n  tags {\n\tgroup = \"Cloudera-Cluster\"\n  }\n}\n\nresource \"azurerm_storage_container\" \"container\" {\n  name \t\t\t= \"vhds\"\n  resource_group_name \t= \"${azurerm_resource_group.terraform_rg.name}\"\n  storage_account_name \t= \"${azurerm_storage_account.storage.name}\"\n  container_access_type = \"private\"\n}<\/pre>\n<p class=\"brush: bash; gutter: true; first-line: 1\">The next step will be to create a security group for all VM. We will implement 2 rules. Allow SSH and HTTP connection from everywhere, which it&#8217;s basically not really secure, but don&#8217;t forget that we are in a volatile testing infrastructure.<\/p>\n<h4 class=\"brush: bash; gutter: true; first-line: 1\">security_group.tf<\/h4>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">resource \"azurerm_network_security_group\" \"nsg_cluster\" {\n  name \t\t\t= \"cdh-nsg\"\n  location \t\t= \"${var.location}\"\n  resource_group_name \t= \"${azurerm_resource_group.terraform_rg.name}\"\n\n  security_rule {\n\tname \t\t\t= \"AllowSSH\"\n\tpriority \t\t= 100\n\tdirection \t\t= \"Inbound\"\n\taccess \t\t        = \"Allow\"\n\tprotocol \t\t= \"Tcp\"\n\tsource_port_range       = \"*\"\n    destination_port_range     \t= \"22\"\n    source_address_prefix      \t= \"*\"\n    destination_address_prefix \t= \"*\"\n  }\n\n  security_rule {\n\tname \t\t\t= \"AllowHTTP\"\n\tpriority\t\t= 200\n\tdirection\t\t= \"Inbound\"\n\taccess \t\t\t= \"Allow\"\n\tprotocol \t\t= \"Tcp\"\n\tsource_port_range       = \"*\"\n    destination_port_range     \t= \"80\"\n    source_address_prefix      \t= \"Internet\"\n    destination_address_prefix \t= \"*\"\n  }\n\n  tags {\n\tgroup = \"Cloudera-Cluster\"\n  }\n}<\/pre>\n<p class=\"brush: bash; gutter: true; first-line: 1\">\u00a0Next we will create the private \/ public IP for our instances.<\/p>\n<h4 class=\"brush: bash; gutter: true; first-line: 1\">ip.tf<\/h4>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">resource \"azurerm_public_ip\" \"manager_pip\" {\nname = \"cdh-pip\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\npublic_ip_address_allocation = \"static\"\n\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\nresource \"azurerm_network_interface\" \"public_nic\" {\nname = \"manager_network_interface\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\nnetwork_security_group_id = \"${azurerm_network_security_group.nsg_cluster.id}\"\n\nip_configuration {\nname = \"manager_ip\"\nsubnet_id = \"${azurerm_subnet.subnet_1.id}\"\nprivate_ip_address_allocation = \"dynamic\"\npublic_ip_address_id = \"${azurerm_public_ip.manager_pip.id}\"\n}\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\n\nresource \"azurerm_public_ip\" \"master_pip\" {\nname = \"cdh-pip1\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\npublic_ip_address_allocation = \"static\"\n\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\nresource \"azurerm_network_interface\" \"public_nic1\" {\nname = \"master_network_interface\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\nnetwork_security_group_id = \"${azurerm_network_security_group.nsg_cluster.id}\"\n\nip_configuration {\nname = \"master_ip\"\nsubnet_id = \"${azurerm_subnet.subnet_2.id}\"\nprivate_ip_address_allocation = \"dynamic\"\npublic_ip_address_id = \"${azurerm_public_ip.master_pip.id}\"\n}\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\n\nresource \"azurerm_public_ip\" \"worker1_pip\" {\nname = \"cdh-pip2\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\npublic_ip_address_allocation = \"static\"\n\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\nresource \"azurerm_network_interface\" \"public_nic2\" {\nname = \"worker1_network_interface\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\nnetwork_security_group_id = \"${azurerm_network_security_group.nsg_cluster.id}\"\n\nip_configuration {\nname = \"worker1_ip\"\nsubnet_id = \"${azurerm_subnet.subnet_2.id}\"\nprivate_ip_address_allocation = \"dynamic\"\npublic_ip_address_id = \"${azurerm_public_ip.worker1_pip.id}\"\n}\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\n\nresource \"azurerm_public_ip\" \"worker2_pip\" {\nname = \"cdh-pip3\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\npublic_ip_address_allocation = \"static\"\n\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\nresource \"azurerm_network_interface\" \"public_nic3\" {\nname = \"worker2_network_interface\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\nnetwork_security_group_id = \"${azurerm_network_security_group.nsg_cluster.id}\"\n\nip_configuration {\nname = \"worker2_ip\"\nsubnet_id = \"${azurerm_subnet.subnet_2.id}\"\nprivate_ip_address_allocation = \"dynamic\"\npublic_ip_address_id = \"${azurerm_public_ip.worker2_pip.id}\"\n}\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\n\nresource \"azurerm_public_ip\" \"worker3_pip\" {\nname = \"cdh-pip4\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\npublic_ip_address_allocation = \"static\"\n\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}\n\nresource \"azurerm_network_interface\" \"public_nic4\" {\nname = \"worker3_network_interface\"\nlocation = \"${var.location}\"\nresource_group_name = \"${azurerm_resource_group.terraform_rg.name}\"\nnetwork_security_group_id = \"${azurerm_network_security_group.nsg_cluster.id}\"\n\nip_configuration {\nname = \"worker3_ip\"\nsubnet_id = \"${azurerm_subnet.subnet_2.id}\"\nprivate_ip_address_allocation = \"dynamic\"\npublic_ip_address_id = \"${azurerm_public_ip.worker3_pip.id}\"\n}\ntags {\ngroup = \"Cloudera-Cluster\"\n}\n}<\/pre>\n<p>Once the network, storage and the security group are configured we can now provision our VM instances with the following configuration:<\/p>\n<ul>\n<li>5 instances<\/li>\n<li>Master and Manager VM size: Standard_DS3_v2<\/li>\n<li>Worker VM size: Standard_DS2_v2<\/li>\n<li>Centos 7.3<\/li>\n<li>1 OS disk + 1 data disk of 100GB<\/li>\n<\/ul>\n<h4>vm.tf<\/h4>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">resource \"azurerm_virtual_machine\" \"la_manager\" {\n  name                  = \"cdh_manager\"\n  location              = \"${var.location}\"\n  resource_group_name   = \"${azurerm_resource_group.terraform_rg.name}\"\n  network_interface_ids = [\"${azurerm_network_interface.public_nic.id}\"]\n  vm_size               = \"Standard_DS3_v2\"\n\n#This will delete the OS disk automatically when deleting the VM\n  delete_os_disk_on_termination = true\n\n  storage_image_reference {\n    publisher = \"OpenLogic\"\n    offer     = \"CentOS\"\n    sku       = \"7.3\"\n    version   = \"latest\"\n\n  }\n\n  storage_os_disk {\n    name          = \"osdisk-1\"\n    vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/osdisk-1.vhd\"\n    caching       = \"ReadWrite\"\n    create_option = \"FromImage\"\n  }\n\n  # Optional data disks\n    storage_data_disk {\n      name          = \"data\"\n      vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/data1.vhd\"\n      disk_size_gb  = \"100\"\n      create_option = \"Empty\"\n      lun           = 0\n    }\n\n  os_profile {\n    computer_name  = \"manager\"\n    admin_username = \"${var.vm_username_manager}\"\n    admin_password = \"${var.vm_password}\"\n  }\n\n  os_profile_linux_config {\n    disable_password_authentication = false\n  }\n\n  tags {\n    group = \"Cloudera-Cluster\"\n  }\n}\n\n\n# Master (High Availability)\nresource \"azurerm_virtual_machine\" \"la_master\" {\n  name                  = \"cdh_master\"\n  location              = \"${var.location}\"\n  resource_group_name   = \"${azurerm_resource_group.terraform_rg.name}\"\n  network_interface_ids = [\"${azurerm_network_interface.public_nic1.id}\"]\n  vm_size               = \"Standard_DS3_v2\"\n\n#This will delete the OS disk automatically when deleting the VM\n  delete_os_disk_on_termination = true\n\n  storage_image_reference {\n    publisher = \"OpenLogic\"\n    offer     = \"CentOS\"\n    sku       = \"7.3\"\n    version   = \"latest\"\n\n  }\n\n  storage_os_disk {\n    name          = \"osdisk-2\"\n    vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/osdisk-2.vhd\"\n    caching       = \"ReadWrite\"\n    create_option = \"FromImage\"\n  }\n\n  # Optional data disks\n    storage_data_disk {\n      name          = \"data\"\n      vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/data2.vhd\"\n      disk_size_gb  = \"100\"\n      create_option = \"Empty\"\n      lun           = 0\n    }\n\n  os_profile {\n    computer_name  = \"master\"\n    admin_username = \"${var.vm_username_master}\"\n    admin_password = \"${var.vm_password}\"\n  }\n\n  os_profile_linux_config {\n    disable_password_authentication = false\n  }\n\n  tags {\n    group = \"Cloudera-Cluster\"\n  }\n}\n\n\n# Worker 1\nresource \"azurerm_virtual_machine\" \"la_worker1\" {\n  name                  = \"cdh_worker1\"\n  location              = \"${var.location}\"\n  resource_group_name   = \"${azurerm_resource_group.terraform_rg.name}\"\n  network_interface_ids = [\"${azurerm_network_interface.public_nic2.id}\"]\n  vm_size               = \"Standard_DS2_v2\"\n\n#This will delete the OS disk automatically when deleting the VM\n  delete_os_disk_on_termination = true\n\n  storage_image_reference {\n    publisher = \"OpenLogic\"\n    offer     = \"CentOS\"\n    sku       = \"7.3\"\n    version   = \"latest\"\n  }\n\n  storage_os_disk {\n    name          = \"osdisk-3\"\n    vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/osdisk-3.vhd\"\n    caching       = \"ReadWrite\"\n    create_option = \"FromImage\"\n  }\n\n  # Optional data disks\n    storage_data_disk {\n      name          = \"data\"\n      vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/data3.vhd\"\n      disk_size_gb  = \"100\"\n      create_option = \"Empty\"\n      lun           = 0\n    }\n\n  os_profile {\n    computer_name  = \"worker1\"\n    admin_username = \"${var.vm_username_worker1}\"\n    admin_password = \"${var.vm_password}\"\n  }\n\n  os_profile_linux_config {\n    disable_password_authentication = false\n  }\n\n  tags {\n    group = \"Cloudera-Cluster\"\n  }\n}\n\n# Worker 2\nresource \"azurerm_virtual_machine\" \"la_worker2\" {\n  name                  = \"cdh_worker2\"\n  location              = \"${var.location}\"\n  resource_group_name   = \"${azurerm_resource_group.terraform_rg.name}\"\n  network_interface_ids = [\"${azurerm_network_interface.public_nic3.id}\"]\n  vm_size               = \"Standard_DS2_v2\"\n\n#This will delete the OS disk automatically when deleting the VM\n  delete_os_disk_on_termination = true\n\n  storage_image_reference {\n    publisher = \"OpenLogic\"\n    offer     = \"CentOS\"\n    sku       = \"7.3\"\n    version   = \"latest\"\n  }\n\n  storage_os_disk {\n    name          = \"osdisk-4\"\n    vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/osdisk-4.vhd\"\n    caching       = \"ReadWrite\"\n    create_option = \"FromImage\"\n  }\n\n  # Optional data disks\n    storage_data_disk {\n      name          = \"data\"\n      vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/data4.vhd\"\n      disk_size_gb  = \"100\"\n      create_option = \"Empty\"\n      lun           = 0\n    }\n\n  os_profile {\n    computer_name  = \"worker2\"\n    admin_username = \"${var.vm_username_worker2}\"\n    admin_password = \"${var.vm_password}\"\n  }\n\n  os_profile_linux_config {\n    disable_password_authentication = false\n  }\n\n  tags {\n    group = \"Cloudera-Cluster\"\n  }\n}\n\n# Worker 3\nresource \"azurerm_virtual_machine\" \"la_worker3\" {\n  name                  = \"cdh_worker3\"\n  location              = \"${var.location}\"\n  resource_group_name   = \"${azurerm_resource_group.terraform_rg.name}\"\n  network_interface_ids = [\"${azurerm_network_interface.public_nic4.id}\"]\n  vm_size               = \"Standard_DS2_v2\"\n\n#This will delete the OS disk automatically when deleting the VM\n  delete_os_disk_on_termination = true\n\n  storage_image_reference {\n    publisher = \"OpenLogic\"\n    offer     = \"CentOS\"\n    sku       = \"7.3\"\n    version   = \"latest\"\n  }\n\n  storage_os_disk {\n    name          = \"osdisk-5\"\n    vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/osdisk-5.vhd\"\n    caching       = \"ReadWrite\"\n    create_option = \"FromImage\"\n  }\n\n  # Optional data disks\n    storage_data_disk {\n      name          = \"data\"\n      vhd_uri       = \"${azurerm_storage_account.storage.primary_blob_endpoint}${azurerm_storage_container.container.name}\/data5.vhd\"\n      disk_size_gb  = \"100\"\n      create_option = \"Empty\"\n      lun           = 0\n    }\n\n  os_profile {\n    computer_name  = \"worker3\"\n    admin_username = \"${var.vm_username_worker3}\"\n    admin_password = \"${var.vm_password}\"\n  }\n\n  os_profile_linux_config {\n    disable_password_authentication = false\n  }\n\n  tags {\n    group = \"Cloudera-Cluster\"\n  }\n}<\/pre>\n<h3>\u00a0Execution<\/h3>\n<p>We can now execute the following command from a shell environment. Note that all files must be placed in the same directory.<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">[root@centos Terraform]# terraform plan -var-file=var_values.tfvars\n<\/pre>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">[root@centos Terraform]# terraform apply -var-file=var_values.tfvars<\/pre>\n<p>&nbsp;<\/p>\n<p>After few minutes, check your resources in your Azure portal.<\/p>\n<p><a href=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screen-Shot-2018-04-10-at-11.49.24.png\"><img loading=\"lazy\" decoding=\"async\" class=\"size-full wp-image-22506 aligncenter\" src=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Screen-Shot-2018-04-10-at-11.49.24.png\" alt=\"Screen Shot 2018-04-10 at 11.49.24\" width=\"1940\" height=\"820\" \/><\/a><\/p>\n<p>To destroy the infrastructure, run the following command:<\/p>\n<pre class=\"brush: bash; gutter: true; first-line: 1\">[root@centos Terraform]# terraform destroy -var-file=var_values.tfvars<\/pre>\n<p>&nbsp;<\/p>\n<p>Once our infrastructure has been fully provisioned by Terraform we can start the installation of the Cloudera distribution of Hadoop with Ansible.<\/p>\n<p>&nbsp;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Deploying a Cloudera distribution of Hadoop automatically is very interesting in terms of time-saving. Infrastructure as Code tools such as Ansible, Puppet, Chef, Terraform, allow now to provision, manage and deploy configuration for large clusters. In this blog posts series, we will see how to deploy and install a CDH cluster with Terraform and Ansible [&hellip;]<\/p>\n","protected":false},"author":109,"featured_media":11080,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1183,955],"tags":[1338,399,1339,1340],"type_dbi":[],"class_list":["post-11079","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-big-data","category-cloud","tag-azure","tag-big-data","tag-cloudera","tag-terraform"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Deploy a Cloudera cluster with Terraform and Ansible in Azure - part 1 - dbi Blog<\/title>\n<meta name=\"description\" content=\"Automatic provisioning of a CDH cluster with Terraform and Ansible\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Deploy a Cloudera cluster with Terraform and Ansible in Azure - part 1\" \/>\n<meta property=\"og:description\" content=\"Automatic provisioning of a CDH cluster with Terraform and Ansible\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-04-10T08:13:33+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png\" \/>\n\t<meta property=\"og:image:width\" content=\"613\" \/>\n\t<meta property=\"og:image:height\" content=\"390\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"DevOps\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DevOps\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/\"},\"author\":{\"name\":\"DevOps\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735\"},\"headline\":\"Deploy a Cloudera cluster with Terraform and Ansible in Azure &#8211; part 1\",\"datePublished\":\"2018-04-10T08:13:33+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/\"},\"wordCount\":498,\"commentCount\":0,\"image\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png\",\"keywords\":[\"Azure\",\"Big Data\",\"Cloudera\",\"Terraform\"],\"articleSection\":[\"Big Data\",\"Cloud\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/\",\"name\":\"Deploy a Cloudera cluster with Terraform and Ansible in Azure - part 1 - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png\",\"datePublished\":\"2018-04-10T08:13:33+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735\"},\"description\":\"Automatic provisioning of a CDH cluster with Terraform and Ansible\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#primaryimage\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png\",\"contentUrl\":\"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png\",\"width\":613,\"height\":390},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Deploy a Cloudera cluster with Terraform and Ansible in Azure &#8211; part 1\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735\",\"name\":\"DevOps\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"caption\":\"DevOps\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/devops\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Deploy a Cloudera cluster with Terraform and Ansible in Azure - part 1 - dbi Blog","description":"Automatic provisioning of a CDH cluster with Terraform and Ansible","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/","og_locale":"en_US","og_type":"article","og_title":"Deploy a Cloudera cluster with Terraform and Ansible in Azure - part 1","og_description":"Automatic provisioning of a CDH cluster with Terraform and Ansible","og_url":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/","og_site_name":"dbi Blog","article_published_time":"2018-04-10T08:13:33+00:00","og_image":[{"width":613,"height":390,"url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png","type":"image\/png"}],"author":"DevOps","twitter_card":"summary_large_image","twitter_misc":{"Written by":"DevOps","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/"},"author":{"name":"DevOps","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735"},"headline":"Deploy a Cloudera cluster with Terraform and Ansible in Azure &#8211; part 1","datePublished":"2018-04-10T08:13:33+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/"},"wordCount":498,"commentCount":0,"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png","keywords":["Azure","Big Data","Cloudera","Terraform"],"articleSection":["Big Data","Cloud"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/","url":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/","name":"Deploy a Cloudera cluster with Terraform and Ansible in Azure - part 1 - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#primaryimage"},"image":{"@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#primaryimage"},"thumbnailUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png","datePublished":"2018-04-10T08:13:33+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735"},"description":"Automatic provisioning of a CDH cluster with Terraform and Ansible","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#primaryimage","url":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png","contentUrl":"https:\/\/www.dbi-services.com\/blog\/wp-content\/uploads\/sites\/2\/2022\/04\/Azure_architecture.png","width":613,"height":390},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/deploy-a-cloudera-cluster-with-terraform-and-ansible-in-azure-part-1\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Deploy a Cloudera cluster with Terraform and Ansible in Azure &#8211; part 1"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735","name":"DevOps","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","caption":"DevOps"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/devops\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11079","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/109"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=11079"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11079\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media\/11080"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=11079"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=11079"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=11079"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=11079"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}