Let’s assume we need to create an HDFS home directory for a user named “dbitest”.<\/p>\n
We need first to verify if the user exists on the local filesystem. It’s important to understand that HDFS is mapping users from the local filesystem.<\/p>\n
[cdhtest@master ~]$ cat \/etc\/passwd | grep dbitest<\/pre>\n\u00a0Create a user on the local file system<\/h4>\n
When the user is not created, we can easily create one with it associated group.<\/p>\n
[cdhtest@master ~]$ sudo groupadd dbitest\n\n[cdhtest@master ~]$ sudo useradd -g dbitest -d\/home\/dbitest dbitest\n\n[cdhtest@master ~]$ cat \/etc\/passwd | grep dbitest\ndbitest:x:1002:1002::\/home\/dbitest:\/bin\/bash\n[cdhtest@master ~]$<\/pre>\nNote that, the user dbitest should be created in all cluster hosts.<\/p>\n
Create a directory in HDFS for a new user<\/h4>\n
Then we can create the directory under \/user in HDFS for the new user dbitest. This directory needs to be created using hdfs user as hdfs user is the super user for admin commands.<\/p>\n
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -mkdir \/user\/dbitest<\/pre>\n<\/p>\n
Verify the owner for our new directory<\/h4>\n
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -ls \/user\nFound 5 items\ndrwxr-xr-x\u00a0\u00a0 - hdfs\u00a0\u00a0 supergroup\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0 2018-07-10 10:10 \/user\/dbitest\ndrwxrwxrwx\u00a0\u00a0 - mapred hadoop\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0 2018-07-10 07:54 \/user\/history\ndrwxrwxr-t\u00a0\u00a0 - hive\u00a0\u00a0 hive\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0 2018-07-10 07:55 \/user\/hive\ndrwxrwxr-x\u00a0\u00a0 - hue\u00a0\u00a0\u00a0 hue\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0 2018-07-10 07:55 \/user\/hue\ndrwxrwxr-x\u00a0\u00a0 - oozie\u00a0 oozie\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0 2018-07-10 07:56 \/user\/oozie<\/pre>\nThe new home directory has been created but it’s owned by hdfs user.<\/p>\n
Change owner for \/user\/dbitest directory<\/h4>\n
Use the below command to change the owner of the new user home directory created.<\/p>\n
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -chown dbitest:dbitest \/user\/dbitest<\/pre>\nLet’s see if the owner has changed.<\/p>\n
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -ls \/user\nFound 5 items\ndrwxr-xr-x - dbitest dbitest 0 2018-07-10 10:10 \/user\/dbitest\ndrwxrwxrwx - mapred hadoop 0 2018-07-10 07:54 \/user\/history\ndrwxrwxr-t - hive hive 0 2018-07-10 07:55 \/user\/hive\ndrwxrwxr-x - hue hue 0 2018-07-10 07:55 \/user\/hue\ndrwxrwxr-x - oozie oozie 0 2018-07-10 07:56 \/user\/oozie<\/pre>\n<\/h4>\n
Change permissions<\/h4>\n
Change the permissions of the newly created home directory so that no other users can have read, write and execute permissions except the owner.<\/p>\n
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -chmod 700 \/user\/dbitest<\/pre>\n[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -ls \/user\nFound 6 items\ndrwxr-xr-x - admins cdhtest 0 2018-07-10 08:56 \/user\/cdhtest\ndrwx------ - dbitest dbitest 0 2018-07-10 10:10 \/user\/dbitest\ndrwxrwxrwx - mapred hadoop 0 2018-07-10 07:54 \/user\/history\ndrwxrwxr-t - hive hive 0 2018-07-10 07:55 \/user\/hive\ndrwxrwxr-x - hue hue 0 2018-07-10 07:55 \/user\/hue\ndrwxrwxr-x - oozie oozie 0 2018-07-10 07:56 \/user\/oozie<\/pre>\n<\/p>\n
Test the user dbitest home directory<\/h4>\n
We can now test the user home directory creation by uploading data into it without specifying the destination directory. The file will be automatically uploaded to the user’s home directory if no destination is specified.<\/p>\n
[cdhtest@master ~]$ sudo su dbitest<\/pre>\n[dbitest@master ~]$ hdfs dfs -ls \/user\/dbitest<\/pre>\n[dbitest@master ~]$ hdfs dfs -put HelloWorld.txt<\/pre>\n[dbitest@master ~]$ hdfs dfs -ls \/user\/dbitest\nFound 1 items\n-rw-r--r-- 3 dbitest dbitest 39 2018-07-10 10:30 \/user\/dbitest\/HelloWorld.txt<\/pre>\n<\/p>\n
Your user home directory has been created successfully.<\/p>\n","protected":false},"excerpt":{"rendered":"
Let’s assume we need to create an HDFS home directory for a user named “dbitest”. We need first to verify if the user exists on the local filesystem. It’s important to understand that HDFS is mapping users from the local filesystem. [cdhtest@master ~]$ cat \/etc\/passwd | grep dbitest \u00a0Create a user on the local file […]<\/p>\n","protected":false},"author":109,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1183],"tags":[1396,1397],"type_dbi":[],"class_list":["post-11403","post","type-post","status-publish","format-standard","hentry","category-big-data","tag-hadoop","tag-hdfs"],"acf":[],"yoast_head":"\n
Create an HDFS user's home directory - dbi Blog<\/title>\n<meta name=\"description\" content=\"How to create a user home directory in hdfs\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Create an HDFS user's home directory\" \/>\n<meta property=\"og:description\" content=\"How to create a user home directory in hdfs\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/\" \/>\n<meta property=\"og:site_name\" content=\"dbi Blog\" \/>\n<meta property=\"article:published_time\" content=\"2018-07-10T10:07:54+00:00\" \/>\n<meta name=\"author\" content=\"DevOps\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"DevOps\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/\"},\"author\":{\"name\":\"DevOps\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735\"},\"headline\":\"Create an HDFS user’s home directory\",\"datePublished\":\"2018-07-10T10:07:54+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/\"},\"wordCount\":257,\"commentCount\":0,\"keywords\":[\"hadoop\",\"hdfs\"],\"articleSection\":[\"Big Data\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/\",\"name\":\"Create an HDFS user's home directory - dbi Blog\",\"isPartOf\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\"},\"datePublished\":\"2018-07-10T10:07:54+00:00\",\"author\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735\"},\"description\":\"How to create a user home directory in hdfs\",\"breadcrumb\":{\"@id\":\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Accueil\",\"item\":\"https:\/\/www.dbi-services.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Create an HDFS user’s home directory\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#website\",\"url\":\"https:\/\/www.dbi-services.com\/blog\/\",\"name\":\"dbi Blog\",\"description\":\"\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735\",\"name\":\"DevOps\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g\",\"caption\":\"DevOps\"},\"url\":\"https:\/\/www.dbi-services.com\/blog\/author\/devops\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Create an HDFS user's home directory - dbi Blog","description":"How to create a user home directory in hdfs","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/","og_locale":"en_US","og_type":"article","og_title":"Create an HDFS user's home directory","og_description":"How to create a user home directory in hdfs","og_url":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/","og_site_name":"dbi Blog","article_published_time":"2018-07-10T10:07:54+00:00","author":"DevOps","twitter_card":"summary_large_image","twitter_misc":{"Written by":"DevOps","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/#article","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/"},"author":{"name":"DevOps","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735"},"headline":"Create an HDFS user’s home directory","datePublished":"2018-07-10T10:07:54+00:00","mainEntityOfPage":{"@id":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/"},"wordCount":257,"commentCount":0,"keywords":["hadoop","hdfs"],"articleSection":["Big Data"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/","url":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/","name":"Create an HDFS user's home directory - dbi Blog","isPartOf":{"@id":"https:\/\/www.dbi-services.com\/blog\/#website"},"datePublished":"2018-07-10T10:07:54+00:00","author":{"@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735"},"description":"How to create a user home directory in hdfs","breadcrumb":{"@id":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.dbi-services.com\/blog\/create-an-hdfs-users-home-directory\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Accueil","item":"https:\/\/www.dbi-services.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Create an HDFS user’s home directory"}]},{"@type":"WebSite","@id":"https:\/\/www.dbi-services.com\/blog\/#website","url":"https:\/\/www.dbi-services.com\/blog\/","name":"dbi Blog","description":"","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.dbi-services.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/www.dbi-services.com\/blog\/#\/schema\/person\/4cd1b5f8a3de93f05a16ab8d7d2b7735","name":"DevOps","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/cdd2dd7441774355062c0f0f68612296b059cd1e2ff6c7af0b15dba0ed64a85f?s=96&d=mm&r=g","caption":"DevOps"},"url":"https:\/\/www.dbi-services.com\/blog\/author\/devops\/"}]}},"_links":{"self":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11403","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/users\/109"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/comments?post=11403"}],"version-history":[{"count":0,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/posts\/11403\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/media?parent=11403"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/categories?post=11403"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/tags?post=11403"},{"taxonomy":"type","embeddable":true,"href":"https:\/\/www.dbi-services.com\/blog\/wp-json\/wp\/v2\/type_dbi?post=11403"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}