Let’s assume we need to create an HDFS home directory for a user named “dbitest”.
We need first to verify if the user exists on the local filesystem. It’s important to understand that HDFS is mapping users from the local filesystem.
[cdhtest@master ~]$ cat /etc/passwd | grep dbitest
Create a user on the local file system
When the user is not created, we can easily create one with it associated group.
[cdhtest@master ~]$ sudo groupadd dbitest [cdhtest@master ~]$ sudo useradd -g dbitest -d/home/dbitest dbitest [cdhtest@master ~]$ cat /etc/passwd | grep dbitest dbitest:x:1002:1002::/home/dbitest:/bin/bash [cdhtest@master ~]$
Note that, the user dbitest should be created in all cluster hosts.
Create a directory in HDFS for a new user
Then we can create the directory under /user in HDFS for the new user dbitest. This directory needs to be created using hdfs user as hdfs user is the super user for admin commands.
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -mkdir /user/dbitest
Verify the owner for our new directory
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -ls /user Found 5 items drwxr-xr-x - hdfs supergroup 0 2018-07-10 10:10 /user/dbitest drwxrwxrwx - mapred hadoop 0 2018-07-10 07:54 /user/history drwxrwxr-t - hive hive 0 2018-07-10 07:55 /user/hive drwxrwxr-x - hue hue 0 2018-07-10 07:55 /user/hue drwxrwxr-x - oozie oozie 0 2018-07-10 07:56 /user/oozie
The new home directory has been created but it’s owned by hdfs user.
Change owner for /user/dbitest directory
Use the below command to change the owner of the new user home directory created.
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -chown dbitest:dbitest /user/dbitest
Let’s see if the owner has changed.
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -ls /user Found 5 items drwxr-xr-x - dbitest dbitest 0 2018-07-10 10:10 /user/dbitest drwxrwxrwx - mapred hadoop 0 2018-07-10 07:54 /user/history drwxrwxr-t - hive hive 0 2018-07-10 07:55 /user/hive drwxrwxr-x - hue hue 0 2018-07-10 07:55 /user/hue drwxrwxr-x - oozie oozie 0 2018-07-10 07:56 /user/oozie
Change permissions
Change the permissions of the newly created home directory so that no other users can have read, write and execute permissions except the owner.
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -chmod 700 /user/dbitest
[cdhtest@master ~]$ sudo -u hdfs hdfs dfs -ls /user Found 6 items drwxr-xr-x - admins cdhtest 0 2018-07-10 08:56 /user/cdhtest drwx------ - dbitest dbitest 0 2018-07-10 10:10 /user/dbitest drwxrwxrwx - mapred hadoop 0 2018-07-10 07:54 /user/history drwxrwxr-t - hive hive 0 2018-07-10 07:55 /user/hive drwxrwxr-x - hue hue 0 2018-07-10 07:55 /user/hue drwxrwxr-x - oozie oozie 0 2018-07-10 07:56 /user/oozie
Test the user dbitest home directory
We can now test the user home directory creation by uploading data into it without specifying the destination directory. The file will be automatically uploaded to the user’s home directory if no destination is specified.
[cdhtest@master ~]$ sudo su dbitest
[dbitest@master ~]$ hdfs dfs -ls /user/dbitest
[dbitest@master ~]$ hdfs dfs -put HelloWorld.txt
[dbitest@master ~]$ hdfs dfs -ls /user/dbitest Found 1 items -rw-r--r-- 3 dbitest dbitest 39 2018-07-10 10:30 /user/dbitest/HelloWorld.txt
Your user home directory has been created successfully.