Create an HDFS user’s home directory

Let’s assume we need to create an HDFS home directory for a user named “dbitest”.

We need first to verify if the user exists on the local filesystem. It’s important to understand that HDFS is mapping users from the local filesystem.

[[email protected] ~]$ cat /etc/passwd | grep dbitest

 Create a user on the local file system

When the user is not created, we can easily create one with it associated group.

[[email protected] ~]$ sudo groupadd dbitest

[[email protected] ~]$ sudo useradd -g dbitest -d/home/dbitest dbitest

[[email protected] ~]$ cat /etc/passwd | grep dbitest
dbitest:x:1002:1002::/home/dbitest:/bin/bash
[[email protected] ~]$

Note that, the user dbitest should be created in all cluster hosts.

Create a directory in HDFS for a new user

Then we can create the directory under /user in HDFS for the new user dbitest. This directory needs to be created using hdfs user as hdfs user is the super user for admin commands.

[[email protected] ~]$ sudo -u hdfs hdfs dfs -mkdir /user/dbitest

Verify the owner for our new directory

[[email protected] ~]$ sudo -u hdfs hdfs dfs -ls /user
Found 5 items
drwxr-xr-x   - hdfs   supergroup          0 2018-07-10 10:10 /user/dbitest
drwxrwxrwx   - mapred hadoop              0 2018-07-10 07:54 /user/history
drwxrwxr-t   - hive   hive                0 2018-07-10 07:55 /user/hive
drwxrwxr-x   - hue    hue                 0 2018-07-10 07:55 /user/hue
drwxrwxr-x   - oozie  oozie               0 2018-07-10 07:56 /user/oozie

The new home directory has been created but it’s owned by hdfs user.

Change owner for /user/dbitest directory

Use the below command to change the owner of the new user home directory created.

[[email protected] ~]$ sudo -u hdfs hdfs dfs -chown dbitest:dbitest /user/dbitest

Let’s see if the owner has changed.

[[email protected] ~]$ sudo -u hdfs hdfs dfs -ls /user
Found 5 items
drwxr-xr-x   - dbitest dbitest          0 2018-07-10 10:10 /user/dbitest
drwxrwxrwx   - mapred  hadoop           0 2018-07-10 07:54 /user/history
drwxrwxr-t   - hive    hive             0 2018-07-10 07:55 /user/hive
drwxrwxr-x   - hue     hue              0 2018-07-10 07:55 /user/hue
drwxrwxr-x   - oozie   oozie            0 2018-07-10 07:56 /user/oozie

Change permissions

Change the permissions of the newly created home directory so that no other users can have read, write and execute permissions except the owner.

[[email protected] ~]$ sudo -u hdfs hdfs dfs -chmod 700 /user/dbitest

[[email protected] ~]$ sudo -u hdfs hdfs dfs -ls /user
Found 6 items
drwxr-xr-x   - admins  cdhtest          0 2018-07-10 08:56 /user/cdhtest
drwx------   - dbitest dbitest          0 2018-07-10 10:10 /user/dbitest
drwxrwxrwx   - mapred  hadoop           0 2018-07-10 07:54 /user/history
drwxrwxr-t   - hive    hive             0 2018-07-10 07:55 /user/hive
drwxrwxr-x   - hue     hue              0 2018-07-10 07:55 /user/hue
drwxrwxr-x   - oozie   oozie            0 2018-07-10 07:56 /user/oozie

Test the user dbitest home directory

We can now test the user home directory creation by uploading data into it without specifying the destination directory. The file will be automatically uploaded to the user’s home directory if no destination is specified.

[[email protected] ~]$ sudo su dbitest

[[email protected] ~]$ hdfs dfs -ls /user/dbitest

[[email protected] ~]$ hdfs dfs -put HelloWorld.txt

[[email protected] ~]$ hdfs dfs -ls /user/dbitest
Found 1 items
-rw-r--r--   3 dbitest dbitest         39 2018-07-10 10:30 /user/dbitest/HelloWorld.txt

Your user home directory has been created successfully.