Implement Hadoop Platform - Initiate HDFS

Configuration Files
/opt/hadoop-2.8.2/etc/hadoop/core-site.xml
hdfs-site.xml, slaves, hadoop-env.sh

core-site.xml (Add blod texts)
:::
<!-- Put site-specific property overrides in this file. -->

<configuration>
        <property>
                <name>fs.defaultFS</name> 
                <value>hdfs://nn:8020</value>
        </property>
</configuration>

(指定NameNode主機)

Refer: http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/core-default.xml

hdfs-site.xml(Add blod texts)

:::

<!-- Put site-specific property overrides in this file. -->

<configuration>

        <property>

                <name>dfs.replication</name>

                <value>2</value>

        </property>

        <property>

                <name>dfs.namenode.name.dir</name>

                <value>file:/home/ubuntu/nn</value>

        </property>

        <property>

                <name>dfs.datanode.data.dir</name>

                <value>file:/home/ubuntu/dn</value>

        </property>

</configuration>

(指定複本數,namenode and datanode 本地目錄)

Refer: http://hadoop.apache.org/docs/r2.8.2/hadoop-project-dist/hadoop-common/DeprecatedProperties.html

slaves

ubuntu@ip-172-31-16-58:/opt$ sudo more /opt/hadoop-2.8.2/etc/hadoop/slaves

#localhost

dn01 

dn02

(指定DataNode主機)

 sudo more /opt/hadoop-2.8.2/etc/hadoop/hadoop-env.sh

:::

export JAVA_HOME=/opt/jre1.8.0_151

:::

# The maximum amount of heap to use, in MB. Default is 1000.

#export HADOOP_HEAPSIZE=

 export HADOOP_HEAPSIZE=256

:::

# Where log files are stored.  $HADOOP_HOME/logs by default.

export HADOOP_LOG_DIR=/tmp

初始化HDFS分散式檔案系統

ubuntu@nn:~$ hdfs namenode -format

17/12/08 08:00:34 INFO namenode.NameNode: STARTUP_MSG:

/************************************************************

STARTUP_MSG: Starting NameNode

STARTUP_MSG:   user = ubuntu

STARTUP_MSG:   host = nn/172.16.1.200

STARTUP_MSG:   args = [-format]

STARTUP_MSG:   version = 2.8.2

STARTUP_MSG:   classpath = /opt/hadoop-2.8.2/etc/hadoop:/opt/hadoop-2.8.2/share/hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop-2.8.2/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop-2.8.2/share/hadoop/comm

:::

17/12/08 08:00:36 INFO util.GSet: capacity      = 2^13 = 8192 entries

17/12/08 08:00:36 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1002866620-172.16.1.200-1512720036248

17/12/08 08:00:36 INFO common.Storage: Storage directory /home/ubuntu/nn has been successfully formatted.

17/12/08 08:00:36 INFO namenode.FSImageFormatProtobuf: Saving image file /home/ubuntu/nn/current/fsimage.ckpt_0000000000000000000 using no compression

17/12/08 08:00:36 INFO namenode.FSImageFormatProtobuf: Image file /home/ubuntu/nn/current/fsimage.ckpt_0000000000000000000 of size 323 bytes saved in 0 seconds.

17/12/08 08:00:36 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0

17/12/08 08:00:36 INFO util.ExitUtil: Exiting with status 0

17/12/08 08:00:36 INFO namenode.NameNode: SHUTDOWN_MSG:

/************************************************************

SHUTDOWN_MSG: Shutting down NameNode at nn/172.16.1.200

************************************************************/

Check HDFS Metadata directory.

ubuntu@nn:~$ tree nn

nn

└── current

    ├── fsimage_0000000000000000000

    ├── fsimage_0000000000000000000.md5

    ├── seen_txid

    └── VERSION

1 directory, 4 files

啟動HDFS

ubuntu@nn:~$ . /opt/hadoop-2.8.2/sbin/start-dfs.sh

Illegal option -b

-bash: cd: Usage: /usr/bin: No such file or directory

Starting namenodes on [nn]

nn: starting namenode, logging to /tmp/hadoop-ubuntu-namenode-nn.out

dn02: Warning: Permanently added 'dn02,172.16.1.211' (ECDSA) to the list of known hosts.

dn01: starting datanode, logging to /tmp/hadoop-ubuntu-datanode-dn01.out

dn02: starting datanode, logging to /tmp/hadoop-ubuntu-datanode-dn02.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.

0.0.0.0: starting secondarynamenode, logging to /tmp/hadoop-ubuntu-secondarynamenode-nn.out

Check clusterID

ubuntu@nn:~$ cat nn/current/VERSION

#Fri Dec 08 08:00:36 UTC 2017

namespaceID=385737814

clusterID=CID-b11e5852-2eed-4685-b3d0-4e6ddfd7ca1e

cTime=1512720036248

storageType=NAME_NODE

blockpoolID=BP-1002866620-172.16.1.200-1512720036248

layoutVersion=-63

ubuntu@dn01:~$ cat dn/current/VERSION

#Fri Dec 08 08:22:00 UTC 2017

storageID=DS-d8538572-f70d-49ab-b15e-032b1a2fb7d3

clusterID=CID-b11e5852-2eed-4685-b3d0-4e6ddfd7ca1e

cTime=0

datanodeUuid=123328bb-4a7b-40aa-a0b5-fefe1145eaaa

storageType=DATA_NODE

layoutVersion=-57

ubuntu@dn02:~$ cat dn/current/VERSION

#Fri Dec 08 08:22:00 UTC 2017

storageID=DS-eeff478f-03b8-4958-97ef-6e1c8a30d42b

clusterID=CID-b11e5852-2eed-4685-b3d0-4e6ddfd7ca1e

cTime=0

datanodeUuid=2dc2bcf4-7c6c-4fe4-a58c-125f2bacb93b

storageType=DATA_NODE

layoutVersion=-57

停止HDFS

ubuntu@nn:~$ . /opt/hadoop-2.8.2/sbin/stop-dfs.sh

Illegal option -b

-bash: cd: Usage: /usr/bin: No such file or directory

Stopping namenodes on [nn]

nn: stopping namenode

dn02: stopping datanode

dn01: stopping datanode

Stopping secondary namenodes [0.0.0.0]

0.0.0.0: stopping secondarynamenode