Implement Hadoop Platform

Hadoop Hosts List

1. NameNode (172.16.1.200)
2. ResourceManager(172.16.1.201)
3. DataNode1 (172.16.1.210)
4. DataNode2 (172.16.1.211)
5. NodeManager

Download  JAVA JDK package
ubuntu@physical:~$ sudo wget "http://javadl.oracle.com/webapps/download/AutoDL?BundleId=227542_e758a0de34e24606bca991d704f6dcbf" -O /opt/jdk-8u151-linux-x64.tar.gz

Download Hadoop
ubuntu@physical:~$ sudo wget http://ftp.twaren.net/Unix/Web/apache/hadoop/common/hadoop-2.9.0/hadoop-2.9.0.tar.gz

Unzip Packages
ubuntu@physical:~$ sudo tar xvfz jdk-8u151-linux-x64.tar.gz -C /opt

ubuntu@physical:~$ sudo tar xvfz hadoop-2.9.0.tar.gz -C /opt

ubuntu@myHDP2:~$ ll /opt
total 78168
drwxr-xr-x  4 root   root       4096 Dec  6 08:44 ./
drwxr-xr-x 21 root   root       4096 Dec  6 08:20 ../
drwxr-xr-x  9 ubuntu ubuntu     4096 Nov 13 23:28 hadoop-2.9.0/
-rw-r--r--  1 root   root   80025700 Sep 14 18:15 jdk-8u151-linux-x64.tar.gz
drwxr-xr-x  6 uucp      143     4096 Sep  6 02:29 jre1.8.0_151/

(可以實際下載到Container主機,或是從外部主機掛載共用方式,請參考另一Blog文: Mount Physical Host file system to Container)

Configure Environment to edit .bashrc
ubuntu@myHDP2:~$ sudo cat .bashrc
:::
alias ping='ping -c 4'
alias bye='sudo shutdown -h now'

export JAVA_HOME=/opt/jre1.8.0_151
export HADOOP_HOME=/opt/hadoop-2.9.0
export PATH=$PATH:$JAVA_HOME/bin
export PATH=$PATH:$HADOOP_HOME/bin:HADOOP_HOME/sbin
<end of file>

Logout and Login
確認Container不僅可以看到外部Physical Machine /opt files, 也能夠使用兩個基礎套件
ubuntu@myHDP2:~$ java -version
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)

ubuntu@myHDP2:~$ hadoop version
Hadoop 2.8.2
Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 66c47f2a01ad9637879e95f80c41f798373828fb
Compiled by jdu on 2017-10-19T20:39Z
Compiled with protoc 2.5.0
From source with checksum dce55e5afe30c210816b39b631a53b1d
This command was run using /opt/hadoop-2.8.2/share/hadoop/common/hadoop-common-2.8.2.jar\

ubuntu@ip-172-31-16-58:~$ sudo lxc-clone myHDP2 rn/nn/dn01/dn02 (複製主機群)
lxc-clone is deprecated in favor of lxc-copy.
Created container dn02 as copy of myHDP2

ubuntu@ip-172-31-16-58:~$ sudo lxc-ls -f

Startup all Container machines
Edit ip address /etc/network/interfaces (因為是copy from the same one template)
Verify JAVA and HADOOP version if work

Verify SSH login to each other 不用敲密碼, 有問題!? 還是要敲密碼 12/07
+Permission denied (publickey)

要關閉使用密碼登入, 只允許使用 Key 來做登入, 可修改此檔案的下述兩個參數: vim /etc/ssh/sshd_config

• PubkeyAuthentication yes
• PasswordAuthentication no