Illegal option -b
-bash: cd: Usage: /usr/bin: No such file or directory?
Starting namenodes on [nn]
nn: starting namenode, logging to /tmp/hadoop-ubuntu-namenode-nn.out
dn02: starting datanode, logging to /tmp/hadoop-ubuntu-datanode-dn02.out
dn01: starting datanode, logging to /tmp/hadoop-ubuntu-datanode-dn01.out
Starting secondary namenodes [0.0.0.0]
0.0.0.0: starting secondarynamenode, logging to /tmp/hadoop-ubuntu-secondarynamenode-nn.out
List
ubuntu@nn:~$ hdfs dfs -ls /
Create Folder
ubuntu@nn:~$ hdfs dfs -mkdir /Big 建立hdfs目錄
ubuntu@nn:~$ hdfs dfs -ls /
Found 1 items
drwxr-xr-x - ubuntu supergroup 0 2017-12-08 10:34 /Big
Copy file from local
ubuntu@nn:~$ hdfs dfs -copyFromLocal .bash*
List
ubuntu@nn:~$ hdfs dfs -ls /Big
Found 3 items
-rw-r--r-- 2 ubuntu supergroup 10427 2017-12-08 10:37 /Big/.bash_history
-rw-r--r-- 2 ubuntu supergroup 220 2017-12-08 10:37 /Big/.bash_logout
-rw-r--r-- 2 ubuntu supergroup 3986 2017-12-08 10:37 /Big/.bashrc
ubuntu@nn:~$ hdfs dfsadmin -printTopology 檢查datanode
Rack: /default-rack
172.16.1.210:50010 (dn01)
172.16.1.211:50010 (dn02)
# file: /
# owner: ubuntu
# group: supergroup
user::rwx
group::r-x
other::r-x
顯示檔案內容
ubuntu@nn:~$ hdfs dfs -cat /Big/.bashrc | head -n 5
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
取回檔案, copy file from hdfs to local file system
ubuntu@nn:~$ hdfs dfs -get /Big/.bashrc /tmp/test.txt
Delet file
ubuntu@nn:~$ hdfs dfs -rm /Big/.bashrc
Deleted /Big/.bashrc
ubuntu@nn:~$ hdfs dfsadmin -printTopology 檢查datanode
Rack: /default-rack
172.16.1.210:50010 (dn01)
172.16.1.211:50010 (dn02)
檢視 HDFS 根目錄權限設定
ubuntu@nn:~$ hdfs dfs -getfacl /# file: /
# owner: ubuntu
# group: supergroup
user::rwx
group::r-x
other::r-x
顯示檔案內容
ubuntu@nn:~$ hdfs dfs -cat /Big/.bashrc | head -n 5
# ~/.bashrc: executed by bash(1) for non-login shells.
# see /usr/share/doc/bash/examples/startup-files (in the package bash-doc)
# for examples
# If not running interactively, don't do anything
取回檔案, copy file from hdfs to local file system
ubuntu@nn:~$ hdfs dfs -get /Big/.bashrc /tmp/test.txt
Delet file
ubuntu@nn:~$ hdfs dfs -rm /Big/.bashrc
Deleted /Big/.bashrc
Delete directory
ubuntu@nn:~$ hdfs dfs -rm -r /Big
Deleted /Big
ubuntu@nn:~$ hdfs dfsadmin -report
Configured Capacity: 51908788224 (48.34 GB)
Present Capacity: 31196621972 (29.05 GB)
DFS Remaining: 31196504064 (29.05 GB)
DFS Used: 117908 (115.14 KB)
DFS Used%: 0.00%
Under replicated blocks: 0
Blocks with corrupt replicas: 0
Missing blocks: 0
Missing blocks (with replication factor 1): 0
Pending deletion blocks: 0
-------------------------------------------------
Live datanodes (2):
Name: 172.16.1.210:50010 (dn01)
Hostname: dn01
Decommission Status : Normal
Configured Capacity: 25954394112 (24.17 GB)
DFS Used: 58954 (57.57 KB)
Non DFS Used: 10339305910 (9.63 GB)
DFS Remaining: 15598252032 (14.53 GB)
DFS Used%: 0.00%
DFS Remaining%: 60.10%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Dec 11 03:15:08 UTC 2017
Name: 172.16.1.211:50010 (dn02)
Hostname: dn02
Decommission Status : Normal
Configured Capacity: 25954394112 (24.17 GB)
DFS Used: 58954 (57.57 KB)
Non DFS Used: 10339305910 (9.63 GB)
DFS Remaining: 15598252032 (14.53 GB)
DFS Used%: 0.00%
DFS Remaining%: 60.10%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Dec 11 03:15:07 UTC 2017
檢查HDFS 檔案儲存資訊
ubuntu@nn:~$ hdfs fsck /test/.bashrc -files -blocks -locations
Connecting to namenode via http://nn:50070/fsck?ugi=ubuntu&files=1&blocks=1&locations=1&path=%2Ftest%2F.bashrc
FSCK started by ubuntu (auth:SIMPLE) from /172.16.1.200 for path /test/.bashrc at Mon Dec 11 03:19:32 UTC 2017
/test/.bashrc 3986 bytes, 1 block(s): OK
0. BP-1002866620-172.16.1.200-1512720036248:blk_1073741830_1006 len=3986 Live_repl=2 [DatanodeInfoWithStorage[172.16.1.210:50010,DS-d8538572-f70d-49ab-b15e-032b1a2fb7d3,DISK], DatanodeInfoWithStorage[172.16.1.211:50010,DS-eeff478f-03b8-4958-97ef-6e1c8a30d42b,DISK]]
Status: HEALTHY
Total size: 3986 B
Total dirs: 0
Total files: 1
Total symlinks: 0
Total blocks (validated): 1 (avg. block size 3986 B)
Minimally replicated blocks: 1 (100.0 %)
Over-replicated blocks: 0 (0.0 %)
Under-replicated blocks: 0 (0.0 %)
Mis-replicated blocks: 0 (0.0 %)
Default replication factor: 2
Average block replication: 2.0
Corrupt blocks: 0
Missing replicas: 0 (0.0 %)
Number of data-nodes: 2
Number of racks: 1
FSCK ended at Mon Dec 11 03:19:32 UTC 2017 in 2 milliseconds
The filesystem under path '/test/.bashrc' is HEALTHY
About Secondary NameNode: fsimage andd edits file
Check Secondary NameNode 預設的儲存目錄
Default /tmp, reboot 會清空
ubuntu@nn:~$ tree /tmp/hadoop-ubuntu/dfs/namesecondary/
/tmp/hadoop-ubuntu/dfs/namesecondary/
├── current
│ ├── edits_0000000000000000001-0000000000000000002
│ ├── edits_0000000000000000004-0000000000000000006
│ ├── edits_0000000000000000007-0000000000000000026
│ ├── edits_0000000000000000027-0000000000000000028
│ ├── edits_0000000000000000029-0000000000000000030
│ ├── edits_0000000000000000031-0000000000000000032
│ ├── edits_0000000000000000033-0000000000000000034
│ ├── edits_0000000000000000035-0000000000000000036
│ ├── edits_0000000000000000037-0000000000000000038
│ ├── edits_0000000000000000039-0000000000000000040
│ ├── edits_0000000000000000041-0000000000000000042
│ ├── edits_0000000000000000043-0000000000000000044
│ ├── edits_0000000000000000045-0000000000000000046
│ ├── edits_0000000000000000047-0000000000000000048
│ ├── edits_0000000000000000049-0000000000000000050
│ ├── edits_0000000000000000051-0000000000000000052
│ ├── edits_0000000000000000053-0000000000000000054
│ ├── edits_0000000000000000055-0000000000000000056
│ ├── edits_0000000000000000057-0000000000000000058
│ ├── edits_0000000000000000059-0000000000000000060
│ ├── edits_0000000000000000061-0000000000000000062
│ ├── edits_0000000000000000063-0000000000000000064
│ ├── edits_0000000000000000065-0000000000000000066
│ ├── edits_0000000000000000067-0000000000000000068
│ ├── edits_0000000000000000069-0000000000000000070
│ ├── edits_0000000000000000071-0000000000000000072
│ ├── edits_0000000000000000073-0000000000000000074
│ ├── edits_0000000000000000075-0000000000000000076
│ ├── edits_0000000000000000077-0000000000000000078
│ ├── edits_0000000000000000079-0000000000000000080
│ ├── edits_0000000000000000081-0000000000000000082
│ ├── edits_0000000000000000083-0000000000000000084
│ ├── edits_0000000000000000085-0000000000000000086
│ ├── edits_0000000000000000087-0000000000000000088
│ ├── edits_0000000000000000089-0000000000000000090
│ ├── edits_0000000000000000091-0000000000000000092
│ ├── edits_0000000000000000093-0000000000000000094
│ ├── edits_0000000000000000095-0000000000000000096
│ ├── edits_0000000000000000097-0000000000000000098
│ ├── edits_0000000000000000099-0000000000000000100
│ ├── edits_0000000000000000101-0000000000000000102
│ ├── edits_0000000000000000103-0000000000000000104
│ ├── edits_0000000000000000105-0000000000000000106
│ ├── edits_0000000000000000107-0000000000000000108
│ ├── edits_0000000000000000109-0000000000000000110
│ ├── edits_0000000000000000111-0000000000000000112
│ ├── edits_0000000000000000113-0000000000000000114
│ ├── edits_0000000000000000115-0000000000000000116
│ ├── edits_0000000000000000117-0000000000000000118
│ ├── edits_0000000000000000119-0000000000000000120
│ ├── edits_0000000000000000121-0000000000000000122
│ ├── edits_0000000000000000123-0000000000000000124
│ ├── edits_0000000000000000125-0000000000000000126
│ ├── edits_0000000000000000127-0000000000000000128
│ ├── edits_0000000000000000129-0000000000000000130
│ ├── edits_0000000000000000131-0000000000000000132
│ ├── edits_0000000000000000133-0000000000000000134
│ ├── edits_0000000000000000135-0000000000000000136
│ ├── edits_0000000000000000137-0000000000000000138
│ ├── edits_0000000000000000139-0000000000000000140
│ ├── edits_0000000000000000141-0000000000000000142
│ ├── edits_0000000000000000143-0000000000000000144
│ ├── edits_0000000000000000145-0000000000000000146
│ ├── edits_0000000000000000147-0000000000000000148
│ ├── edits_0000000000000000149-0000000000000000150
│ ├── edits_0000000000000000151-0000000000000000152
│ ├── fsimage_0000000000000000150
│ ├── fsimage_0000000000000000150.md5
│ ├── fsimage_0000000000000000152
│ ├── fsimage_0000000000000000152.md5
│ └── VERSION
└── in_use.lock
1 directory, 72 files
在hdfs-site.xml加入secondary node configuration
ubuntu@nn:~$ sudo cat /opt/hadoop-2.8.2/etc/hadoop/hdfs-site.xml
:::
<property>
<name>dfs.namenode.checkpoint.dir</name>
<value>file:/home/ubuntu/sn</value>
</property>
</configuration>
Check Secondary NameNode 更新的狀態 default 60 mins refresh
ubuntu@nn:~$ tail -n 30 /tmp/hadoop-ubuntu-secondarynamenode-nn.log
:::
2017-12-11 02:35:03,310 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNam eNode: Checkpoint done. New Image Size: 618
:::
2017-12-11 02:35:03,310 WARN org.apache.hadoop.hdfs.server.namenode.SecondaryNam eNode: Checkpoint done. New Image Size: 618
:::
是的,可更改設定為10 mins refresh一次 Edit hdfs-stie.xml P6-31
設定Rack Awareness, 將IP address轉為Rack位置 P6-34
HDFS系統空間不足時,可加DataNode P6-36
DataNode運算主機的管理策略 hdfs.allow 白名單 P6-47
HDFS Balance P6-50
NameNode損毀救援 P6-57
HDFS 分散式檔案系統權限設定 P6-65
沒有留言:
張貼留言