1、 啟動(dòng)
hadoop的啟動(dòng)是通過其sbin目錄下的腳本來啟動(dòng)的。與啟動(dòng)相關(guān)的叫腳本有以下幾個(gè)：
start-all.sh、start-dfs.sh、start-yarn.sh、hadoop-daemon.sh、yarn-daemon.sh。
hadoop-daemon.sh是用來啟動(dòng)與hdfs相關(guān)的服務(wù)的
yarn-daemon.sh是用來啟動(dòng)和yarn相關(guān)的服務(wù)
start-dfs.sh是用來啟動(dòng)hdfs集群的
start-yarn.sh是用來啟動(dòng)yarn集群
start-all.sh是用來啟動(dòng)yarn和hdfs集群的
這幾個(gè)start開頭的腳本都是通過調(diào)用那兩個(gè)daemon腳本來啟動(dòng)的。
2、 腳本分析
這里先從start-all.sh開始分析，然后逐步分析其腳本的調(diào)用。
start-all.sh腳本內(nèi)容如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
|
#!/usr/bin/env bash# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.# Start all hadoop daemons. Run this on master node.echo "This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh"bin=`dirname "${BASH_SOURCE-$0}"`bin=`cd "$bin"; pwd`DEFAULT_LIBEXEC_DIR="$bin"/../libexecHADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}. $HADOOP_LIBEXEC_DIR/hadoop-config.sh# start hdfs daemons if hdfs is presentif [ -f "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh ]; then "${HADOOP_HDFS_HOME}"/sbin/start-dfs.sh --config $HADOOP_CONF_DIRfi# start yarn daemons if yarn is presentif [ -f "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh ]; then "${HADOOP_YARN_HOME}"/sbin/start-yarn.sh --config $HADOOP_CONF_DIRfi |
這個(gè)腳本的重點(diǎn)在第31行到末尾，這里是兩個(gè)if語句，第一個(gè)if語句里（第34行）調(diào)用的是start-dfs.sh腳本，第二個(gè)if語句里（第37行）調(diào)用的是start-yarn.sh。
然后我們以start-dfs.sh為例，繼續(xù)向下分析。
start-dfs.sh的內(nèi)容如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
|
#!/usr/bin/env bash# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.# Start hadoop dfs daemons.# Optinally upgrade or rollback dfs state.# Run this on master node.usage="Usage: start-dfs.sh [-upgrade|-rollback] [other options such as -clusterId]"bin=`dirname "${BASH_SOURCE-$0}"`bin=`cd "$bin"; pwd`DEFAULT_LIBEXEC_DIR="$bin"/../libexecHADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}. $HADOOP_LIBEXEC_DIR/hdfs-config.sh# get argumentsif [[ $# -ge 1 ]]; then startOpt="$1" shift case "$startOpt" in -upgrade) nameStartOpt="$startOpt" ;; -rollback) dataStartOpt="$startOpt" ;; *) echo $usage exit 1 ;; esacfi#Add other possible optionsnameStartOpt="$nameStartOpt $@"#---------------------------------------------------------# namenodesNAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -namenodes)echo "Starting namenodes on [$NAMENODES]""$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \ --config "$HADOOP_CONF_DIR" \ --hostnames "$NAMENODES" \ --script "$bin/hdfs" start namenode $nameStartOpt#---------------------------------------------------------# datanodes (using default slaves file)if [ -n "$HADOOP_SECURE_DN_USER" ]; then echo \ "Attempting to start secure cluster, skipping datanodes. " \ "Run start-secure-dns.sh as root to complete startup."else "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \ --config "$HADOOP_CONF_DIR" \ --script "$bin/hdfs" start datanode $dataStartOptfi#---------------------------------------------------------# secondary namenodes (if any)SECONDARY_NAMENODES=$($HADOOP_PREFIX/bin/hdfs getconf -secondarynamenodes 2>/dev/null)if [ -n "$SECONDARY_NAMENODES" ]; then echo "Starting secondary namenodes [$SECONDARY_NAMENODES]" "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \ --config "$HADOOP_CONF_DIR" \ --hostnames "$SECONDARY_NAMENODES" \ --script "$bin/hdfs" start secondarynamenodefi#---------------------------------------------------------# quorumjournal nodes (if any)SHARED_EDITS_DIR=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.namenode.shared.edits.dir 2>&-)case "$SHARED_EDITS_DIR" inqjournal://*) JOURNAL_NODES=$(echo "$SHARED_EDITS_DIR" | sed 's,qjournal://\([^/]*\)/.*,\1,g; s/;/ /g; s/:[0-9]*//g') echo "Starting journal nodes [$JOURNAL_NODES]" "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \ --config "$HADOOP_CONF_DIR" \ --hostnames "$JOURNAL_NODES" \ --script "$bin/hdfs" start journalnode ;;esac#---------------------------------------------------------# ZK Failover controllers, if auto-HA is enabledAUTOHA_ENABLED=$($HADOOP_PREFIX/bin/hdfs getconf -confKey dfs.ha.automatic-failover.enabled)if [ "$(echo "$AUTOHA_ENABLED" | tr A-Z a-z)" = "true" ]; then echo "Starting ZK Failover Controllers on NN hosts [$NAMENODES]" "$HADOOP_PREFIX/sbin/hadoop-daemons.sh" \ --config "$HADOOP_CONF_DIR" \ --hostnames "$NAMENODES" \ --script "$bin/hdfs" start zkfcfi# eof |
首先是第23行到30行，這里是在處理hadoop的路徑。
然后是第32行到51行是在處理腳本傳入的參數(shù)。最后就啟動(dòng)hdfs的各個(gè)角色了。
首先啟動(dòng)的是namenode，在第60行到63行，這段代碼是調(diào)用了hadoop-daemons.sh腳本來啟動(dòng)，這個(gè)腳本和之前提到的hadoop-daemon.sh腳本的區(qū)別在于這個(gè)腳本可以在集群的其他機(jī)器上啟動(dòng)該啟動(dòng)的角色，而hadoop-daemon.sh只能啟動(dòng)當(dāng)前機(jī)器上的角色。其實(shí)hadoop-daemons.sh也是通過調(diào)用hadoop-daemon.sh來啟動(dòng)的，這個(gè)稍后再分析。這個(gè)腳本還有幾個(gè)參數(shù)，其中最重要的是：start namenode。它表示啟動(dòng)namenode。
然后是第65行到第76行，這里是在啟動(dòng)datanode，啟動(dòng)的方式和namenode相同。啟動(dòng)DataNode的代碼在第73行。
然后是第78行到第90行，這里在啟動(dòng)secondarynamenode。如果是配置了namenode的高可用，secondarynamenode便不會(huì)啟動(dòng)。
然后是第92行到第105行，這里在啟動(dòng)journalnode。如果配置了namenode的高可用，journalnode才會(huì)啟動(dòng)。
最后是第107行到末尾，這里啟動(dòng)的是zkfc。同樣這個(gè)也是要配置高可用才會(huì)啟動(dòng)。
如果按照文檔（2）中配置的高可用來看，這里啟動(dòng)的角色應(yīng)該為：namenode、datanode、journalnode、zkfc。
啟動(dòng)上述角色調(diào)用的hadoop-daemons.sh腳本內(nèi)容如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
|
#!/usr/bin/env bash# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.# Run a Hadoop command on all slave hosts.usage="Usage: hadoop-daemons.sh [--config confdir] [--hosts hostlistfile] [start|stop] command args..."# if no args specified, show usageif [ $# -le 1 ]; then echo $usage exit 1fibin=`dirname "${BASH_SOURCE-$0}"`bin=`cd "$bin"; pwd`DEFAULT_LIBEXEC_DIR="$bin"/../libexecHADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}. $HADOOP_LIBEXEC_DIR/hadoop-config.shexec "$bin/slaves.sh" --config $HADOOP_CONF_DIR cd "$HADOOP_PREFIX" \; "$bin/hadoop-daemon.sh" --config $HADOOP_CONF_DIR "$@" |
這個(gè)腳本的重點(diǎn)在最后一行。這里有兩個(gè)腳本slaves.sh和hadoop-daemon.sh。slaves.sh腳本會(huì)使用ssh登錄到指定的服務(wù)器中然后執(zhí)行其中的hadoop-daemon.sh腳本。這里就不分析slaves.sh腳本了。
我們繼續(xù)看hadoop-daemon.sh腳本。
其內(nèi)容如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
|
#!/usr/bin/env bash# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.# Runs a Hadoop command as a daemon.## Environment Variables## HADOOP_CONF_DIR Alternate conf dir. Default is ${HADOOP_PREFIX}/conf.# HADOOP_LOG_DIR Where log files are stored. PWD by default.# HADOOP_MASTER host:path where hadoop code should be rsync'd from# HADOOP_PID_DIR The pid files are stored. /tmp by default.# HADOOP_IDENT_STRING A string representing this instance of hadoop. $USER by default# HADOOP_NICENESS The scheduling priority for daemons. Defaults to 0.##usage="Usage: hadoop-daemon.sh [--config <conf-dir>] [--hosts hostlistfile] [--script script] (start|stop) <hadoop-command> <args...>"# if no args specified, show usageif [ $# -le 1 ]; then echo $usage exit 1fibin=`dirname "${BASH_SOURCE-$0}"`bin=`cd "$bin"; pwd`DEFAULT_LIBEXEC_DIR="$bin"/../libexecHADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}. $HADOOP_LIBEXEC_DIR/hadoop-config.sh# get arguments#default valuehadoopScript="$HADOOP_PREFIX"/bin/hadoopif [ "--script" = "$1" ] then shift hadoopScript=$1 shiftfistartStop=$1shiftcommand=$1shifthadoop_rotate_log (){ log=$1; num=5; if [ -n "$2" ]; then num=$2 fi if [ -f "$log" ]; then # rotate logs while [ $num -gt 1 ]; do prev=`expr $num - 1` [ -f "$log.$prev" ] && mv "$log.$prev" "$log.$num" num=$prev done mv "$log" "$log.$num"; fi}if [ -f "${HADOOP_CONF_DIR}/hadoop-env.sh" ]; then . "${HADOOP_CONF_DIR}/hadoop-env.sh"fi# Determine if we're starting a secure datanode, and if so, redefine appropriate variablesif [ "$command" == "datanode" ] && [ "$EUID" -eq 0 ] && [ -n "$HADOOP_SECURE_DN_USER" ]; then export HADOOP_PID_DIR=$HADOOP_SECURE_DN_PID_DIR export HADOOP_LOG_DIR=$HADOOP_SECURE_DN_LOG_DIR export HADOOP_IDENT_STRING=$HADOOP_SECURE_DN_USER starting_secure_dn="true"fi#Determine if we're starting a privileged NFS, if so, redefine the appropriate variablesif [ "$command" == "nfs3" ] && [ "$EUID" -eq 0 ] && [ -n "$HADOOP_PRIVILEGED_NFS_USER" ]; then export HADOOP_PID_DIR=$HADOOP_PRIVILEGED_NFS_PID_DIR export HADOOP_LOG_DIR=$HADOOP_PRIVILEGED_NFS_LOG_DIR export HADOOP_IDENT_STRING=$HADOOP_PRIVILEGED_NFS_USER starting_privileged_nfs="true"fiif [ "$HADOOP_IDENT_STRING" = "" ]; then export HADOOP_IDENT_STRING="$USER"fi# get log directoryif [ "$HADOOP_LOG_DIR" = "" ]; then export HADOOP_LOG_DIR="$HADOOP_PREFIX/logs"fiif [ ! -w "$HADOOP_LOG_DIR" ] ; then mkdir -p "$HADOOP_LOG_DIR" chown $HADOOP_IDENT_STRING $HADOOP_LOG_DIRfiif [ "$HADOOP_PID_DIR" = "" ]; then HADOOP_PID_DIR=/tmpfi# some variablesexport HADOOP_LOGFILE=hadoop-$HADOOP_IDENT_STRING-$command-$HOSTNAME.logexport HADOOP_ROOT_LOGGER=${HADOOP_ROOT_LOGGER:-"INFO,RFA"}export HADOOP_SECURITY_LOGGER=${HADOOP_SECURITY_LOGGER:-"INFO,RFAS"}export HDFS_AUDIT_LOGGER=${HDFS_AUDIT_LOGGER:-"INFO,NullAppender"}log=$HADOOP_LOG_DIR/hadoop-$HADOOP_IDENT_STRING-$command-$HOSTNAME.outpid=$HADOOP_PID_DIR/hadoop-$HADOOP_IDENT_STRING-$command.pidHADOOP_STOP_TIMEOUT=${HADOOP_STOP_TIMEOUT:-5}# Set default scheduling priorityif [ "$HADOOP_NICENESS" = "" ]; then export HADOOP_NICENESS=0ficase $startStop in (start) [ -w "$HADOOP_PID_DIR" ] || mkdir -p "$HADOOP_PID_DIR" if [ -f $pid ]; then if kill -0 `cat $pid` > /dev/null 2>&1; then echo $command running as process `cat $pid`. Stop it first. exit 1 fi fi if [ "$HADOOP_MASTER" != "" ]; then echo rsync from $HADOOP_MASTER rsync -a -e ssh --delete --exclude=.svn --exclude='logs/*' --exclude='contrib/hod/logs/*' $HADOOP_MASTER/ "$HADOOP_PREFIX" fi hadoop_rotate_log $log echo starting $command, logging to $log cd "$HADOOP_PREFIX" case $command in namenode|secondarynamenode|datanode|journalnode|dfs|dfsadmin|fsck|balancer|zkfc) if [ -z "$HADOOP_HDFS_HOME" ]; then hdfsScript="$HADOOP_PREFIX"/bin/hdfs else hdfsScript="$HADOOP_HDFS_HOME"/bin/hdfs fi nohup nice -n $HADOOP_NICENESS $hdfsScript --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null & ;; (*) nohup nice -n $HADOOP_NICENESS $hadoopScript --config $HADOOP_CONF_DIR $command "$@" > "$log" 2>&1 < /dev/null & ;; esac echo $! > $pid sleep 1 head "$log" # capture the ulimit output if [ "true" = "$starting_secure_dn" ]; then echo "ulimit -a for secure datanode user $HADOOP_SECURE_DN_USER" >> $log # capture the ulimit info for the appropriate user su --shell=/bin/bash $HADOOP_SECURE_DN_USER -c 'ulimit -a' >> $log 2>&1 elif [ "true" = "$starting_privileged_nfs" ]; then echo "ulimit -a for privileged nfs user $HADOOP_PRIVILEGED_NFS_USER" >> $log su --shell=/bin/bash $HADOOP_PRIVILEGED_NFS_USER -c 'ulimit -a' >> $log 2>&1 else echo "ulimit -a for user $USER" >> $log ulimit -a >> $log 2>&1 fi sleep 3; if ! ps -p $! > /dev/null ; then exit 1 fi ;; (stop) if [ -f $pid ]; then TARGET_PID=`cat $pid` if kill -0 $TARGET_PID > /dev/null 2>&1; then echo stopping $command kill $TARGET_PID sleep $HADOOP_STOP_TIMEOUT if kill -0 $TARGET_PID > /dev/null 2>&1; then echo "$command did not stop gracefully after $HADOOP_STOP_TIMEOUT seconds: killing with kill -9" kill -9 $TARGET_PID fi else echo no $command to stop fi rm -f $pid else echo no $command to stop fi ;; (*) echo $usage exit 1 ;;esac |
這段代碼的重點(diǎn)在第131行到結(jié)束。這里是真正在啟動(dòng)服務(wù)的代碼，這個(gè)文件在調(diào)用的時(shí)候，會(huì)傳入兩個(gè)重要的參數(shù)start/stop xxx。用于啟動(dòng)或停止某些服務(wù)。以啟動(dòng)服務(wù)為例，其重點(diǎn)在第153行，這里會(huì)執(zhí)行一個(gè)hdfsScript腳本。這個(gè)參數(shù)的定義在第155行，
這里可以看見它實(shí)際是hadoop的bin目錄下的hdfs文件
文件的內(nèi)容如下：
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
|
#!/usr/bin/env bash# Licensed to the Apache Software Foundation (ASF) under one or more# contributor license agreements. See the NOTICE file distributed with# this work for additional information regarding copyright ownership.# The ASF licenses this file to You under the Apache License, Version 2.0# (the "License"); you may not use this file except in compliance with# the License. You may obtain a copy of the License at## http://www.apache.org/licenses/LICENSE-2.0## Unless required by applicable law or agreed to in writing, software# distributed under the License is distributed on an "AS IS" BASIS,# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.# See the License for the specific language governing permissions and# limitations under the License.# Environment Variables## JSVC_HOME home directory of jsvc binary. Required for starting secure# datanode.## JSVC_OUTFILE path to jsvc output file. Defaults to# $HADOOP_LOG_DIR/jsvc.out.## JSVC_ERRFILE path to jsvc error file. Defaults to $HADOOP_LOG_DIR/jsvc.err.bin=`which $0`bin=`dirname ${bin}`bin=`cd "$bin" > /dev/null; pwd`DEFAULT_LIBEXEC_DIR="$bin"/../libexecHADOOP_LIBEXEC_DIR=${HADOOP_LIBEXEC_DIR:-$DEFAULT_LIBEXEC_DIR}. $HADOOP_LIBEXEC_DIR/hdfs-config.shfunction print_usage(){ echo "Usage: hdfs [--config confdir] [--loglevel loglevel] COMMAND" echo " where COMMAND is one of:" echo " dfs run a filesystem command on the file systems supported in Hadoop." echo " classpath prints the classpath" echo " namenode -format format the DFS filesystem" echo " secondarynamenode run the DFS secondary namenode" echo " namenode run the DFS namenode" echo " journalnode run the DFS journalnode" echo " zkfc run the ZK Failover Controller daemon" echo " datanode run a DFS datanode" echo " dfsadmin run a DFS admin client" echo " haadmin run a DFS HA admin client" echo " fsck run a DFS filesystem checking utility" echo " balancer run a cluster balancing utility" echo " jmxget get JMX exported values from NameNode or DataNode." echo " mover run a utility to move block replicas across" echo " storage types" echo " oiv apply the offline fsimage viewer to an fsimage" echo " oiv_legacy apply the offline fsimage viewer to an legacy fsimage" echo " oev apply the offline edits viewer to an edits file" echo " fetchdt fetch a delegation token from the NameNode" echo " getconf get config values from configuration" echo " groups get the groups which users belong to" echo " snapshotDiff diff two snapshots of a directory or diff the" echo " current directory contents with a snapshot" echo " lsSnapshottableDir list all snapshottable dirs owned by the current user" echo " Use -help to see options" echo " portmap run a portmap service" echo " nfs3 run an NFS version 3 gateway" echo " cacheadmin configure the HDFS cache" echo " crypto configure HDFS encryption zones" echo " storagepolicies list/get/set block storage policies" echo " version print the version" echo "" echo "Most commands print help when invoked w/o parameters." # There are also debug commands, but they don't show up in this listing.}if [ $# = 0 ]; then print_usage exitfiCOMMAND=$1shiftcase $COMMAND in # usage flags --help|-help|-h) print_usage exit ;;esac# Determine if we're starting a secure datanode, and if so, redefine appropriate variablesif [ "$COMMAND" == "datanode" ] && [ "$EUID" -eq 0 ] && [ -n "$HADOOP_SECURE_DN_USER" ]; then if [ -n "$JSVC_HOME" ]; then if [ -n "$HADOOP_SECURE_DN_PID_DIR" ]; then HADOOP_PID_DIR=$HADOOP_SECURE_DN_PID_DIR fi if [ -n "$HADOOP_SECURE_DN_LOG_DIR" ]; then HADOOP_LOG_DIR=$HADOOP_SECURE_DN_LOG_DIR HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.dir=$HADOOP_LOG_DIR" fi HADOOP_IDENT_STRING=$HADOOP_SECURE_DN_USER HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.id.str=$HADOOP_IDENT_STRING" starting_secure_dn="true" else echo "It looks like you're trying to start a secure DN, but \$JSVC_HOME"\ "isn't set. Falling back to starting insecure DN." fifi# Determine if we're starting a privileged NFS daemon, and if so, redefine appropriate variablesif [ "$COMMAND" == "nfs3" ] && [ "$EUID" -eq 0 ] && [ -n "$HADOOP_PRIVILEGED_NFS_USER" ]; then if [ -n "$JSVC_HOME" ]; then if [ -n "$HADOOP_PRIVILEGED_NFS_PID_DIR" ]; then HADOOP_PID_DIR=$HADOOP_PRIVILEGED_NFS_PID_DIR fi if [ -n "$HADOOP_PRIVILEGED_NFS_LOG_DIR" ]; then HADOOP_LOG_DIR=$HADOOP_PRIVILEGED_NFS_LOG_DIR HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.log.dir=$HADOOP_LOG_DIR" fi HADOOP_IDENT_STRING=$HADOOP_PRIVILEGED_NFS_USER HADOOP_OPTS="$HADOOP_OPTS -Dhadoop.id.str=$HADOOP_IDENT_STRING" starting_privileged_nfs="true" else echo "It looks like you're trying to start a privileged NFS server, but"\ "\$JSVC_HOME isn't set. Falling back to starting unprivileged NFS server." fifiif [ "$COMMAND" = "namenode" ] ; then CLASS='org.apache.hadoop.hdfs.server.namenode.NameNode' HADOOP_OPTS="$HADOOP_OPTS $HADOOP_NAMENODE_OPTS"elif [ "$COMMAND" = "zkfc" ] ; then CLASS='org.apache.hadoop.hdfs.tools.DFSZKFailoverController' HADOOP_OPTS="$HADOOP_OPTS $HADOOP_ZKFC_OPTS"elif [ "$COMMAND" = "secondarynamenode" ] ; then CLASS='org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode' HADOOP_OPTS="$HADOOP_OPTS $HADOOP_SECONDARYNAMENODE_OPTS"elif [ "$COMMAND" = "datanode" ] ; then CLASS='org.apache.hadoop.hdfs.server.datanode.DataNode' if [ "$starting_secure_dn" = "true" ]; then HADOOP_OPTS="$HADOOP_OPTS -jvm server $HADOOP_DATANODE_OPTS" else HADOOP_OPTS="$HADOOP_OPTS -server $HADOOP_DATANODE_OPTS" fielif [ "$COMMAND" = "journalnode" ] ; then CLASS='org.apache.hadoop.hdfs.qjournal.server.JournalNode' HADOOP_OPTS="$HADOOP_OPTS $HADOOP_JOURNALNODE_OPTS"elif [ "$COMMAND" = "dfs" ] ; then CLASS=org.apache.hadoop.fs.FsShell HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"elif [ "$COMMAND" = "dfsadmin" ] ; then CLASS=org.apache.hadoop.hdfs.tools.DFSAdmin HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"elif [ "$COMMAND" = "haadmin" ] ; then CLASS=org.apache.hadoop.hdfs.tools.DFSHAAdmin CLASSPATH=${CLASSPATH}:${TOOL_PATH} HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"elif [ "$COMMAND" = "fsck" ] ; then CLASS=org.apache.hadoop.hdfs.tools.DFSck HADOOP_OPTS="$HADOOP_OPTS $HADOOP_CLIENT_OPTS"elif [ "$COMMAND" = "balancer" ] ; then CLASS=org.apache.hadoop.hdfs.server.balancer.Balancer HADOOP_OPTS="$HADOOP_OPTS $HADOOP_BALANCER_OPTS"elif [ "$COMMAND" = "mover" ] ; then CLASS=org.apache.hadoop.hdfs.server.mover.Mover HADOOP_OPTS="${HADOOP_OPTS} ${HADOOP_MOVER_OPTS}"elif [ "$COMMAND" = "storagepolicies" ] ; then CLASS=org.apache.hadoop.hdfs.tools.StoragePolicyAdminelif [ "$COMMAND" = "jmxget" ] ; then CLASS=org.apache.hadoop.hdfs.tools.JMXGetelif [ "$COMMAND" = "oiv" ] ; then CLASS=org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerPBelif [ "$COMMAND" = "oiv_legacy" ] ; then CLASS=org.apache.hadoop.hdfs.tools.offlineImageViewer.OfflineImageViewerelif [ "$COMMAND" = "oev" ] ; then CLASS=org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewerelif [ "$COMMAND" = "fetchdt" ] ; then CLASS=org.apache.hadoop.hdfs.tools.DelegationTokenFetcherelif [ "$COMMAND" = "getconf" ] ; then CLASS=org.apache.hadoop.hdfs.tools.GetConfelif [ "$COMMAND" = "groups" ] ; then CLASS=org.apache.hadoop.hdfs.tools.GetGroupselif [ "$COMMAND" = "snapshotDiff" ] ; then CLASS=org.apache.hadoop.hdfs.tools.snapshot.SnapshotDiffelif [ "$COMMAND" = "lsSnapshottableDir" ] ; then CLASS=org.apache.hadoop.hdfs.tools.snapshot.LsSnapshottableDirelif [ "$COMMAND" = "portmap" ] ; then CLASS=org.apache.hadoop.portmap.Portmap HADOOP_OPTS="$HADOOP_OPTS $HADOOP_PORTMAP_OPTS"elif [ "$COMMAND" = "nfs3" ] ; then CLASS=org.apache.hadoop.hdfs.nfs.nfs3.Nfs3 HADOOP_OPTS="$HADOOP_OPTS $HADOOP_NFS3_OPTS"elif [ "$COMMAND" = "cacheadmin" ] ; then CLASS=org.apache.hadoop.hdfs.tools.CacheAdminelif [ "$COMMAND" = "crypto" ] ; then CLASS=org.apache.hadoop.hdfs.tools.CryptoAdminelif [ "$COMMAND" = "version" ] ; then CLASS=org.apache.hadoop.util.VersionInfoelif [ "$COMMAND" = "debug" ]; then CLASS=org.apache.hadoop.hdfs.tools.DebugAdminelif [ "$COMMAND" = "classpath" ]; then if [ "$#" -gt 0 ]; then CLASS=org.apache.hadoop.util.Classpath else # No need to bother starting up a JVM for this simple case. if $cygwin; then CLASSPATH=$(cygpath -p -w "$CLASSPATH" 2>/dev/null) fi echo $CLASSPATH exit 0 fielse CLASS="$COMMAND"fi# cygwin path translationif $cygwin; then CLASSPATH=$(cygpath -p -w "$CLASSPATH" 2>/dev/null) HADOOP_LOG_DIR=$(cygpath -w "$HADOOP_LOG_DIR" 2>/dev/null) HADOOP_PREFIX=$(cygpath -w "$HADOOP_PREFIX" 2>/dev/null) HADOOP_CONF_DIR=$(cygpath -w "$HADOOP_CONF_DIR" 2>/dev/null) HADOOP_COMMON_HOME=$(cygpath -w "$HADOOP_COMMON_HOME" 2>/dev/null) HADOOP_HDFS_HOME=$(cygpath -w "$HADOOP_HDFS_HOME" 2>/dev/null) HADOOP_YARN_HOME=$(cygpath -w "$HADOOP_YARN_HOME" 2>/dev/null) HADOOP_MAPRED_HOME=$(cygpath -w "$HADOOP_MAPRED_HOME" 2>/dev/null)fiexport CLASSPATH=$CLASSPATHHADOOP_OPTS="$HADOOP_OPTS -Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,NullAppender}"# Check to see if we should start a secure datanodeif [ "$starting_secure_dn" = "true" ]; then if [ "$HADOOP_PID_DIR" = "" ]; then HADOOP_SECURE_DN_PID="/tmp/hadoop_secure_dn.pid" else HADOOP_SECURE_DN_PID="$HADOOP_PID_DIR/hadoop_secure_dn.pid" fi JSVC=$JSVC_HOME/jsvc if [ ! -f $JSVC ]; then echo "JSVC_HOME is not set correctly so jsvc cannot be found. jsvc is required to run secure datanodes. " echo "Please download and install jsvc from http://archive.apache.org/dist/commons/daemon/binaries/ "\ "and set JSVC_HOME to the directory containing the jsvc binary." exit fi if [[ ! $JSVC_OUTFILE ]]; then JSVC_OUTFILE="$HADOOP_LOG_DIR/jsvc.out" fi if [[ ! $JSVC_ERRFILE ]]; then JSVC_ERRFILE="$HADOOP_LOG_DIR/jsvc.err" fi exec "$JSVC" \ -Dproc_$COMMAND -outfile "$JSVC_OUTFILE" \ -errfile "$JSVC_ERRFILE" \ -pidfile "$HADOOP_SECURE_DN_PID" \ -nodetach \ -user "$HADOOP_SECURE_DN_USER" \ -cp "$CLASSPATH" \ $JAVA_HEAP_MAX $HADOOP_OPTS \ org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter "$@"elif [ "$starting_privileged_nfs" = "true" ] ; then if [ "$HADOOP_PID_DIR" = "" ]; then HADOOP_PRIVILEGED_NFS_PID="/tmp/hadoop_privileged_nfs3.pid" else HADOOP_PRIVILEGED_NFS_PID="$HADOOP_PID_DIR/hadoop_privileged_nfs3.pid" fi JSVC=$JSVC_HOME/jsvc if [ ! -f $JSVC ]; then echo "JSVC_HOME is not set correctly so jsvc cannot be found. jsvc is required to run privileged NFS gateways. " echo "Please download and install jsvc from http://archive.apache.org/dist/commons/daemon/binaries/ "\ "and set JSVC_HOME to the directory containing the jsvc binary." exit fi if [[ ! $JSVC_OUTFILE ]]; then JSVC_OUTFILE="$HADOOP_LOG_DIR/nfs3_jsvc.out" fi if [[ ! $JSVC_ERRFILE ]]; then JSVC_ERRFILE="$HADOOP_LOG_DIR/nfs3_jsvc.err" fi exec "$JSVC" \ -Dproc_$COMMAND -outfile "$JSVC_OUTFILE" \ -errfile "$JSVC_ERRFILE" \ -pidfile "$HADOOP_PRIVILEGED_NFS_PID" \ -nodetach \ -user "$HADOOP_PRIVILEGED_NFS_USER" \ -cp "$CLASSPATH" \ $JAVA_HEAP_MAX $HADOOP_OPTS \ org.apache.hadoop.hdfs.nfs.nfs3.PrivilegedNfsGatewayStarter "$@"else # run it exec "$JAVA" -Dproc_$COMMAND $JAVA_HEAP_MAX $HADOOP_OPTS $CLASS "$@"fi |
這段代碼的重點(diǎn)在第134行到219行和第304行。首先看第134行到219行，這段代碼雖然很長(zhǎng)但全是一堆if else語句。其邏輯也很簡(jiǎn)單，就是根據(jù)傳入的COMMAND的值來為CLASS和HADOOP_OPTS賦值。然后是第304行，執(zhí)行CLASS類。以第134行的namenode類為例，這里CLASS的值為org.apache.hadoop.hdfs.server.namenode.NameNode，這是一個(gè)java類隨后便會(huì)執(zhí)行這個(gè)類啟動(dòng)namenode。
如果想要debug hdfs的源碼，最好在這里設(shè)置遠(yuǎn)程調(diào)試。因?yàn)檫@里有單個(gè)的服務(wù)類與啟動(dòng)參數(shù)，可以準(zhǔn)確的定位需要的服務(wù)。
以上就是Hadoop源碼分析之啟動(dòng)及腳本剖析的詳細(xì)內(nèi)容，本系列下一篇文章傳送門Hadoop源碼分析四遠(yuǎn)程debug調(diào)試更多關(guān)于Hadoop源碼分析分析的資料請(qǐng)繼續(xù)關(guān)注服務(wù)器之家其它相關(guān)文章！
原文鏈接：https://blog.csdn.net/qq_39210987/article/details/113922124
