hadoop2安装手顺

[复制链接]
wangzuopu 发表于 2019-1-30 23:06:18 | 显示全部楼层 |阅读模式



一、环境准备,参看hadoop安装手顺(一到四章节 jdk ssh)
服务器规划如下:
master1: 192.168.10.201 主机名:master1        (active namenode,RM)
master1-ha: 192.168.10.202 主机名:master1ha (standby namenode,jn)
master2: 192.168.10.203 主机名:master2 (active namenode,jn)
master2-ha: 192.168.10.204 主机名:master2ha (standby namenode,jn)
slave1: 192.168.10.205 主机名:slave1 (datanode,nodemanager)
slave2: 192.168.10.206 主机名:slave2 (datanode,nodemanager)
slave3: 192.168.10.207 主机名:slave3 (datanode,nodemanager)

master:active namenode,RM
masterha:standby namenode,jn
slave1:datanode,nodemanager,jn,active namenode
slave2:datanode,nodemanager,jn,standby namenode
slave3:datanode,nodemanager

二、找三台机器安装zookeeper,
本例中三台服务器为masterha1、masterha2和master2
三、安装hadoop2
vi /etc/profile
export HADOOP_HOME=/home/hadoop/hadoop
export PATH=$PATHHADOOP_HOME/sbin


1.解压缩hadoop-2.2.0.tar.gz 并改名为hadoop2 添加环境变量HADOOP_HOME、PATH(注意除了bin目录外还有sbin目录)
2.cd ~/hadoop
创建以下目录 权限设置为755(mkdir -m 755 xxx)
mkdir -m 755 namedir
mkdir -m 755 datadir
mkdir -m 755 tmp
mkdir -m 755 jndir
mkdir -m 755 hadoopmrsys
mkdir -m 755 hadoopmrlocal
mkdir -m 755 nodemanagerlocal
mkdir -m 755 nodemanagerlogs

cd /home/hadoop/hadoop2/etc/hadoop

修改core-site.xml

<configuration>
<property>
<name>fs.defaultFS</name>
<value>viewfs:///</value>
</property>
<property>
<name>fs.viewfs.mounttable.default.link./tmp</name>
<value>hdfs://hadoop-cluster1/tmp</value>
</property>
<property>
<name>fs.viewfs.mounttable.default.link./tmp2</name>
<value>hdfs://hadoop-cluster2/tmp2</value>
</property>
</configuration>

修改hdfs-site.xml
<configuration>
<!-- 使用federation时,使用了2个HDFS集群。这里抽象出两个NameService实际上就是给这2个HDFS集群起了个别名。名字可以随便起,相互不重复即可 -->
<property>
<name>dfs.nameservices</name>
<value>hadoop-cluster1,hadoop-cluster2</value>
</property>

<!--cluster1 start-->
<!-- 指定NameService是cluster1时的namenode有哪些,这里的值也是逻辑名称,名字随便起,相互不重复即可 -->
<property>
<name>dfs.ha.namenodes.hadoop-cluster1</name>
<value>nn1,nn1ha</value>
</property>
<!-- 指定nn1的RPC地址 -->
<property>
<name>dfs.namenode.rpc-address.hadoop-cluster1.nn1</name>
<value>master1:9000</value>
</property>
<!-- 指定nn1ha的RPC地址 -->
<property>
<name>dfs.namenode.rpc-address.hadoop-cluster1.nn1ha</name>
<value>master1ha:9000</value>
</property>
<!-- 指定nn1的http地址 -->
<property>
<name>dfs.namenode.http-address.hadoop-cluster1.nn1</name>
<value>master1:50070</value>
</property>
<!-- 指定nn1ha的http地址 -->
<property>
<name>dfs.namenode.http-address.hadoop-cluster1.nn1ha</name>
<value>master1ha:50070</value>
</property>
<!-- 指定nn1secondary的http地址 -->
<property>
<name>dfs.namenode.secondary.http-address.hadoop-cluster1.nn1</name>
<value>master1:9001</value>
</property>
<!-- 指定nn1hasecondary的http地址 -->
<property>
<name>dfs.namenode.secondary.http-address.hadoop-cluster1.nn1ha</name>
<value>master1ha:9001</value>
</property>
<!-- 指定cluster1出故障时,哪个实现类负责执行故障切换-->
<property>
<name>dfs.client.failover.proxy.provider.hadoop-cluster1</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!--cluster1 end-->

<!--cluster2 start-->
<property>
<name>dfs.ha.namenodes.hadoop-cluster2</name>
<value>nn2,nn2ha</value>
</property>
<property>
<name>dfs.namenode.rpc-address.hadoop-cluster2.nn2</name>
<value>master2:9000</value>
</property>
<property>
<name>dfs.namenode.rpc-address.hadoop-cluster2.nn2ha</name>
<value>master2ha:9000</value>
</property>
<property>
<name>dfs.namenode.http-address.hadoop-cluster2.nn2</name>
<value>master2:50070</value>
</property>
<property>
<name>dfs.namenode.http-address.hadoop-cluster2.nn2ha</name>
<value>master2ha:50070</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address.hadoop-cluster2.nn2</name>
<value>master2:9001</value>
</property>
<property>
<name>dfs.namenode.secondary.http-address.hadoop-cluster2.nn2ha</name>
<value>master2ha:9001</value>
</property>
<property>
<name>dfs.client.failover.proxy.provider.hadoop-cluster2</name>
<value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<!--cluster2 end-->
<property>
<name>dfs.namenode.name.dir</name>
<value>/home/hadoop/hadoop/namedir</value>
</property>

<!-- nn1共享文件夹地址-->
<property>
<name>dfs.namenode.shared.edits.dir.hadoop-cluster1.nn1</name>
<value>qjournal://master1ha:8485;master2:8485;master2ha:8485/cluster1</value>
</property>
<!-- nn1ha共享文件夹地址-->
<property>
<name>dfs.namenode.shared.edits.dir.hadoop-cluster1.nn1ha</name>
<value>qjournal://master1ha:8485;master2:8485;master2ha:8485/cluster1</value>
</property>
<!-- nn2共享文件夹地址-->
<property>
<name>dfs.namenode.shared.edits.dir.hadoop-cluster2.nn2</name>
<value>qjournal://master1ha:8485;master2:8485;master2ha:8485/cluster2</value>
</property>
<!-- nn2ha共享文件夹地址-->
<property>
<name>dfs.namenode.shared.edits.dir.hadoop-cluster2.nn2ha</name>
<value>qjournal://master1ha:8485;master2:8485;master2ha:8485/cluster2</value>
</property>
<!-- datanode文件存放地址-->
<property>
<name>dfs.datanode.data.dir</name>
<value>/home/hadoop/hadoop/datadir</value>
</property>
<property>
<name>ha.zookeeper.quorum</name>
<value>master1:2181,master1ha:2181,master2:2181</value>
</property>
<!-- 一旦需要NameNode切换,使用ssh方式进行操作-->
<property>
<name>dfs.ha.fencing.methods</name>
<value>sshfence</value>
</property>
<!-- 超时-->
<property>
<name>ha.zookeeper.session-timeout.ms</name>
<value>5000</value>
</property>
<!--指定集群是否启动自动故障恢复,即当NameNode出故障时,是否自动切换到另一台NameNode-->
<property>
<name>dfs.ha.automatic-failover.enabled</name>
<value>true</value>
</property>
<!-- 指定JournalNode集群在对NameNode的目录进行共享时,自己存储数据的磁盘路径-->
<property>
<name>dfs.journalnode.edits.dir</name>
<value>/home/hadoop/hadoop/jndir</value>
</property>
<!--指定DataNode存储block的副本数量。默认值是3个 -->
<property>
<name>dfs.replication</name>
<value>2</value>
</property>


<!--不做权限 -->
<property>
<name>dfs.permission</name>
<value>false</value>
</property>
<!--是否允许web访问 -->
<property>
<name>dfs.webhdfs.enabled</name>
<value>true</value>
</property>
<!--是否支持append -->
<property>
<name>dfs.support.append</name>
<value>true</value>
</property>
<property>
<name>hadoop.proxyuser.hduser.hosts</name>
<value>*</value>
</property>
<property>
<name>hadoop.proxyuser.hduser.groups</name>
<value>*</value>
</property>
<!--如果使用ssh进行故障切换,使用ssh通信时用的密钥存储的位置 -->
<property>
<name>dfs.ha.fencing.ssh.private-key-files</name>
<value>/home/hadoop/.ssh/id_rsa</value>
</property>
</configuration>

cp mapred-site.xml.template mapred-site.xml
修改mapred-site.xml
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapreduce.job.tracker</name>
<value>master1:54311</value>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master1:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master1:19888</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/hadoop/hadoop/hadoopmrsys</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/hadoop/hadoop/hadoopmrlocal</value>
<final>true</final>
</property>
</configuration>

修改yarn-site.xml
<configuration>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>/home/hadoop/hadoop/nodemanagerlocal</value>
</property>
<property>
<name>yarn.nodemanager.log-dirs</name>
<value>/home/hadoop/hadoop/nodemanagerlogs</value>
</property>
<property>
<name>yarn.nodemanager.remote-app-log-dir</name>
<value>/home/hadoop/hadoop/nodemanagerremote</value>
</property>
<!--自定ResourceManager的地址,还是单点,这是隐患 -->
<property>
<name>yarn.resourcemanager.address</name>
<value>master1:18032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master1:18030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master1:18031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master1:18033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master1:18088</value>
</property>
</configuration>

修改slaves
slave1
slave2
slave3

修改hadoop-env.sh
就是修改这一行内容,修改后的结果如下
export JAVA_HOME=/usr/jdk
【这里的JAVA_HOME的值是jdk的安装路径。如果你那里不一样,请修改为自己的地址】


修改yarn-env.sh
export JAVA_HOME=/usr/jdk


四、将配置好的hadoop分发到其余服务器上。
scp -r /home/hadoop/hadoop hadoop@192.168.10.202:/home/hadoop/hadoop
scp -r /home/hadoop/hadoop hadoop@192.168.10.203:/home/hadoop/hadoop
scp -r /home/hadoop/hadoop hadoop@192.168.10.204:/home/hadoop/hadoop
scp -r /home/hadoop/hadoop hadoop@192.168.10.205:/home/hadoop/hadoop
scp -r /home/hadoop/hadoop hadoop@192.168.10.206:/home/hadoop/hadoop
scp -r /home/hadoop/hadoop hadoop@192.168.10.207:/home/hadoop/hadoop
scp -r /home/hadoop/hadoop hadoop@192.168.10.208:/home/hadoop/hadoop
五、启动hdfs
5.1启动ZooKeeper集群
启动master1、master1hamaster2节点上的zookeeper
zkServer.sh start
验证:
[hadoop@master1 sbin]$ jps7600 QuorumPeerMain
[hadoop@master1 hadoop]# zkServer.sh statusJMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower



5.2格式化ZooKeeper集群,目的是在ZooKeeper集群上建立HA的相应节点。
在master1和master2节点上执行:
/home/hadoop/hadoop/bin/hdfs zkfc -formatZK
验证:
[root@master1 hadoop]# zkCli.sh

[zk: localhost:2181(CONNECTED) 0] ls /
[hadoop-ha, zookeeper]
[zk: localhost:2181(CONNECTED) 1] ls /hadoop-ha
[cluster2, cluster1]

[zk: localhost:2181(CONNECTED) 2]
【集群c2也格式化,产生一个新的ZK节点cluster2】

5.3启动JournalNode集群

在master1ha、master2、master2ha节点上执行:
/home/hadoop/hadoop/sbin/hadoop-daemon.sh start journalnode
命令输出(以master1为例):
[root@master1 hadoop]# /usr/local/hadoop/sbin/hadoop-daemon.sh start journalnode
starting journalnode, logging to /usr/local/hadoop/logs/hadoop-root-journalnode-hadoop101.out
[root@master1 hadoop]#
在每个节点执行完启动命令后,每个节点都执行以下验证。
[root@master1 hadoop]# jps
23396 JournalNode
23598 Jps
22491 QuorumPeerMain
[root@master1 hadoop]#
【产生一个java进程JournalNode】

5.4格式化master1的NameNode
在master1节点上执行:
/home/hadoop/hadoop/bin/hdfs namenode -format -clusterId hellokitty

验证:
[root@master1 hadoop]# ls temp/
dfs
[root@master1 hadoop]# ls temp/dfs/
name


5.5启动master1中刚才格式化的NameNode
/home/hadoop/hadoop/sbin/hadoop-daemon.sh start namenode
验证:
[root@master1 hadoop]# jps
23396 JournalNode
23598 Jps
23558 NameNode
22491 QuorumPeerMain

[root@master1 hadoop]#
【启动后,产生一个新的java进程NameNode】
通过浏览器访问,也可以看到下图所示
image



5.6把NameNode的数据从master1同步到master1ha中
在master1ha节点上执行:
/home/hadoop/hadoop/bin/hdfs namenode -bootstrapStandby
验证:
[root@master1ha hadoop]# jps
5.7启动master1ha 的Namenode

/home/hadoop/hadoop/sbin/hadoop-daemon.sh start namenode
验证:
[root@master1ha hadoop]# jps
12355 JournalNode
12611 Jps
12573 NameNode
12081 QuorumPeerMain

[root@master1ha hadoop]#
【产生java进程NameNode】
通过浏览器访问,也可以看到下图所示
image





5.8将master1置成active状态
在master1,master1ha节点上执行:(将master1置成active状态)
/home/hadoop/hadoop/sbin/hadoop-daemon.sh start zkfc
或者:$bin/hdfs haadmin -ns hadoop-cluster1 -transitionToActive nn1 这个是手动的方法(在master1上操作),
如果前面的配置不是自动切换可以用这个
验证:
刷新浏览器

5.9格式化集群master2的一个NameNode
在master2节点上执行:
/home/hadoop/hadoop/bin/hdfs namenode -format -clusterId hellokitty

验证:
[root@master2 hadoop]#jps
5.10启动master2刚才格式化的NameNode

/home/hadoop/hadoop/sbin/hadoop-daemon.sh start namenode
验证:
[root@hadoop103 hadoop]# jps
11290 JournalNode
11560 NameNode
10972 QuorumPeerMain
11600 Jps
[root@hadoop103 hadoop]#
也可以通过浏览器访问http://hadoop103:50070,可以看到如上图页面,此处省略截图。


5.11把NameNode的数据从master2同步到master2ha中
在master2ha节点上执行:
/home/hadoop/hadoop/bin/hdfs namenode -bootstrapStandby
验证:
[root@hadoop104 hadoop]# jps
5.12.启动master2ha的Namenode

/home/hadoop/hadoop/sbin/hadoop-daemon.sh start namenode
验证:
[root@hadoop104 hadoop]# jps
8822 NameNode
8975 Jps

[root@hadoop104 hadoop]#
也可以通过浏览器访问http://hadoop104:50070,可以看到如上图页面,此处省略截图。


5.13将master2置成active状态
在master2,master2ha节点上执行:(将master2置成active状态)
/home/hadoop/hadoop/sbin/hadoop-daemon.sh start zkfc
或者:$bin/hdfs haadmin -ns hadoop-cluster2 -transitionToActive nn3

5.14启动所有的DataNode
在slave1、slave2、slave3节点上执行:(启动datanode)
/home/hadoop/hadoop/sbin/hadoop-daemon.sh start datanode
验证hdfs:
http://master1:50070
hadoop fs -mkdir hdfs://hadoop-cluster1/tmp3
hadoop fs -mkdir hdfs://hadoop-cluster2/tmp4
hadoop fs -ls /
//hadoop shell命令验证
//注意操作的时候需要这样写了:hadoop fs -mkdir hdfs://hadoop-cluster1/aaa
//hadoop fs -ls hdfs://hadoop-cluster1


六、启动mapreduce和yarn
在master1节点上执行:
/home/hadoop/hadoop/sbin/start-yarn.sh
命令输出:
[root@master1 hadoop]# /usr/local/hadoop/sbin/start-yarn.sh
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-root-resourcemanager-master1.out
slave3: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-slave3.out
slave2: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-slave2.out
slave1: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-root-nodemanager-slave1.out
[root@master1 hadoop]#
验证:
[root@master1 hadoop]# jps
23396 JournalNode
25154 ResourceManager
25247 NodeManager
24232 DataNode
23558 NameNode
22491 QuorumPeerMain
25281 Jps

[root@master1 hadoop]#
【产生java进程ResourceManager和NodeManager】
也可以通过浏览器访问,如下图
image





验证HDFS是否好用
在任意一个节点上执行以下命令(这里以hadoop101为例),把数据上传到HDFS集群中
[root@hadoop101 hadoop]# pwd
/usr/local/hadoop/etc/hadoop
[root@hadoop101 hadoop]# ls
capacity-scheduler.xml hadoop-metrics.properties httpfs-site.xml ssl-server.xml.example
configuration.xsl hadoop-policy.xml log4j.properties startall.sh
container-executor.cfg hdfs2-site.xml mapred-env.sh yarn-env.sh
core-site.xml hdfs-site.xml mapred-queues.xml.template yarn-site.xml
fairscheduler.xml httpfs-env.sh mapred-site.xml zookeeper.out
hadoop-env.sh httpfs-log4j.properties slaves
hadoop-metrics2.properties httpfs-signature.secret ssl-client.xml.example
[root@hadoop101 hadoop]# hadoop fs -put core-site.xml /
【上传到集群中,默认是上传到HDFS联盟的c1集群中】
验证:
[root@hadoop101 hadoop]# hadoop fs -ls /
Found 1 items
-rw-r--r-- 2 root supergroup 446 2014-02-12 09:00 /core-site.xml
[root@hadoop101 hadoop]#
也可以通过浏览器查看,数据默认是放在第一个集群中的
image

验证Yarn是否好用
在hadoop101上执行以下命令 hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /core-site.xml /out
命令输出:
[root@hadoop101 hadoop]# hadoop jar /usr/local/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /core-site.xml /out
14/02/12 11:43:55 INFO client.RMProxy: Connecting to ResourceManager at hadoop101/192.168.80.101:8032
14/02/12 11:43:59 INFO input.FileInputFormat: Total input paths to process : 1
14/02/12 11:43:59 INFO mapreduce.JobSubmitter: number of splits:1
14/02/12 11:43:59 INFO Configuration.deprecation: user.name is deprecated. Instead, use mapreduce.job.user.name
14/02/12 11:43:59 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
14/02/12 11:43:59 INFO Configuration.deprecation: mapred.output.value.class is deprecated. Instead, use mapreduce.job.output.value.class
14/02/12 11:43:59 INFO Configuration.deprecation: mapreduce.combine.class is deprecated. Instead, use mapreduce.job.combine.class
14/02/12 11:43:59 INFO Configuration.deprecation: mapreduce.map.class is deprecated. Instead, use mapreduce.job.map.class
14/02/12 11:43:59 INFO Configuration.deprecation: mapred.job.name is deprecated. Instead, use mapreduce.job.name
14/02/12 11:43:59 INFO Configuration.deprecation: mapreduce.reduce.class is deprecated. Instead, use mapreduce.job.reduce.class
14/02/12 11:43:59 INFO Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir
14/02/12 11:43:59 INFO Configuration.deprecation: mapred.output.dir is deprecated. Instead, use mapreduce.output.fileoutputformat.outputdir
14/02/12 11:43:59 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
14/02/12 11:43:59 INFO Configuration.deprecation: mapred.output.key.class is deprecated. Instead, use mapreduce.job.output.key.class
14/02/12 11:43:59 INFO Configuration.deprecation: mapred.working.dir is deprecated. Instead, use mapreduce.job.working.dir
14/02/12 11:44:01 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1392169506119_0002
14/02/12 11:44:04 INFO impl.YarnClientImpl: Submitted application application_1392169506119_0002 to ResourceManager at hadoop101/192.168.80.101:8032
14/02/12 11:44:05 INFO mapreduce.Job: The url to track the job: http://hadoop101:8088/proxy/application_1392169506119_0002/
14/02/12 11:44:05 INFO mapreduce.Job: Running job: job_1392169506119_0002
14/02/12 11:44:41 INFO mapreduce.Job: Job job_1392169506119_0002 running in uber mode : false
14/02/12 11:44:41 INFO mapreduce.Job: map 0% reduce 0%
14/02/12 11:45:37 INFO mapreduce.Job: map 100% reduce 0%
14/02/12 11:46:54 INFO mapreduce.Job: map 100% reduce 100%
14/02/12 11:47:01 INFO mapreduce.Job: Job job_1392169506119_0002 completed successfully
14/02/12 11:47:02 INFO mapreduce.Job: Counters: 43
File System Counters
FILE: Number of bytes read=472
FILE: Number of bytes written=164983
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=540
HDFS: Number of bytes written=402
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=63094
Total time spent by all reduces in occupied slots (ms)=57228
Map-Reduce Framework
Map input records=17
Map output records=20
Map output bytes=496
Map output materialized bytes=472
Input split bytes=94
Combine input records=20
Combine output records=16
Reduce input groups=16
Reduce shuffle bytes=472
Reduce input records=16
Reduce output records=16
Spilled Records=32
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=632
CPU time spent (ms)=3010
Physical memory (bytes) snapshot=255528960
Virtual memory (bytes) snapshot=1678471168
Total committed heap usage (bytes)=126660608
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=446
File Output Format Counters
Bytes Written=402
[root@hadoop101 hadoop]#
验证:
[root@hadoop101 hadoop]# hadoop fs -ls /out
Found 2 items
-rw-r--r-- 2 root supergroup 0 2014-02-12 11:46 /out/_SUCCESS
-rw-r--r-- 2 root supergroup 402 2014-02-12 11:46 /out/part-r-00000
[root@hadoop101 hadoop]# hadoop fs -text /out/part-r-00000
</configuration> 1
</property> 3
<?xml 1
<?xml-stylesheet 1
<configuration> 1
<name>fs.defaultFS</name> 1
<name>ha.zookeeper.quorum</name> 1
<name>hadoop.tmp.dir</name> 1
<property> 3
<value>/usr/local/hadoop/tmp</value> 1
<value>hadoop101:2181,hadoop102:2181,hadoop103:2181</value> 1
<value>hdfs://cluster1</value> 1
encoding="UTF-8"?> 1
href="configuration.xsl"?> 1
type="text/xsl" 1
version="1.0" 1
[root@hadoop101 hadoop]#
验证HA的故障自动转移是否好用
观察cluster1的两个NameNode的状态,hadoop101的状态是standby,hadoop102的状态是active,如下图。
image


image

下面我们杀死hadoop102的NameNode进程,观察hadoop101的状态是否会自动切换成active。
在hadoop102执行以下命令
[root@hadoop102 hadoop]# jps
13389 DFSZKFailoverController
12355 JournalNode
13056 DataNode
15660 Jps
14496 NodeManager
12573 NameNode
12081 QuorumPeerMain
[root@hadoop102 hadoop]# kill -9 12573
[root@hadoop102 hadoop]# jps
13389 DFSZKFailoverController
12355 JournalNode
13056 DataNode
14496 NodeManager
15671 Jps
12081 QuorumPeerMain
[root@hadoop102 hadoop]#
再观察页面,发现如下图所示
image


image

证明HDFS的高可靠是可用的。









七、关闭服务
在master1节点上执行:
/home/hadoop/hadoop/sbin/stop-yarn.sh


在slave1、slave2、slave3节点上执行:
/home/hadoop/hadoop/sbin/hadoop-daemons.sh stop datanode


在master1,masterha1,master2,masterha2上执行:
/home/hadoop/hadoop/sbin/hadoop-daemon.sh stop namenode



在masterha1,master2,masterha2上执行:
/home/hadoop/hadoop/sbin/hadoop-daemon.sh stop journalnode


在master1,masterha1,master2,masterha2上执行:
/home/hadoop/hadoop/sbin/hadoop-daemon.sh stop zkfc



在masterha1,master2,masterha2上执行:
zkServer.sh stop



八、QA

为什么不启动SecondaryNameNode??

启动log:/home/hadoop/hadoop/sbin/hadoop-daemon.sh start secondarynamenode
报错:2014-08-28 11:35:11,361 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2014-08-28 11:35:11,807 FATAL org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: Failed to start secondary namenode
java.io.IOException: Cannot use SecondaryNameNode in an HA cluster. The Standby Namenode will perform checkpointing.
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:194)
at org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:652)
2014-08-28 11:35:11,817 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2014-08-28 11:35:11,819 INFO org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down SecondaryNameNode at master1/192.168.56.151

错误提示说:集群中的namenode standby从事了secondarynamenode的工作,不需要启动,但是如果你想单独配置呢?

hadoop2.X如何将namenode与SecondaryNameNode分开配置
1.如何将namenode与SecondaryNameNode分开?

2.SecondaryNameNode单独配置,需要修改那些配置文件?


3.masters文件的作用是什么?





我们这里假设你已经安装配置了hadoop2.2,至于如何配置可以参考,hadoop2.2完全分布式最新高可靠安装文档
在这个基础上,我们对配置文件做一些修改:
1.增加masters文件
[color=rgb(51, 102, 153) !important]复制代码





这里面放什么内容还是比较关键的,这里我们指定slave1节点上运行SecondaryNameNode。

注意:如果你想单独配置一台机器,那么在这个文件里面,填写这个节点的ip地址或则是hostname,如果是多台,则在masters里面写上多个,一行一个,我们这里指定一个

  • slave1

[color=rgb(51, 102, 153) !important]复制代码



2.修改hdfs-site.xml
在下面文件中增加如下内容:(记得下面亦可写成ip地址,这里为了理解方便,写的是hostname)
<property>
<name>dfs.http.address</name>
<value>master:50070</value>
<description>
The address and the base port where the dfs namenode web ui will listen on.
If the port is 0 then the server will start on a free port.
</description>
</property>
<property>
<name>dfs.namenode.secondary.http-address</name>
<value>slave1:50090</value>
</property>






3.修改core-site.xml文件

  • <property>
  • <name>fs.checkpoint.period</name>
  • <value>3600</value>
  • <description>The number of seconds between two periodic checkpoints.
  • </description>
  • </property>
  • <property>
  • <name>fs.checkpoint.size</name>
  • <value>67108864</value>
  • </property>

[color=rgb(51, 102, 153) !important]复制代码




上面修改完毕,相应的节点也做同样的修改





下面我们开始启动节点:
  • start-dfs.sh

[color=rgb(51, 102, 153) !important]复制代码

输出如下内容:
  • Starting namenodes on [master]
  • master: starting namenode, logging to /usr/hadoop/logs/hadoop-aboutyun-namenode-master.out
  • slave2: starting datanode, logging to /usr/hadoop/logs/hadoop-aboutyun-datanode-slave2.out
  • slave1: starting datanode, logging to /usr/hadoop/logs/hadoop-aboutyun-datanode-slave1.out
  • Starting secondary namenodes [slave1]
  • slave1: starting secondarynamenode, logging to /usr/hadoop/logs/hadoop-aboutyun-secondarynamenode-slave1.out

[color=rgb(51, 102, 153) !important]复制代码





然后查看节点:
(1)master节点:
  • aboutyun@master:/usr/hadoop/etc/hadoop$ jps
  • 5994 NameNode
  • 6201 Jps

[color=rgb(51, 102, 153) !important]复制代码






(2)slave1节点
  • aboutyun@slave1:/usr/hadoop/etc/hadoop$ jps
  • 5199 SecondaryNameNode
  • 5015 DataNode
  • 5291 Jps

[color=rgb(51, 102, 153) !important]复制代码






(3)slave2节点
  • aboutyun@slave2:/usr/hadoop/etc/hadoop$ jps
  • 3628 DataNode
  • 3696 Jps

[color=rgb(51, 102, 153) !important]复制代码







停止节点:
  • master: stopping namenode
  • slave1: stopping datanode
  • slave2: stopping datanode
  • Stopping secondary namenodes [slave1]
  • slave1: stopping secondarynamenode

[color=rgb(51, 102, 153) !important]复制代码












【智云杂货铺 bbs.0936sht.com】
回复 论坛版权

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则