建站服務(wù)器
一、首先說(shuō)一下，hadoop有三種運(yùn)行模式。第一個(gè)是獨(dú)立(或本地)運(yùn)行模式：無(wú)需運(yùn)行任何守護(hù)進(jìn)程，所有程序都在一個(gè)同一個(gè)jvm上執(zhí)行。第二個(gè)是偽分布模式：hadoop守護(hù)進(jìn)程運(yùn)行在本地機(jī)器上，模擬一個(gè)小規(guī)模的集群。第三個(gè)是全分布模式，也就是通常所說(shuō)的完全分布式模式：hadoop守護(hù)進(jìn)程運(yùn)行在一個(gè)集群上。下面簡(jiǎn)單介紹下完全分布式的配置：首先準(zhǔn)備三臺(tái)機(jī)器，系統(tǒng)公共配置部分：系統(tǒng)信息：[root@hadoop1~]#cat/etc/redhat-releasecentosrelease6.7(final)[centos@hadoop1~]$uname-r2.6.32-573.el6.x86_64[root@hadoop1~]#uname-mx86_64公共操作用戶(hù)：centoscentos用戶(hù)在/etc/sudoers中的公共配置:[root@hadoop1~]#grepcentos/etc/sudoerscentosall=(all)nopasswd:all環(huán)境變量如下：[root@hadoop1~]#tail-16/etc/profile#javajava_home=/home/centos/download/jdkpath=$java_home/bin:$pathclasspath=.:$java_home/lib/dt.jar:$java_home/lib/tools.jarexportjava_homeexportclasspath#hadoophadoop_home=/home/centos/download/hadooppath=$path:$hadoop_home/bin:$hadoop_home/sbinexportpath主機(jī)ip域名映射如下：[root@hadoop1~]#cat/etc/hosts10.0.0.101hadoop110.0.0.102hadoop210.0.0.103hadoop3主節(jié)點(diǎn)ip為：[root@hadoop1~]#ifconfig|awk-f\\\'[:] \\\'\\\'nr==2{print$4}\\\'10.0.0.101從節(jié)點(diǎn)ip分別為：[root@hadoop2~]#ifconfig|awk-f\\\'[:] \\\'\\\'nr==2{print$4}\\\'10.0.0.102[root@hadoop3~]#ifconfig|awk-f\\\'[:] \\\'\\\'nr==2{print$4}\\\'10.0.0.103java版本[root@hadoop1~]#java-versionjavaversion1.8.0_65java(tm)seruntimeenvironment(build1.8.0_65-b17)javahotspot(tm)64-bitservervm(build25.65-b01,mixedmode)hadoop版本[root@hadoop1~]#hadoopversionhadoop2.7.3subversionhttps://git-wip-us.apache.org/repos/asf/hadoop.git-rbaa91f7c6bc9cb92be5982de4719c1c8af91ccffcompiledbyrooton2016-08-18t01:41zcompiledwithprotoc2.5.0fromsourcewithchecksum2e4ce5f957ea4db193bce3734ff29ff4thiscommandwasrunusing/home/centos/download/hadoop-2.7.3/share/hadoop/common/hadoop-common-2.7.3.jar二、配置ssh首先在三臺(tái)機(jī)器上的家目錄分別執(zhí)行下述生成密鑰對(duì)腳本，這樣的話(huà)，三臺(tái)機(jī)器的centos之間就可以免密碼登錄、免密碼傳輸文件等操作。[centos@hadoop1~]$llcreatessh.sh-rwxr-xr-x1centoscentos51may1411:05createssh.sh[centos@hadoop1~]$catcreatessh.sh#!/bin/shssh-keygen-trsa-p\\\'\\\'-f~/.ssh/id_rsa然后在三臺(tái)機(jī)器上的家目錄分別執(zhí)行下述分發(fā)公鑰[centos@hadoop1~]$llfen.sh-rwxr-xr-x1centoscentos142may1412:14fen.sh[centos@hadoop1~]$catfen.sh#!/bin/shssh-copy-id-i~/.ssh/id_rsa.pubcentos@hadoop1ssh-copy-id-i~/.ssh/id_rsa.pubcentos@hadoop2ssh-copy-id-i~/.ssh/id_rsa.pubcentos@hadoop3#上述只是在執(zhí)行的時(shí)候會(huì)輸入yes和系統(tǒng)用戶(hù)密碼，其他在執(zhí)行的時(shí)候就不會(huì)再次輸入密碼了。執(zhí)行ssh的目的是，比如在配置好完全分布式的時(shí)候，主節(jié)點(diǎn)會(huì)自動(dòng)吊起從節(jié)點(diǎn)的日志并且往里面寫(xiě)入數(shù)據(jù)，此時(shí)ssh的密鑰對(duì)就起作用了。#經(jīng)常碰到的坑，要配置三臺(tái)機(jī)器家目錄中.ssh的權(quán)限為700，而且要配置家目錄中.ssh目錄中的authorized_keys權(quán)限為644.[centos@hadoop1~]$chmod700.ssh[centos@hadoop1~]$chmod644.ssh/authorized_keys三、重點(diǎn)的配置文件內(nèi)容#三臺(tái)機(jī)器共同配置[centos@hadoop1~]$grepjava_homedownload/hadoop/etc/hadoop/hadoop-env.sh#theonlyrequiredenvironmentvariableisjava_home.allothersare#setjava_homeinthisfile,sothatitiscorrectlydefinedonexportjava_home=/home/centos/download/jdk#java環(huán)境變量要在這再配置一下[centos@hadoop1~]$catdownload/hadoop/etc/hadoop/core-site.xml<?xmlversion=1.0?><configuration><property><name>fs.defaultfs</name><value>hdfs://hadoop1</value></property></configuration>[centos@hadoop1~]$catdownload/hadoop/etc/hadoop/hdfs-site.xml<?xmlversion=1.0?><configuration><property><name>dfs.replication</name><value>2</value>#副本數(shù)，對(duì)應(yīng)從節(jié)點(diǎn)數(shù)</property></configuration>[centos@hadoop1~]$catdownload/hadoop/etc/hadoop/mapred-site.xml<?xmlversion=1.0?><configuration><property><name>mapreduce.framework.name</name><value>yarn</value></property></configuration>[centos@hadoop1~]$catdownload/hadoop/etc/hadoop/yarn-site.xml<?xmlversion=1.0?><configuration><property><name>yarn.resourcemanager.hostname</name><value>hadoop1</value></property><property><name>yarn.nodemanager.aux-services</name><value>mapreduce_shuffle</value></property></configuration>#注意這里只配置從節(jié)點(diǎn)的域名[centos@hadoop1~]$catdownload/hadoop/etc/hadoop/slaveshadoop2hadoop3四、啟動(dòng)hadoop#下述命令只在主節(jié)點(diǎn)hadoop1中執(zhí)行！#首先執(zhí)行格式化命令[centos@hadoop1~]$hadoopnamenode-format#然后啟動(dòng)hadoop[centos@hadoop1~]$start-all.sh#查看啟動(dòng)進(jìn)程#主節(jié)點(diǎn)執(zhí)行，名稱(chēng)節(jié)點(diǎn)只會(huì)存在于主節(jié)點(diǎn)中，包括文件名稱(chēng)等內(nèi)容。[centos@hadoop1~]$jps2736secondarynamenode2961jps2889resourcemanager25