Thursday, May 12, 2016

Hadoop 2 CentOS build(from pseudo to real distributed)

We gonna have four machines:
192.168.56.95    hadoop-yarn.gvace.com       hadoop-yarn
192.168.56.96    hadoop-node1.gvace.com     hadoop-node1
192.168.56.97    hadoop-node2.gvace.com     hadoop-node2
192.168.56.98    hadoop-node3.gvace.com     hadoop-node3
We can have all of them as virtual machines, start from the first one, then clone hadoop-yarn to other nodes.

We will start with building a pseudo distributed first, start with only hadoop-yarn.gvace.com(I will use VirtualBox), which runs everything by it self.
After everything tests ok in pseudo distributed, we move on to real distributed(I will use VirtualBox to build all servers).

Linux Version
CentOS-7-x86_64-Minimal-1511.iso
In VirtualBox

Hadoop Version
hadoop-2.6.4

JDK Version
OpenJDK 7

Disable iptable
File: /etc/sysconfig/selinux
Config: SELINUX=disable

Network config:

Choose your own host-only interface which set from virtualbox, mine here is enp0s8
Set a static IP with the same network from host machine
/etc/sysconfig/network-scripts/ifcfg-enp0s8 
DEVICE=enp0s8
BOOTPROTO=none
ONBOOT=yes
NETWORK=192.168.56.0
NETMASK=255.255.255.0
IPADDR=192.168.56.95
USERCTL=no

If you want to connect internet(Optional)
This version of CentOS does not provide default DNSChoose your own NAT interface which set from virtualbox, mine here is enp0s3
/etc/sysconfig/network-scripts/ifcfg-enp0s3
Config
BOOTPROTO=dhcp
ONBOOT=yes
DNS1=8.8.8.8
DNS2=4.2.2.2 
Install net-tools for convenient, like ifconfig commands(Optional)
sudo yum install net-tools

Change Hostname
Change to my own HOSTNAME in /etc/sysconfig/network
hostname hadoop-yarn.gvace.com
Also change in hosts /etc/hosts
127.0.0.1   localhost
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

192.168.56.95    hadoop-yarn.gvace.com      hadoop-yarn
192.168.56.96    hadoop-node1.gvace.com     hadoop-node1
192.168.56.97    hadoop-node2.gvace.com     hadoop-node2
192.168.56.98    hadoop-node3.gvace.com     hadoop-node3

Note:Standard config: IP followed by hostname with domain followed by hostname

Also change in here /etc/hostname
hadoop-yarn
This may not necessary, but let's put it here first.

Setup SSH

Some intro to ssh
http://gvace.blogspot.com/2016/04/ssh.html

For distributed hadoop, it's all related to ssh communication.
So the Namenode server required to talk to any of the Datanode server.
One first step is to call "start-dfs.sh", and this script is using ssh to communicate with other servers.
We need to enable SSH without password from Namenode server to any slave.

For pseudo hadoop server, it also required to ssh to itself if we use "start-dfs.sh" like scripts.

So we can try start-dfs.sh first, let's see what server/ip/hostname it requesting to have ssh login.


As we see, start-dfs.sh requested three times from me to enter password

  1. namenode: yushan@hadoop-yarn.gvace.com
  2. datanode: 0.0.0.0
  3. secondarynamenode: 0.0.0.0

We know 0.0.0.0 means localhost, but when we are in distributed hadoop, which is normally we will do, there will be a lot different datanodes with different hostname/ip. So all these required to have remote ssh without password.

So in Namenode server
First, generate public key and private key: ssh-keygen -t rsa
Second, copy my public key to myself: ssh-copy-id yushan@localhost  this solves ssh to 0.0.0.0
Third, copy my public key to any other servers: ssh-copy-id yushan@datanode1
In real world, when starting start-dfs.sh and start-yarn.sh, we may have more times to enter password.
So remember these host names that requiring password, and then copy public key to those hosts. This will make sure your hadoop system runs without manually typing password.




Install JDK
sudo yum install java-1.7.0-openjdk-devel
If install succeed, we can try command java and javac to verify.
Use "whereis java" to find where it installed, you will jump through it's link and finally find the install location
Here it is: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.101-2.6.6.1.el7_2.x86_64
This is too long. Luckily we have also several links which are much shorter in /usr/lib/jvm.
It's pointing from /etc/alternatives/, and all finally point to same jdk directory
We can use /usr/lib/jvm/java-1.7.0 as $JAVA_HOME

Set JDK env
Create file /etc/profile.d/java.sh
export JAVA_HOME=/usr/lib/jvm/java-1.7.0export PATH=.:$JAVA_HOME/bin:$PATH
Update new configs: source /etc/profile

Install Hadoop

download hadoop-2.6.4.tar.gz
extract it to /usr/lib/

Config files

We can find default xml files in source code for core/hdfs/yarn/mapred. So we can pick up the things we need to change from the default xml files, and put them into our xml files.

In /usr/lib/hadoop/conf
  1. hadoop-env.sh
  2. yarn-env.sh(optional)
  3. mapred-env.sh(optional)
  4. core-site.xml
  5. hdfs-site.xml
  6. yarn-site.xml
  7. mapred-site.xml
hadoop-env.sh

Change JAHA_HOME to jdk directory
export JAVA_HOME=/usr/lib/jvm/java-1.7.0

yarn-env.sh(optional)
Uncomment and change:
export JAVA_HOME=/usr/lib/jvm/java-1.7.0

mapred-env.sh(optional)
Uncomment and change:
export JAVA_HOME=/usr/lib/jvm/java-1.7.0

core-site.xml
Assign hdfs default name, data/tmp directory, manually create this directory
This need to be point to hostname of Namenode
<configuration>
        <property>
                <name>fs.default.name</name>
                <value>hdfs://hadoop-yarn.gvace.com:8020</value>
                <description>NameNode</description>
        </property>
        <property>
                <name>hadoop.tmp.dir</name>
                <value>/usr/lib/hadoop-2.6.4/data/tmp</value>
        </property>
</configuration>

I just found that fs.default.name is Deprecated in hadoop2. We can use fs.defaultFS instead.

hdfs-site.xml
dfs.replication default value is 3
Rule: replication number must NOT bigger than datanode number.

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>1</value>
        </property>
        <property>
                <name>dfs.permissions</name>
                <value>false</value>
        </property>
</configuration>


yarn-site.xml

<configuration>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
</configuration>


mapred-site.xml
<configuration>
        <property>
                <name>mapreduce.framework.name</name>
                <value>yarn</value>
        </property>
</configuration>


Format Hadoop

hadoop namenode -format

We may find some issue by user permissions.

This is the error I got.
16/05/02 00:47:35 INFO namenode.FSNamesystem: fsOwner=yushan
16/05/02 00:47:35 INFO namenode.FSNamesystem: supergroup=supergroup
16/05/02 00:47:35 INFO namenode.FSNamesystem: isPermissionEnabled=false
16/05/02 00:47:35 INFO namenode.FSNamesystem: dfs.block.invalidate.limit=100
16/05/02 00:47:35 INFO namenode.FSNamesystem: isAccessTokenEnabled=false accessKeyUpdateInterval=0 min(s), accessTokenLifetime=0 min(s)
16/05/02 00:47:36 INFO namenode.FSEditLog: dfs.namenode.edits.toleration.length = 0
16/05/02 00:47:36 INFO namenode.NameNode: Caching file names occuring more than 10 times 
java.io.IOException: Cannot create directory /user/lib/hadoop-2.6.4/data/tmp/dfs/name/current

As wee see, by default, hadoop user is the creating user's username, and group is "supergroup"
We can change the whole folder /usr/lib/hadoop-2.6.4 and all its sub forlder/files to the username and group,(create the group name if not exist) and assign 755 on the whole folder. This will be fixed.

Cannot do hadoop namenode -format more than once
If want to format more than once, we can delete all files from /user/lib/hadoop-2.6.4/data/tmp/ folder, and format again

The namenode format generates a Cluster ID
You can also assign a Cluster ID if needed
bin/hdfs namenode -format -clusterid yarn-cluster

namenode.HostFileManager: read includes:
HostSet(
)
namenode.HostFileManager: read excludes:
HostSet(
)


Safe Mode

  • When namenode starts, it loads fsimage into memory, and run operations from edits.
  • Once the memory built, it creates a new fsimage and an empty edits
  • Right in this moment, namenode is in Safe Mode. Now Namenode is read-only for clients
  • In this moment, Namenode collects reports from datanodes. When data block reached smallest replication number line, it can be seen as "Safe". 
  • When a portion(configurable) of data blocks are "Safe", it waits for a moment, then exit Safe Mode.
  • When detect replication number of data block is lower than smallest line, that datablock will be duplicating until it reached smallest line.
    Datablock's position is not determined by Namenode, it is stored in datanode as block list.


If Safe Mode is off, that means in hdfs-site.xml, we set dfs.permissions to false
This means any user can enter the hdfs do file create/edits/delete to hdfs



Different ways of starting Hadoop services
We have the following scripts to start/stop services
  • start-all.sh (Deprecated)
  • start-dfs.sh
  • start-yarn.sh
  • hadoop-daemons.sh
  • hadoop-daemon.sh
  • yarn-daemons.sh
  • yarn-daemon.sh
  • slaves.sh
Looking at each files, they are actually calling each other
If use start-dfs.sh and start-yarn.sh, they require ssh in slave sub scripts, so it can start all services from remote. After ssh to remote server, it uses hadoop-daemon.sh to start each individual services.
Single pseudo mode 

Start HDFS

NameNode, DataNode, SecondaryNameNode

Start namenode
sbin/hadoop-daemon.sh start namenode

Start datanode
sbin/hadoop-daemon.sh start datanode

Start secondarynamenode
sbin/hadoop-daemon.sh start secondarynamenode

Now goto http://hadoop-yarn.gvace.com:50070/ we will see namenode info from webpage

Now goto http://hadoop-yarn.gvace.com:50090/ we will see secondarynamenode info from webpage

Start YARN
ResourceManager, NodeManager

Start ResourceManager
sbin/yarn-daemon.sh start resourcemanager

Start NodeManager

sbin/yarn-daemon.sh start nodemanager

Now goto http://hadoop-yarn.gvace.com:8088/ we will see yarn info from webpage
Now goto http://hadoop-yarn.gvace.com:8042/ we will see yarn node info from webpage

Stop HDFS and YARN

NameNode, DataNode, SecondaryNameNode, ResourceManager, NodeManager

Stop namenode
sbin/hadoop-daemon.sh stop namenode

Stop datanode
sbin/hadoop-daemon.sh stop datanode

Stop secondarynamenode
sbin/hadoop-daemon.sh stop secondarynamenode

Stop ResourceManager
sbin/yarn-daemon.sh stop resourcemanager

Stop NodeManager
sbin/yarn-daemon.sh stop nodemanager

Distributed Mode

All things above are for pseudo mode. Once we done pseudo mode, we can easily apply the mostly same configs to slaves(datanodes). If all nodes are virtual box, we can clone the Namenode, then apply a little config changes to the new nodes.(Remember select to reinitialize MAC address for all network cards when clone in VirtualBox/VM)

Before we clone the machine, we need to do some config changes for the Namenode.

hdfs-site.xml
We increase dfs.replication to 3(which default is also 3)
As we know, safe mode is related to dfs.replication, so the number of dfs.replication required to be lower/equal to the number of slaves we have.
In this case, we have 3 slaves, so 3 replication is just about good.

We also move SecondaryNameNode from hadoop-yarn server to hadoop-node1, normally NameNode and SecondaryNameNode are not in the save server.

<configuration>
        <property>
                <name>dfs.replication</name>
                <value>3</value>
        </property>
        <property>
                <name>dfs.permissions</name>
                <value>false</value>
        </property>
        <property>
                <name>dfs.namenode.secondary.http-address</name>
                <value>hadoop-node1.gvace.com:50090</value>
        </property>
        <property>
                <name>dfs.namenode.secondary.https-address</name>
                <value>hadoop-node1.gvace.com:50091</value>
        </property>
</configuration>

yarn-site.xml
yarn will still run in Namenode, I just replace default value of 0.0.0.0 to real hostname hadoop-yarn.

<configuration>

<!-- Site specific YARN configuration properties -->
        <property>
                <description>The hostname of the RM.</description>
                <name>yarn.resourcemanager.hostname</name>
                <value>hadoop-yarn.gvace.com</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        <property>
                <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
                <value>org.apache.hadoop.mapred.ShuffleHandler</value>
        </property>
</configuration>


slaves
The slaves config file located in the same folder as hdfs-site.xml and yarn-site.xml
Add the following lines to slaves, which points out all nodes

hadoop-node1.gvace.com
hadoop-node2.gvace.com
hadoop-node3.gvace.com

Up to here, we can shutdown Namenode and start clone if you also using VM for all the nodes.

Again, here is the list of all four boxes:
192.168.56.95    hadoop-yarn.gvace.com
192.168.56.96    hadoop-node1.gvace.com
192.168.56.97    hadoop-node2.gvace.com
192.168.56.98    hadoop-node3.gvace.com

The goal is: Use start-dfs.sh and start-yarn.sh from Namenode without password, this require Namenode can ssh to all other nodes without password.
To do this, make sure all slaves hosts has ~/.ssh/authorized_keys of Namenode's public key.
(If before clone, you can ssh from Namenode host to Namenode hostname itself without passowrd, then probably we will be OK, since each cloned host will have Namenode's public key)

We need to change the following configs for each node, as node1, node2, node3
change the IP:
/etc/sysconfig/network-scripts/ifcfg-enp0s8
change hostname: /etc/sysconfig/network
change hostname: /etc/hostname

After all nodes configured, we will do start-dfs.sh from Namenode manually once, from command line. 
If prompt to ask for yes/no, type yes.(For the first time running start-dfs.sh in distributed mode, the lines of start-dfs.sh output may be queued up. So when you see output contains "yes/no", just type "yes", no matter which output line you are in.)

All we want to make sure is: ssh can go through from Namenode to every other nodes without password.

This is the final result when we try start-dfs.sh.
This is the final result when we try start-yarn.sh.


The web url will change some based on which service running on which server
NameNode: http://hadoop-yarn:50070/

SecondaryNameNode: http://hadoop-node1.gvace.com:50090/
Yarn(Resource Manager): http://hadoop-yarn.gvace.com:8088/
Each slave server has a Node Manager
Node Manager: http://hadoop-node1.gvace.com:8042/
Node Manager: http://hadoop-node2.gvace.com:8042/
Node Manager: http://hadoop-node3.gvace.com:8042/

MapReduce History Server
Provides MapReduce's working logs, for example: Number of Maps, Number of Reduce, job submit time, job start time, job finish time, etc..
When we click on History of application, it's showing ERR_CONNECTION_REFUSED
It's pointing to port 19888, but port 19888 is not opened by default.

using netstat -tnlp to see the port it's really not opened.

To Start History Server
sbin/mr-jobhistory-daemon.sh start historyserver

Web UI
http://hadoop-yarn.gvace.com:19888/

To Stop History Server
sbin/mr-jobhistory-daemon.sh stop historyserver

































No comments:

Post a Comment