Yarn

Four Components

Client request job to Resource Manager

Resource Manager

• handles Client request
• Start/Monitor ApplicationMaster
• Monitor NodeManager
• Resource allocation and call

NodeManger

Node Manager sends status heartbeat to Resource Manager

• single node self resource management
• handle commands from ResourceManager
• handle commands from ApplicationMaster

Container: (in NodeManager)a unit of Resource(Memory, disk size, cpu, network io, disk io),  there will be a job task in each Container. Some task in Container also will send Task Status to App Master.

App Master: (in NodeManager)Manager(monitor, manage) for one app. Calculate how much Container it needs, then request the Container from Resource Manager. Once get Container, run the app.

• cut data to blocks
• request resource from Resource Manager, and allocate resource to jobs
• monitor jobs and failure compatible

Example: MapReduce App Master, MPI App Master, etc..

MapReduce in Yarn flow