CD2H gitForager

elodina/hdfs-mesos

Name: hdfs-mesos

Owner: Elodina

Description: Apache Hadoop HDFS Data Node Scheduler

Created: 2016-02-09 16:06:45.0

Updated: 2017-10-19 10:05:16.0

Pushed: 2016-06-04 18:44:56.0

Homepage: http://www.elodina.net

Size: 3494

Language: Java

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

Intro

This project allows running HDFS on Mesos.

You should be familiar with HDFS and Mesos basics:

http://mesos.apache.org/documentation/latest/
https://hadoop.apache.org/docs/r2.7.2/hdfs_design.html

Project requires:

Mesos 0.23.0+
JDK 1.7.x
Hadoop 1.2.x or 2.7.x

Mesos in Vagrant

Project includes vagrant environment, that allows to run Mesos cluster locally.

If you are going to use external Mesos cluster, you can skip this section.

1. Start vagrant nodes:

 hdfs-mesos/vagrant
grant up

It creates mesos master and slave nodes.

2. Add vagrant node names to /etc/hosts

Now Mesos in vagrant should be running. You can proceed with starting scheduler.

For more details about vagrant environment please read vagrant/README.md

Running Scheduler

1. Download hdfs-mesos\*.jar OR clone & build the project:

Download jar:

dir hdfs-mesos
 hdfs-mesos
et https://github.com/elodina/hdfs-mesos/releases/download/0.0.1.0/hdfs-mesos-0.0.1.0.jar

OR clone & build:

t clone https://github.com/elodina/hdfs-mesos.git
 hdfs-mesos
gradlew jar

2. Download hadoop tarball:

et https://archive.apache.org/dist/hadoop/core/hadoop-2.7.2/hadoop-2.7.2.tar.gz

3. Start scheduler:

hdfs-mesos.sh scheduler --api=http://$scheduler:7000 --master=zk://$master:2181/mesos --user=vagrant
-03-18 15:04:48,785 [main] INFO hdfs.Scheduler - Starting Scheduler:
 http://$scheduler:7000
s: jar:./hdfs-mesos-0.0.1.0.jar, hadoop:./hadoop-1.2.1.tar.gz
s: master:master:5050, user:vagrant, principal:<none>, secret:<none>
ework: name:hdfs, role:*, timeout:30d
-03-18 15:04:48,916 [main] INFO hdfs.HttpServer - started on port 7000
8 15:04:49.008314 19123 sched.cpp:164] Version: 0.25.0
8 15:04:49.017160 19155 sched.cpp:262] New master detected at master@192.168.3.5:5050
8 15:04:49.019287 19155 sched.cpp:272] No credentials provided. Attempting to register without authentication
8 15:04:49.029218 19155 sched.cpp:641] Framework registered with 20160310-141004-84125888-5050-10895-0006
-03-18 15:04:49,044 [Thread-17] INFO hdfs.Scheduler - [registered] framework:#-0006 master:#326bb pid:master@192.168.3.5:5050 hostname:master
-03-18 15:04:49,078 [Thread-18] INFO hdfs.Scheduler - [resourceOffers]
e0#-O761 cpus:1.00; mem:2500.00; disk:35164.00; ports:[5000..32000]
er#-O762 cpus:1.00; mem:2500.00; disk:35164.00; ports:[5000..32000]

-03-18 15:04:49,078 [Thread-18] INFO hdfs.Scheduler - [resourceOffers]

where:

$scheduler is scheduler address accessible from slave nodes;
$master master address accessible from scheduler node;

Scheduler should register itself and start receiving resource offers. If scheduler is not receiving offers it could be required to specify LIBPROCESS_IP:

port LIBPROCESS_IP=$scheduler_ip

Now scheduler should be running and you can proceed with starting HDFS nodes.

Running HDFS Cluster

Project provides CLI & REST API for managing HDFS nodes. We will focus first on CLI.

1. Add namenode & datanode:

hdfs-mesos.sh node add nn --type=namenode
 added:
: nn
pe: namenode
ate: idle
sources: cpus:0.5, mem:512

hdfs-mesos.sh node add dn0 --type=datanode
 added:
: dn0
pe: datanode
ate: idle
sources: cpus:0.5, mem:512

2. Start nodes:

hdfs-mesos.sh node start \*
s started:
: nn
pe: namenode
ate: running
sources: cpus:0.5, mem:512
servation: cpus:0.5, mem:512, ports:http=5000,ipc=5001
ntime:
task: 383aaab9-982b-400e-aa35-463e66cdcb3b
executor: 19065e07-a006-49a4-8f2b-636d8b1f2ad6
slave: 241be3a2-39bc-417c-a967-82b4018a0762-S0 (master)

: dn0
pe: datanode
ate: running
sources: cpus:0.5, mem:512
servation: cpus:0.5, mem:512, ports:http=5002,ipc=5003,data=5004
ntime:
task: 37f3bcbb-10a5-4323-96d2-aef8846aa281
executor: 088463c9-5f2e-4d1d-8195-56427168b86f
slave: 241be3a2-39bc-417c-a967-82b4018a0762-S0 (master)

Nodes are up & running now.

Note: starting may take some time. You can view the progress via Mesos UI.

3. Do some FS operations:

doop fs -mkdir hdfs://master:5001/dir
doop fs -ls hdfs://master:5001/
d 1 items
r-xr-x   - vagrant supergroup          0 2016-03-17 12:46 /dir

Note: namenode host and ipc port is used in fs url.

Using CLI

Project provides CLI with following structure:

hdfs-mesos.sh help
e: <cmd> ...

ands:
lp [cmd [cmd]] - print general or command-specific help
heduler        - start scheduler
de             - node management

Help is provided for each command and sub-command:

hdfs-mesos.sh help node
 management commands
e: node <cmd>

ands:
st       - list nodes
d        - add node
date     - update node
art      - start node
op       - stop node
move     - remove node

`help node <cmd>` to see details of specific command

hdfs-mesos.sh help node add
node
e: node add <ids> [options]

on (* = required)  Description
-----------------  -----------
re-site-opts       Hadoop core-site.xml options.
us <Double>        CPU amount (0.5, 1, 2).
ecutor-jvm-opts    Executor JVM options.
doop-jvm-opts      Hadoop JVM options.
fs-site-opts       Hadoop hdfs-site.xml options.
m <Long>           Mem amount in Mb.
type               node type (name_node, data_node).

ric Options
on  Description
--  -----------
i   REST api url (same as --api option for
      scheduler).

All node-related commands support bulk operations using node-id-expressions. Examples:

hdfs-mesos.sh node add dn0..1 --type=datanode
s added:
: dn0
pe: datanode
.

: dn1
pe: datanode
.

hdfs-mesos.sh node update dn* --cpus=1
s updated:
: dn0
.
sources: cpus:1.0, mem:512

: dn1
.
sources: cpus:1.0, mem:512

hdfs-mesos.sh node start dn0,dn1
s started:
: dn0
.

: dn0
.

Id expression examples:

nn - matches node with id nn
* - matches any node (should be slash-escaped in shell)
dn* - matches node with id starting with dn
dn0..2 - matches nodes dn0, dn1, dn2

Using REST

Scheduler uses embedded HTTP server. Server serves two functions:

distributing binaries of Hadoop, JRE and executor;
serving REST API, invoked by CLI;

Most CLI commands map to REST API call. Examples:

REST calls accepts plain HTTP params and return JSON responses. Examples:

rl http://$scheduler:7000/api/node/list

{
    "id": "nn",
    "type": "namenode",
    ...
},
{
    "id": "dn0",
    "type": "datanode",
    ...
}


rl http://$scheduler:7000/api/node/start?node=nn,dn0

"status": "started",
"nodes": [
    {
        "id": "nn",
        "state": "running",
        ...
    },
    {
        "id": "dn0",
        "state": "running",
        ...
    }
]

CLI params maps one-to-one to REST params. CLI params use dashed style while REST params use camel-case. Example of mappings:

REST API call could return error in some cases. Errors are marked with status code other than 200. Error response is returned in JSON format.

Example:

rl -v http://192.168.3.1:7000/api/node/start?node=unknown

/1.1 400 node not found

ror":"node not found","code":400}

For more detail on REST API please refer to sources.

Having Issue

Please read this README carefully, to make sure you problem is not already described.

Also make sure that your issue is not duplicating any existing one.

DO NOT post general question like “I am having problem with mesos” to the issue list. Please use generic QA sites like http://stackoverflow.com for that.

General rules for posting issues are:

be precise: provide a minimalistic reproduce-scenario;
provide details: provide all required log snippets (stdout/err from scheduler or stdout/err from executor's sandbox, mesos logs if required);
be helpful: provide PR for the bug-fix if possible;

Issues list: https://github.com/elodina/hdfs-mesos/issues

This work is supported by the National Institutes of Health's National Center for Advancing Translational Sciences, Grant Number U24TR002306. This work is solely the responsibility of the creators and does not necessarily represent the official views of the National Institutes of Health.