Name: h2o3-nae
Owner: H2O.ai
Description: null
Created: 2017-04-27 15:50:28.0
Updated: 2018-03-26 14:10:04.0
Pushed: 2018-03-26 14:10:03.0
Homepage: null
Size: 189
Language: Shell
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
H2O scales statistics, machine learning, and math over Big Data.
H2O uses familiar interfaces like R, Python, Scala, the Flow notebook graphical interface, Excel, & JSON so that Big Data enthusiasts & experts can explore, munge, model, and score datasets using a range of algorithms including advanced ones like Deep Learning. H2O is extensible so that developers can add data transformations and model algorithms of their choice and access them through all of those clients.
Data collection is easy. Decision making is hard. H2O makes it fast and easy to derive insights from your data through faster and better predictive modeling. H2O allows online scoring and modeling in a single platform.
While most of this README is written for developers who do their own builds, most H2O users just download and use a pre-built version. If that's you, just follow these steps:
For documentation, please visit http://docs.h2o.ai.
Most people interact with three primary open source resources: GitHub (which you've already found), JIRA (for issue tracking), and h2ostream (a community discussion forum).
(Note: There is only one issue tracking system for the project. GitHub issues are not enabled; you must use JIRA.)
You can browse and create new issues in our open source JIRA: http://jira.h2o.ai
Issues
menuSearch for issues
Log In
button on the top right of the screenCreate an acccount
near the bottom of the login boxCreate
button on the menu to create an issueEvery nightly build publishes R, Python, Java, and Scala artifacts to a build-specific repository. In particular, you can find Java artifacts in the maven/repo directory.
Here is an example snippet of a gradle build file using h2o-3 as a dependency. Replace x, y, z, and nnnn with valid numbers.
2o-3 dependency information
h2oBranch = 'master'
h2oBuildNumber = 'nnnn'
h2oProjectVersion = "x.y.z.${h2oBuildNumber}"
sitories {
h2o-3 dependencies
ven {
url "https://s3.amazonaws.com/h2o-release/h2o-3/${h2oBranch}/${h2oBuildNumber}/maven/repo/"
ndencies {
mpile "ai.h2o:h2o-core:${h2oProjectVersion}"
mpile "ai.h2o:h2o-algos:${h2oProjectVersion}"
mpile "ai.h2o:h2o-web:${h2oProjectVersion}"
mpile "ai.h2o:h2o-app:${h2oProjectVersion}"
Refer to the latest H2O-3 bleeding edge nightly build page for information about installing nightly build artifacts.
Refer to the h2o-droplets GitHub repository for a working example of how to use Java artifacts with gradle.
Note: Stable H2O-3 artifacts are periodically published to Maven Central (click here to search) but may substantially lag behind H2O-3 Bleeding Edge nightly builds.
Getting started with H2O development requires JDK 1.7, Node.js, and Gradle. We use the Gradle wrapper (called gradlew
) to ensure up-to-date local versions of Gradle and other dependencies are installed in your development directory.
To build H2O from the repository, perform the following steps.
ild H2O
clone https://github.com/h2oai/h2o-3.git
2o-3
adlew build -x test
may encounter problems: e.g. npm missing. Install it:
install npm
art H2O
-jar build/h2o.jar
int browser to http://localhost:54321
clone https://github.com/h2oai/h2o-3.git
2o-3
adlew syncSmalldata
adlew syncRPackages
adlew build
Notes:
- Running tests starts five test JVMs that form an H2O cluster and requires at least 8GB of RAM (preferably 16GB of RAM).
- Running
./gradlew syncRPackages
is supported on Windows, OS X, and Linux, and is strongly recommended but not required../gradlew syncRPackages
ensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using./gradlew syncRPackages
. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):r -p ${WORKSPACE}/Rlibrary rt R_LIBS_USER=${WORKSPACE}/Rlibrary
pull
adlew syncSmalldata
adlew syncRPackages
adlew clean
adlew build
We recommend using ./gradlew clean
after each git pull
.
Skip tests by adding -x test
at the end the gradle build command line. Tests typically run for 7-10 minutes on a Macbook Pro laptop with 4 CPUs (8 hyperthreads) and 16 GB of RAM.
Syncing smalldata is not required after each pull, but if tests fail due to missing data files, then try ./gradlew syncSmalldata
as the first troubleshooting step. Syncing smalldata downloads data files from AWS S3 to the smalldata directory in your workspace. The sync is incremental. Do not check in these files. The smalldata directory is in .gitignore. If you do not run any tests, you do not need the smalldata directory.
Running ./gradlew syncRPackages
is supported on Windows, OS X, and Linux, and is strongly recommended but not required. ./gradlew syncRPackages
ensures a complete and consistent environment with pre-approved versions of the packages required for tests and builds. The packages can be installed manually, but we recommend setting an ENV variable and using ./gradlew syncRPackages
. To set the ENV variable, use the following format (where `${WORKSPACE} can be any path):
r -p ${WORKSPACE}/Rlibrary
rt R_LIBS_USER=${WORKSPACE}/Rlibrary
adlew clean && ./gradlew build -x test && (export DO_FAST=1; ./gradlew dist)
target/docs-website/h2o-docs/index.html
Note: The following instructions assume you have installed the latest version of Pip, which is installed with the latest version of Python.
sudo
if unsuccessful)pip install grip
pip install tabulate
pip install wheel
pip install scikit-learn
Python tests require:
pip install scikit-learn
pip install numpy
pip install scipy
pip install pandas
pip install statsmodels
pip install patsy
pip install future
From the command line, validate python
is using the newly installed package by using which python
(or sudo which python
). Update the Environment variable with the WinPython path.
pip install grip
pip install tabulate
pip install wheel
Install Java 1.7 and add the appropriate directory C:\Program Files\Java\jdk1.7.0_65\bin
with java.exe to PATH in Environment Variables. To make sure the command prompt is detecting the correct Java version, run:
javac -version
The CLASSPATH variable also needs to be set to the lib subfolder of the JDK:
CLASSPATH=/<path>/<to>/<jdk>/lib
Install Node.js and add the installed directory C:\Program Files\nodejs
, which must include node.exe and npm.cmd to PATH if not already prepended.
To install these packages from within an R session, enter:
install.packages("RCurl")
install.packages("jsonlite")
install.packages("statmod")
install.packages(c("devtools", "roxygen2", "testthat"))
Install R and add the preferred bin\i386 or bin\x64 directory to your PATH.
Note: Acceptable versions of R are >= 2.13 && <= 3.0.0 && >= 3.1.1.
To manually install packages, download the releases of the following R packages:
cd Downloads
R CMD INSTALL bitops_x.x-x.zip
R CMD INSTALL RCurl_x.xx-x.x.zip
R CMD INSTALL jsonlite_x.x.xx.zip
R CMD INSTALL statmod_x.x.xx.zip
R CMD INSTALL Rcpp_x.xx.x.zip
R CMD INSTALL digest_x.x.x.zip
R CMD INSTALL testthat_x.x.x.zip
R CMD INSTALL stringr_x.x.x.zip
R CMD INSTALL roxygen2_x.x.x.zip
R CMD INSTALL devtools_x.x.x.zip
Finally, install Rtools, which is a collection of command line tools to facilitate R development on Windows.
NOTE: During Rtools installation, do not install Cygwin.dll.
NOTE: During installation of Cygwin, deselect the Python packages to avoid a conflict with the Python.org package.
If Cygwin is already installed, remove the Python packages or ensure that Native Python is before Cygwin in the PATH variable.
If you don't already have a Git client, please install one. The default one can be found here http://git-scm.com/downloads. Make sure that command prompt support is enabled before the installation.
Download and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3
cd h2o-3
./gradlew.bat build
If you encounter errors run again with
--stacktrace
for more instructions on missing dependencies.
If you don't have Homebrew, we recommend installing it. It makes package management for OS X easy.
Install Java 1.7. To make sure the command prompt is detecting the correct Java version, run:
javac -version
Using Homebrew:
brew install node
Otherwise, install from the NodeJS website.
Install R and add the bin directory to your PATH if not already included.
Install the following R packages:
cd Downloads
R CMD INSTALL bitops_x.x-x.tgz
R CMD INSTALL RCurl_x.xx-x.x.tgz
R CMD INSTALL jsonlite_x.x.xx.tgz
R CMD INSTALL statmod_x.x.xx.tgz
R CMD INSTALL Rcpp_x.xx.x.tgz
R CMD INSTALL digest_x.x.x.tgz
R CMD INSTALL testthat_x.x.x.tgz
R CMD INSTALL stringr_x.x.x.tgz
R CMD INSTALL roxygen2_x.x.x.tgz
R CMD INSTALL devtools_x.x.x.tgz
To install these packages from within an R session:
R> install.packages("RCurl")
R> install.packages("jsonlite")
R> install.packages("statmod")
R> install.packages(c("devtools", "roxygen2", "testthat"))
OS X should already have Git installed. To download and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3
cd h2o-3
./gradlew build
If you encounter errors run again with
--stacktrace
for more instructions on missing dependencies.
curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
sudo apt-get install -y nodejs
Install Java 1.7. Installation instructions can be found here JDK installation. To make sure the command prompt is detecting the correct Java version, run:
javac -version
Installation instructions can be found here R installation. Click ?Download R for Linux?. Click ?ubuntu?. Follow the given instructions.
To install the required packages, follow the same instructions as for OS X above.
Note: If the process fails to install RStudio Server on Linux, run one of the following:
sudo apt-get install libcurl4-openssl-dev
or
sudo apt-get install libcurl4-gnutls-dev
If you don't already have a Git client:
sudo apt-get install git
Download and update h2o-3 source codes:
git clone https://github.com/h2oai/h2o-3
cd h2o-3
./gradlew build
If you encounter errors, run again using
--stacktrace
for more instructions on missing dependencies.Make sure that you are not running as root, since
bower
will reject such a run.
curl -sL https://deb.nodesource.com/setup_0.12 | sudo bash -
sudo apt-get install -y nodejs
For users of Intellij's IDEA, generate project files with:
./gradlew idea
For users of Eclipse, generate project files with:
./gradlew eclipse
opt
wget --no-cookies --no-check-certificate --header "Cookie: gpw_e24=http%3A%2F%2Fwww.oracle.com%2F; oraclelicense=accept-securebackup-cookie" "http://download.oracle.com/otn-pub/java/jdk/7u79-b15/jdk-7u79-linux-x64.tar.gz"
tar xzf jdk-7u79-linux-x64.tar.gz
dk1.7.0_79
alternatives --install /usr/bin/java java /opt/jdk1.7.0_79/bin/java 2
alternatives --install /usr/bin/jar jar /opt/jdk1.7.0_79/bin/jar 2
alternatives --install /usr/bin/javac javac /opt/jdk1.7.0_79/bin/javac 2
alternatives --set jar /opt/jdk1.7.0_79/bin/jar
alternatives --set javac /opt/jdk1.7.0_79/bin/javac
opt
wget http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-5.noarch.rpm
rpm -ivh epel-release-7-5.noarch.rpm
echo "multilib_policy=best" >> /etc/yum.conf
yum -y update
yum -y install R R-devel git python-pip openssl-devel libxml2-devel libcurl-devel gcc gcc-c++ make openssl-devel kernel-devel texlive texinfo texlive-latex-fonts libX11-devel mesa-libGL-devel mesa-libGL nodejs npm python-devel numpy scipy python-pandas
pip install scikit-learn grip tabulate statsmodels wheel
r ~/Rlibrary
rt JAVA_HOME=/opt/jdk1.7.0_79
rt JRE_HOME=/opt/jdk1.7.0_79/jre
rt PATH=$PATH:/opt/jdk1.7.0_79/bin:/opt/jdk1.7.0_79/jre/bin
rt R_LIBS_USER=~/Rlibrary
stall local R packages
'install.packages(c("RCurl","jsonlite","statmod","devtools","roxygen2","testthat"), dependencies=TRUE, repos="http://cran.rstudio.com/")'
clone https://github.com/h2oai/h2o-3.git
2o-3
ild H2O
adlew syncSmalldata
adlew syncRPackages
adlew build -x test
java -jar build/h2o.jar
Pre-built H2O-on-Hadoop zip files are available on the download page. Each Hadoop distribution version has a separate zip file in h2o-3.
To build H2O with Hadoop support yourself, first install sphinx for python: pip install sphinx
Then start the build by entering the following from the top-level h2o-3 directory:
(export BUILD_HADOOP=1; ./gradlew build -x test)
./gradlew dist
This will create a directory called 'target' and generate zip files there. Note that BUILD_HADOOP
is the default behavior when the username is jenkins
(refer to settings.gradle
); otherwise you have to request it, as shown above.
In the h2o-hadoop
directory, each Hadoop version has a build directory for the driver and an assembly directory for the fatjar.
You need to:
build.gradle
file) in h2o-hadoop
h2o-3/settings.gradle
HADOOP_VERSIONS
in make-dist.sh
h2o-dist/buildinfo.json
These are the required steps to debug HDFS in IDEA as a standalone H2O process.
Debugging H2O on Hadoop as a hadoop jar
hadoop mapreduce job is a difficult thing to do. However, what you can do relatively easily is tweak the gradle settings for the project so that H2OApp has HDFS as a dependency. Here are the steps:
hadoop-client
version in h2o-persist-hdfs
to the desired versionh2o-persist-hdfs
as a dependency to h2o-app
./gradlew cleanIdea
./gradlew idea
h2o-persist-hdfs
is normally only a dependency of the assembly modules, since those are not used by any downstream modules. We want the final module to define its own version of HDFS if any is desired.
Note this example is for MapR 4, which requires the additional org.json
dependency to work properly.
t diff
--git a/h2o-app/build.gradle b/h2o-app/build.gradle
x af3b929..097af85 100644
a/h2o-app/build.gradle
b/h2o-app/build.gradle
8,5 +8,6 @@ dependencies {
ompile project(":h2o-algos")
ompile project(":h2o-core")
ompile project(":h2o-genmodel")
ompile project(":h2o-persist-hdfs")
--git a/h2o-persist-hdfs/build.gradle b/h2o-persist-hdfs/build.gradle
x 41b96b2..6368ea9 100644
a/h2o-persist-hdfs/build.gradle
b/h2o-persist-hdfs/build.gradle
2,5 +2,6 @@ description = "H2O Persist HDFS"
endencies {
ompile project(":h2o-core")
ompile("org.apache.hadoop:hadoop-client:2.0.0-cdh4.3.0")
ompile("org.apache.hadoop:hadoop-client:2.4.1-mapr-1408")
ompile("org.json:org.json:chargebee-1.0")
Sparkling Water combines two open-source technologies: Apache Spark and H2O, our machine learning engine. It makes H2O?s library of Advanced Algorithms, including Deep Learning, GLM, GBM, K-Means, and Distributed Random Forest, accessible from Spark workflows. Spark users can select the best features from either platform to meet their Machine Learning needs. Users can combine Spark's RDD API and Spark MLLib with H2O?s machine learning algorithms, or use H2O independently of Spark for the model building process and post-process the results in Spark.
Sparkling Water Resources:
Visit http://docs.h2o.ai for the top-level introduction to documentation on H2O projects.
(Source code for the above page is here.)
To generate the REST API documentation, use the following commands:
cd ~/h2o-3
cd py
python ./generate_rest_api_docs.py # to generate Markdown only
python ./generate_rest_api_docs.py --generate_html --github_user GITHUB_USER --github_password GITHUB_PASSWORD # to generate Markdown and HTML
The default location for the generated documentation is build/docs/REST
.
If the build fails, try gradlew clean
, then git clean -f
.
Documentation for each bleeding edge nightly build is available on the nightly build page.
If you use H2O as part of your workflow, please cite your H2O resource(s) using the following BibTex entry:
@Manual{h2o_package_or_module,
title = {package_or_module_title},
author = {The H2O.ai team},
year = {year},
month = {month},
note = {version_information},
url = {resource_url},
}
Formatted H2O Software citation examples:
The H2O.ai team (Oct. 2016). Python Interface for H2O, Python package version 3.10.0.8. [https://github.com/h2oai/h2o-3].
@Manual{h2o_booklet_name,
title = {booklet_title},
author = {list_of_authors},
year = {year},
month = {month},
url = {link_url},
}
Formatted booklet citation examples:
Arora, A., Candel, A., Lanford, J., LeDell, E., and Parmar, V. (Oct. 2016). Deep Learning with H2O. [http://h2o.ai/resources].
Click, C., Lanford, J., Malohlava, M., Parmar, V., and Roark, H. (Oct. 2016). Gradient Boosted Models with H2O. [http://h2o.ai/resources].
We will breathe & sustain a vibrant community with the focus of taking a software engineering approach to data science and empowering everyone interested in data to be able to hack data using math and algorithms. Join us on google groups at h2ostream and feel free to file issues directly on our JIRA.
Team & Committers
atish Ambati
f Click
Kraljevic
s Nykodym
al Malohlava
n Normoyle
cer Aiello
Fu
i Mehta
Candel
phine Wang
Wang
Schloemer
Peck
hvi Prabhu
don Hill
Gambera
l Rao
j Parmar
all Harris
d Avati
ica Lanford
Tellez
son Washburn
Wang
Eckstrand
aja Madabhushi
stian Vidrio
Sabrin
Dowle
Landry
LeDell
Rogynskyy
Martin
y Jordan
ant Kalonia
ne Hussami
Cramer
ie Spreitzer
d Iyengar
lene Windom
g Sanghavi
Scientific Advisory Council
hen Boyd
Tibshirani
or Hastie
Systems, Data, FileSystems and Hadoop
Lea
s Pouliot
ba Borthakur
nu Bhattacharjee, Nexus Venture Partners
d Babu Periasamy
d Rajaraman
Bhardwaj
sh Mathur
ael Marks
rt Bierman
sh Ambati