Intel-bigdata/mkl_wrapper_for_non_CDH

Name: mkl_wrapper_for_non_CDH

Owner: Intel-bigdata

Description: null

Created: 2017-05-23 09:51:26.0

Updated: 2018-03-29 23:50:32.0

Pushed: 2017-05-23 10:05:24.0

Homepage: null

Size: 247

Language: null

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

mklWrapper

An MKL wrapper which directly bind with Intel MKL.

Setting and Testing:

• Assume the MKL installation location is /opt/intel/mkl (you cannot just copy libmkl_rt.so).

• Copy mkl_wrapper.jar to /opt/intel/mkl/wrapper/mkl_wrapper.jar

• Copy mkl_wrapper.so to /opt/intel/mkl_wrapper/mkl_wrapper.so

• By default, the MKL wrapper will search /opt/intel/mkl/wrapper to find mkl_wrapper.so. If you put it on other path, please use -Dcom.intel.mkl.wrapper=/path/to/find/mkl_wrapper.so.

• Spark configuration. (on each node):

• Create: /etc/ld.so.conf.d/mkl_blas.conf

• Input: /opt/intel/mkl/lib/intel64 (your MKL library Dir) to mkl_blas.conf

• Run: ldconfig

• A simple sanity check to see if native MKLBLAS/LAPACK libs can be properly loaded:

  bin/spark-shell –driver-class-path $CLASSPATH:/opt/intel/mkl/wrapper/mkl_wrapper.jar

scala> import com.github.fommil.netlib.BLAS;

scala> System.setProperty(“com.github.fommil.netlib.BLAS”, “com.intel.mkl.MKLBLAS”)

scala> System.out.println(BLAS.getInstance().getClass().getName());

   OUTPUT: com.intel.mkl.MKLBLAS

scala> import com.github.fommil.netlib.LAPACK;

scala> System.setProperty(“com.github.fommil.netlib.LAPACK”, “com.intel.mkl.MKLLAPACK”)

scala> System.out.println(LAPACK.getInstance().getClass().getName());

   OUTPUT: com.intel.mkl.MKLLAPACK

How to use it:

• It is simple to use this MKL Wrapper, just with the following option to submit your job.
• -Dcom.github.fommil.netlib.BLAS=com.intel.mkl.MKLBLAS
• -Dcom.github.fommil.netlib.LAPACK=com.intel.mkl.MKLLAPACK

Example 1 (check MKL can be successfully loaded):

• bin/spark-submit –master spark://sr576:7077
• –class TestMKLWrapper
• –conf “spark.executor.extraJavaOptions=-Dcom.github.fommil.netlib.BLAS=com.intel.mkl.MKLBLAS -Dcom.github.fommil.netlib.LAPACK=com.intel.mkl.MKLLAPACK”
• –conf “spark.executor.extraClassPath=/opt/intel/mkl/wrapper/mkl_wrapper.jar”
• –conf “spark.executorEnv.MKL_VERBOSE=1”
• target/scala-2.11/test-netflix-mllib_2.11-1.0.jar

Example 1 (Real test):

• bin/spark-submit –master spark://sr576:7077
• –class TestMKLWrapper
• –conf “spark.executor.extraJavaOptions=-Dcom.github.fommil.netlib.BLAS=com.intel.mkl.MKLBLAS -Dcom.github.fommil.netlib.LAPACK=com.intel.mkl.MKLLAPACK”
• –conf “spark.executor.extraClassPath=/opt/intel/mkl/wrapper/mkl_wrapper.jar”
• target/scala-2.11/test-netflix-mllib_2.11-1.0.jar

For some MLLIB algorithm (SVD), you also need to set driver extraJavaOptions and driver extraClassPath