Name: spark-tpc-ds-performance-test
Owner: International Business Machines
Description: Use the TPC-DS benchmark to test Spark SQL performance
Created: 2017-09-12 19:54:03.0
Updated: 2018-01-18 02:04:43.0
Pushed: 2017-12-01 19:46:45.0
Size: 372324
Language: C
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
Apache Spark is a popular distributed data processing engine that is built around speed, ease of use and sophisticated analytics, with APIs in Java, Scala, Python, R, and SQL. Like other data processing engines, Spark has a unified optimization engine that computes the optimal way to execute a workload with the main purpose of reducing the disk IO and CPU usage.
We can evaluate and measure the performance of Spark SQL using the TPC-DS benchmark. TPC-DS is a widely used industry standard decision support benchmark that is used to evaluate performance of data processing engines. Given that TPC-DS exercises some key data warehouse features, running TPC-DS successfully reflects the readiness of Spark in terms of addressing the need of a data warehouse application. Apache Spark v2.0 supports all the ninety-nine decision support queries that is part of this TPC-DS benchmark.
This Code Pattern is aimed at helping Spark developers quickly setup and run the TPC-DS benchmark in their own development setup.
When the reader has completed this Code Pattern, they will understand the following:
There are two modes of exercising this Code Pattern:
Clone the spark-tpc-ds-performance-test
repo locally. In a terminal, run:
t clone https://github.com/IBM/spark-tpc-ds-performance-test
Make sure the required development tools are installed in your platform. This Code Pattern is supported on Mac and Linux platforms only. Depending on your platform, run the following command to install the necessary development tools:
To successfully run the TPC-DS tests, Spark must be installed and pre-configured to work with an Apache Hive metastore.
Perform 1 or more of the following options to ensure that Spark is installed and configured correctly. Once completed, modify `bin/tpcdsenv.sh
to set
SPARK_HOME` pointing to your Spark installation directory.
Option 1 - If you already have Spark installed, complete the following steps to ensure your Spark version is properly configured:
$SPARK_HOME
n/spark-shell
Enter the following command at the scala prompt
ala> spark.conf
ale> spark.conf.get("spark.sql.catalogImplementation")
s5: String = hive
ala> <ctrl-c>
Note: You must exit out of the spark-shell process or you will encounters errors when performing the TPC-DS tests.
If the prompt returns String = hive
, then your installation is properly configured.
Option 2 - If you don't have an installed Spark version, or your current installation is not properly configured, we suggest trying to pull down version 2.2.0 from the Spark downloads page. This version should be configured to work with Apache Hive, but please run the test in the previous option to make sure.
Option 3 - The last option available is it to download and build it yourself. The first step is to clone the Spark repo:
t clone https://github.com/apache/spark.git
Then build it using these instructions. Please make sure to build Spark with Hive support by following the Building With Hive and JDBC Support
section.
Note: Verify that the bin/tpcdsenv.sh
script has SPARK_HOME
setup correctly.
Now that we have Spark setup and the TPC-DS scripts downloaded, we are ready to setup and start running the TPC-DS queries using the bin/tpcdsspark.sh
utility script. This driver script will allow you to compile the TPC-DS toolkit to produce the data and the queries, and then run them to collect results.
Perform the following steps to complete the execution of the script:
d spark-tpc-ds-performance-test
in/tpcdsspark.sh
==========================================
DS On Spark Menu
------------------------------------------
P
Compile TPC-DS toolkit
Generate TPC-DS data with 1GB scale
Create spark tables
Generate TPC-DS queries
Run a subset of TPC-DS queries
Run All (99) TPC-DS Queries
NUP
Cleanup toolkit
Quit
------------------------------------------
se enter your choice followed by [ENTER]:
The most recent toolkit can be downloaded from http://www.tpc.org/tpcds/. To make it easier for users, a toolkit based on v2.4
is available locally in src/toolkit. If you download the newer toolkit from the official tpc-ds site, then make sure you overlay the code in src/toolkit before proceeding with this option.
This option compiles the toolkit to produce the data generation (dsdgen) and query generation (dsqgen) binaries.
Below is the screen-shot when this option is chosen.
==========================================
DS On Spark Menu
------------------------------------------
P
Compile TPC-DS toolkit
Generate TPC-DS data with 1GB scale
Create spark tables
Generate TPC-DS queries
Run a subset of TPC-DS queries
Run All (99) TPC-DS Queries
NUP
Cleanup toolkit
Quit
------------------------------------------
se enter your choice followed by [ENTER]: 1
------------------------------------------
: Starting to compile..
: make OS=MACOS
: Completed building toolkit successfully..
s any key to continue
This option uses the data generation binary produced in the previous step to generate the test data at a 1GB scale factor. The data is generated in the directory TPCDS_GENDATA_DIR
. The default location of TPCDS_GENDATA_DIR
is the local directory gendata
. This can be changed by modifying the script bin/tpcdsenv.sh
.
Technically, this option can be used to generate data at a different scale. However, since this Code Pattern is targeted towards the developer environment, the scale has been fixed at 1GB. To modify this script to generate data at a different scale factor, see the discussion in the scaling upto 100TB
section below.
Below is the screenshot when this option is chosen.
==========================================
DS On Spark Menu
------------------------------------------
P
Compile TPC-DS toolkit
Generate TPC-DS data with 1GB scale
Create spark tables
Generate TPC-DS queries
Run a subset of TPC-DS queries
Run All (99) TPC-DS Queries
NUP
Cleanup toolkit
Quit
------------------------------------------
se enter your choice followed by [ENTER]: 2
------------------------------------------
: Starting to generate data. Will take a few minutes ...
: Progress : [########################################] 100%
: TPCDS data is generated successfully at spark-tpc-ds-performance-test/gendata
s any key to continue
After data generation has completed, this option creates the tables in the database name specified by TPCDS_DBNAME
defined in bin/tpcdsenv.sh
. The default name is TPCDS
but can be changed if needed.
The SQL statements to create the tables can be found in src/ddl/create_tables.sql
, and are created in parquet format.
Below is the screenshot when this option is chosen.
==========================================
DS On Spark Menu
------------------------------------------
P
Compile TPC-DS toolkit
Generate TPC-DS data with 1GB scale
Create spark tables
Generate TPC-DS queries
Run a subset of TPC-DS queries
Run All (99) TPC-DS Queries
NUP
Cleanup toolkit
Quit
------------------------------------------
se enter your choice followed by [ENTER]: 3
------------------------------------------
: Creating tables. Will take a few minutes ...
: Progress : [########################################] 100%
: Spark tables created successfully..
s any key to continue
This option uses the query generation binary (dsqgen) produced in “option (1)” to generate the 99 TPC-DS queries. The queries are generated in the TPCDS_GEN_QUERIES_DIR
, with a default location of genqueries
. This can be changed my modifying the `bin/tpcdsenv.sh' script.
Below is the screenshot when this option is chosen.
==========================================
DS On Spark Menu
------------------------------------------
P
Compile TPC-DS toolkit
Generate TPC-DS data with 1GB scale
Create spark tables
Generate TPC-DS queries
Run a subset of TPC-DS queries
Run All (99) TPC-DS Queries
NUP
Cleanup toolkit
Quit
------------------------------------------
se enter your choice followed by [ENTER]: 4
------------------------------------------
: Generating TPC-DS qualification queries.
: Completed generating TPC-DS qualification queries.
s any key to continue
A comma separated list of queries can be specified in this option. The result of each query in the supplied list is generated in TPCDS_WORK_DIR
, with a default directory location of work
. The format of the result file is query<number>.res
.
A summary file named run_summary.txt
is also generated. It contains information about query number, execution time and number of rows returned.
Note: The query number is a two digit number, so for query 1 the results will be in query01.res
.
Note: If you are debugging and running queries using this option, make sure to save run_summary.txt
after each of your runs.
==========================================
DS On Spark Menu
------------------------------------------
P
Compile TPC-DS toolkit
Generate TPC-DS data with 1GB scale
Create spark tables
Generate TPC-DS queries
Run a subset of TPC-DS queries
Run All (99) TPC-DS Queries
NUP
Cleanup toolkit
Quit
------------------------------------------
se enter your choice followed by [ENTER]: 5
------------------------------------------
r a comma separated list of queries to run (ex: 1, 2), followed by [ENTER]:
: Checking pre-reqs for running TPC-DS queries. May take a few seconds..
: Checking pre-reqs for running TPC-DS queries is successful.
: Running TPCDS queries. Will take a few minutes depending upon the number of queries specified..
: Progress : [########################################] 100%
: TPCDS queries ran successfully. Below are the result details
: Individual result files: spark-tpc-ds-performance-test/work/query<number>.res
: Summary file: spark-tpc-ds-performance-test/work/run_summary.txt
s any key to continue
The only difference between this and option (5)
is that all 99 TPC-DS queries are run instead of a subset.
Note: If you are running this on your laptop, it can take a few hours to run all 99 TPC-DS queries.
==========================================
DS On Spark Menu
------------------------------------------
P
Compile TPC-DS toolkit
Generate TPC-DS data with 1GB scale
Create spark tables
Generate TPC-DS queries
Run a subset of TPC-DS queries
Run All (99) TPC-DS Queries
NUP
Cleanup toolkit
Quit
------------------------------------------
se enter your choice followed by [ENTER]: 6
------------------------------------------
: Checking pre-reqs for running TPC-DS queries. May take a few seconds..
: Checking pre-reqs for running TPC-DS queries is successful.
: Running TPCDS queries. Will take a few minutes depending upon the number of queries specified..
: Progress : [########################################] 100%
: TPCDS queries ran successfully. Below are the result details
: Individual result files: spark-tpc-ds-performance-test/work/query<number>.res
: Summary file: spark-tpc-ds-performance-test/work/run_summary.txt
s any key to continue
This will clean up all of the files generated during option steps 1, 2, 3, and 4. If you use this option, make sure to run the setup steps ( 1, 2, 3, 4) before running queries using option 5 and 6.
This will exit the script.
Sign up for IBM's Data Science Experience. By signing up for the Data Science Experience, two services: DSX-Spark
and DSX-ObjectStore
will be created in your IBM Cloud account. If these services do not exist, or if you are already using them for some other application, you will need to create new instances.
To create these services:
DSX-Spark
.DSX-ObjectStorage
.Note: When creating your Object Storage service, select the
Swift
storage type in order to avoid having to pay an upgrade fee.
Take note of your service names as you will need to select them in the following steps.
First you must create a new Project:
Get Started
tab at the top or scroll down to Recently updated projects
.New project
under Recently updated projects
.Name
and optional Description
.Spark Service
, select your Apache Spark service name.Storage Type
, select the Object Storage (Swift API)
option.Target Object Storage Instance
, select your Object Storage service name.Create
.Create the Notebook:
add notebooks
.From URL
and enter a Name
and optional Description
.Notebook URL
enter: https://github.com/IBM/spark-tpc-ds-performance-test/blob/master/notebooks/run-tpcds-on-spark.ipynbSpark Service
, select your Apache Spark service name.Create Notebook
.When a notebook is executed, what is actually happening is that each code cell in the notebook is executed, in order, from top to bottom.
Each code cell is selectable and is preceded by a tag in the left margin. The tag
format is In [x]:
. Depending on the state of the notebook, the x
can be:
*
, this indicates that the cell is currently executing.There are several ways to execute the code cells in your notebook:
Play
button in the toolbar.Cell
menu bar, there are several options available. For example, you
can Run All
cells in your notebook, or you can Run All Below
, that will
start executing from the first cell under the currently selected cell, and then
continue executing all cells that follow.Schedule
button located in the top right section of your notebook
panel. Here you can schedule your notebook to be executed once at some future
time, or repeatedly at your specified interval.Under the File
menu, there are several ways to save your notebook:
Save
will simply save the current state of your notebook, without any version
information.Save Version
will save your current state of your notebook with a version tag
that contains a date and time stamp. Up to 10 versions of your notebook can be
saved, each one retrievable by selecting the Revert To Version
menu item.You can share your notebook by selecting the ?Share? button located in the top right section of your notebook panel. The end result of this action will be a URL link that will display a ?read-only? version of your notebook. You have several options to specify exactly what you want shared from your notebook:
Only text and output
: will remove all code cells from the notebook view.All content excluding sensitive code cells
: will remove any code cells
that contain a sensitive tag. For example, # @hidden_cell
is used to protect
your dashDB credentials from being shared.All content, including code
: displays the notebook as is.download as
options are also available in the menu.This Code Pattern walks us through the steps that need to be performed to run the TPC-DS benchmark with the qualification scale factor(1GB). Since this is a performance benchmark, typically we need to run the benchmark with varying scale factors to gauge the throughput of the underlying data processing engine. In the section below, we will briefly touch on things to be considered while increasing the data and running the workload against a production cluster.