hammerlab/spark-util

Name: spark-util

Owner: Hammer Lab

Description: Low-level helpers for Spark libraries and tests

Created: 2016-11-13 16:29:40.0

Updated: 2017-11-30 18:39:07.0

Pushed: 2018-01-13 21:06:09.0

Homepage: null

Size: 65

Language: Scala

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

spark-util

Spark, Hadoop, and Kryo utilities

Kryo registration

Classes that implement the Registrar interface can use various shorthands for registering classes with Kryo.

Adapted from RegistrationTest:

ster(
s[A],                  // comes with an AlsoRegister that loops in other classes
r[Foo],                // register a class and an Array of that class
s[B] ? BSerializer(),  // use a custom Serializer
Registrar              // register all of another Registrar's registrations

• custom Serializers and AlsoRegisters are picked up implicitly if not provided explicitly.
• AlsoRegisters are recursive, allowing for much easier and more robust accountability about what is registered and why, and ensurance that needed registrations aren't overlooked.

Configuration/Context wrappers

• Configuration: serializable Hadoop-Configuration wrapper
• Context: SparkContext wrapper that is also a Hadoop Configuration, for unification of “global configuration access” patterns
• Conf: load a SparkConf with settings from file(s) specified in the SPARK_PROPERTIES_FILES environment variable

Spark Configuration

• SparkConfBase: trait that brokers setting config key-values and creating a SparkConf
• many mix-ins for common spark-configuration groups:
  • kryo registration
  • dynamic allocation
  • event-logging
  • task-speculation

Misc

• KeyPartitioner / Partitioner: shorthands for common Spark-Partitioner-creation patterns
  • from the first field of tuple-like objects
  • from a partial function
  • from a function
• Histogram accumulator