Name: mojoland
Owner: H2O.ai
Description: Testing ground to ensure mojos backward compatibility
Created: 2016-11-05 16:14:33.0
Updated: 2016-11-22 02:19:41.0
Pushed: 2017-03-29 18:57:40.0
Homepage: null
Size: 13230
Language: Java
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
This is the testing ground to ensure mojos backward compatibility. The goal of this project is to collect pre-trained mojo models (for different mojo versions and various datasets) and their expected scoring results. Then we will routinely test that for each new version of the genmodel runtime all saved mojo models still produce the same results as originally. This will enable us to have full backward compatibility of mojos: old models are guaranteed to be invariant with respect to the updates of the runtime.
When speaking about backward compatibility, it is essential to understand which things are, and which are not expected to be stable over time.
Data -> H2O Model (training) : not stable over time. Given same data and same input parameters (including the seed), it is entirely possible that future versions of H2O may build a slightly different model. Hopefully, a better one.
H2O Model -> Mojo : not stable over time. For the same model we may produce a different mojo in a future version of H2O. This will always be accompanied by mojo version increase.
Mojo -> MojoModel -> results : stable. This is the contract that is being tested here, so it deserves a thorough explanation. A mojo is a binary file, which cannot be run by itself. It requires the mojo-runtime.jar for it to run (in the future we will be adding runtimes for other languages as well). This runtime instantiates a MojoModel object from the mojo file. Public API of this object is what is being tested.
The stability property guarantees that given the same input data, the MojoModel will produce identical results even when instantiated in runtimes of future versions. More specifically, the mojo is guaranteed to produce identical results with any runtime which is current or newer than was at the time of mojo's creation. At the same time instantiating a mojo against an older version of the runtime should produce an error.
MojoModels' public API : not stable. This means that mojos of different versions are allowed to have different APIs. Such changes should be infrequent. However if API changes do occur, they do not apply retroactively: new API functions should not be made available for old mojos.
In practice we will maintain the following hierarchy: all mojos are instances of the base MojoModel class. This
class only has factory methods for instantiating mojos (public static MojoModel load(? source)). The MojoModel
class is subclassed by MojoModel1 (or MojoModel2, etc) which is further subclassed by GbmMojoModel1,
DrfMojoModel2, etc. The public interface of each <Algo>MojoModel<N> never changes. However different versions
of mojos may produce instances of <Algo>MojoModel<N> with different Ns.
It is expected that concrete versions of the algo-specific interfaces may differ from the versions of MojoModel.
For example, we may have DrfMojoModel3 extending MojoModel1.
Each mojo file carries an internal version, which is a decimal number for example 1.00 or 2.03. Mojos for different
algorithms can have different versions. The minor version (after the decimal dot) should be a 2-digit number, to ensure
proper sorting both numerically and lexicographically. The major version must coincide with the version of the
MojoModel being created.
Thus, mojo's major version is bumped whenever the MojoModels interface needs to be changed, and the minor version is
increased when any internal mojo change occurs.
To summarize:
<Algo>MojoModel<N>s API (i.e. addition or removal of methods, change in methods' parameters or return
types) – bump mojo's version to (N+1).00 and create new <Algo>MojoModel<N+1>. Note that any API change,
including adding new functions, warrants creating new API class.<Algo>MojoModel<N>s private/protected API – no need to change mojo's version.At the top level of the runtime's hierarchy is the MojoModel class, which has only minimal amount of functionality
(few static functions). This class is extended by MojoModel1, MojoModel2, etc – each class providing bare-bones
functionality for their descendants. Each MojoModel{N} class should have only protected or private methods. These
classes can then be extended by {Algo}BaseModel{N} classes carrying algo-specific functionality shared by the apis
of different versions. Again, these classes must not expose any public methods. Finally, at the last level of the
hierarchy live classes {Algo}MojoModel{N}. These are the concrete classes that get instantiated when a mojo is loaded,
and they define the API of each mojo: a mojo's API is the set of declared public methods of the corresponding
{Algo}MojoModel{N} class.
Separate from this, there is a collection of (versioned) interfaces, each covering some part of the functionality of
each {Algo}MojoModel{N}. These interfaces are expected to be more stable over time than the concrete classes, and
they can extend each other (for example an interface of version n may extend an interface of version n - 1). The
interfaces don't have any predefined naming pattern.
If a mojo model class uses an externally visible class or enum, it should be properly versioned too. For example,
consider the “GlrmLoss” enum. “Proper versioning” requires that the enum was called GlrmLoss1, anticipating that
there will be future versions GlrmLoss2, GlrmLoss3, etc. Each specific algo then specifies which version of the
enum it expects.
To build the project, please use the provided gradlew command:
./gradlew build
IntelliJ IDEA
Generate Idea configuration files via
./gradlew idea
Then open project's build.gradle in Idea via File > Open
Note: To clean up Idea project files use ./gradlew cleanIdea.
Eclipse
Generate Eclipse project files via
./gradlew eclipse
Then open project in Eclipse via File > Import > Existing Projects into Workspace
Note: To clean up Eclipse project files use ./gradlew cleanEclipse.