Name: npm-bazel
Owner: Redfin
Description: Bazel build rules and workspace generators for building node modules
Created: 2016-05-25 21:48:14.0
Updated: 2018-05-16 06:14:07.0
Pushed: 2016-10-11 04:51:44.0
Homepage: null
Size: 2831
Language: Python
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
This project provides a generator tool, npm-bazel-gen.py
, to generate WORKSPACE
rules and BUILD
files for npm modules. It also provides Skylark rules in build_tools/npm.bzl
to build npm
modules in Bazel's sandbox.
This is a public clone of code that we're using internally at Redfin. The version we actually use seems to be working, but we cannot guarantee that this repository will be stable/maintained. (But hope springs eternal, and if people seem to like it, we could stabilize it further.)
Generate the WORKSPACE
and BUILD
files using the Python script, then build the example projects.
on npm-bazel-gen.py
l build foo
npm install
kinda does, a little about how node_modules worknpm install
do?npm install
with no arguments does two-and-a-half things.
“Download” dependencies declared in package.json
into ./node_modules
(de-duping dependencies and resolving circular dependencies as needed)
“Rebuild” those downloaded dependencies. (You can npm rebuild
to re-run this.)
node_modules/.bin
For example, node_modules/.bin/gulp
will be a symlink to node_modules/gulp/bin/gulp.js
package.json
declares a scripts.prepublish
step, it'll run that. A common scripts.prepublish step is: “gulp prepublish”. (npm automatically adds node_modules/.bin
to the path when running this script.)To generate a .tgz
output file, we can run npm pack
, which does two things:
Some of these steps violate Bazel's sandbox.
node-gyp
, which tries to download header files on first launch and caches them in ~/.node-gyp
.nsp
to query for known Node security vulnerabilities. (nsp
has an offline mode, but it's “unofficial, unsupported.“)build_tools/npm_installer.sh
npm-bazel-gen.py
scans all package.json files, finding the full dependency tree (tree of trees? forest?) and declares all external dependencies as rules in the WORKSPACE
filebuild_tools/install-npm-dependencies.py
is a script that simulates what npm install
would have done, including de-duping dependencies and handling circular dependencies. (npm
allows circular dependencies! ?) BEWARE This script is currently imperfect! It doesn't do exactly what npm
would have done.npm rebuild
directly after we run install_npm_dependencies
.HOME=/tmp
and pre-install /tmp/.node-gyp
node_modules
folder.build_tools/npm_packer.sh
node_modules
folder into the work directorynpm pack
which runs the prepublish script, if any, and generates the final output.npm.bzl
npm.bzl
defines two rules, external_npm_module
and internal_npm_module
. “external” means “third party,” as opposed to “internal” modules being built by Bazel.
The internal_npm_module
rule is responsible for running npm_installer.sh
and then npm_packer.sh
as ctx.action
s.
The majority of npm.bzl
is a bunch of code to marshall the correct list of inputs and outputs for the “install” phase and then the “pack” phase. Both types of rules return a struct()
containing three fields:
internal
(boolean): external_npm_module
sets this to false, internal_npm_module sets this to truetransitive_external_deps
(listtransitive_internal_deps
(listexternal_npm_module
returns an empty list for this.In addition, internal modules return these two fields:
tarball
(file): The generated tarball file for this modulepackage_json
(file): The package.json
file input (used by other rules to compute the dependency tree)Skylark calls this system of a rule returning a struct()
a “provider” in Skylark. http://bazel.io/docs/skylark/rules.html (Cmd-F for the “Providers” section)
Skylark's documentation on providers is pretty light, but all it means is: rules can return a struct()
of data to provide it to dependent rules.
The top of npm.bzl
is taking the list of deps
and the list of dev_deps
, sorting them into internal and external dependencies, and assembling four lists:
transitive_internal_deps
transitive_external_deps
transitive_internal_dev_deps
transitive_external_dev_deps
It then creates an internal_modules.json
file using the two “internal” lists, so install_npm_dependencies.py
knows where to look for internal modules.
It then runs two ctx.action
commands:
build_tools/npm_installer.sh
to generate foo_node_modules.tar.gz
with these inputs:package.json
files for all internal dependenciesinternal_modules.json
build_tools/npm_packer.sh
to generate foo.tgz
with these inputs:In our project at Redfin, we have a bunch of projects that do weird/non-standard stuff, usually during the “pack” phase, but sometimes during the “install” phase.
The internal_npm_module
rule has install_tool
and pack_tool
attributes, which default to build_tools:npm_installer
and build_tools:npm_packer
but you can override them to anything you want, including defining a sh_binary
with arbitrary dependencies. (You have to hack in special cases to npm-bazel-gen.py
to make it add those for you, when you want it.)
In addition, the default packer tool looks for a ./extra-bazel-script
file in the current directory, and if it finds one, it just runs whatever it sees there. In some cases, that's enough pluggability.
@node
WORKSPACE differently on OS X and Linux. How do we make a single target that DTRT on other platforms?