FormidableLabs/lank

Name: lank

Owner: Formidable

Description: Link and control a bunch of repositories.

Created: 2017-03-15 20:38:37.0

Updated: 2017-04-01 01:56:04.0

Pushed: 2017-04-27 17:35:10.0

Homepage: null

Size: 73

Language: JavaScript

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

lank

Working on multiple repositories that depend on each other can be a real pain. npm link-ing often has limitations. Watching and copying files from one project to another has lag and is often wonky.

Lank enters this space with a simple proposition: do nothing.

More specifically:

1. Take a nested, interrelated dependency tree and delete any projects that you would like to simultaneously edit from node_modules in each project.
2. Add NODE_PATH enhancements to allow Node.js require resolution to resolve to the actual checked out projects instead of installed node_modules dependencies.

Installation

Install lank globally:

m install -g lank
rn global add lank

Configuration

The first basic step with lank is “linking” several projects together. This really means:

1. lank has some means of knowing which projects are connected – accomplished by a .lankrc.js file.
2. lank then provides helper tasks to delete linked projects from withing each project's node_modules dependency tree.
3. Things are now set up file-wise. The final aspect is that lank provides execution helpers to set up NODE_PATH for proper inter-project resolution while invoking (1) multi / “all project” tasks and (2) central “project under development” tasks. This last step is optional as a user can just mutate NODE_PATH on their own and run manual steps.

Configuration File

Let's start with the first step - configuring project linking. To link projects, you needs to create a .lankrc.js file. For an example set up of:

ECTS/
e/
o/
ree/

Say that you wish to run tasks from PROJECTS/one. You can create a configuration file in either the controlling project directory, or one level above, so:

If you would like to link the projects and run tasks from PROJECTS, then you should create a:

ECTS/
e/.lankrc.js

or

ECTS/.lankrc.js

On initialization, lank will look for a .lankrc.js in the current working directory. If there is none, then lank will secondarily look up one directory if it finds a .lankrc.js file. In either case, the project directories for all linked files will be checked for presence, otherwise lank will throw a configuration error.

Simple Version

The .lankrc.js file is simply an array of strings where each string corresponds to the published package.json:name of a package and is the name of a directory at the same level as all other project directories controlled by lank. For example:

.lankrc.json`

ne",
wo",
hree"


.lankrc.js`
le.exports = [
ne",
wo",
hree"

Advanced Version

The .lankrc.js file can also be an object supporting the following fields:

• module: The string name of the module at issue.
• tags: A string for single tag or array of strings for multiple tags. Tags are used to filter multi-project commands.

lank supports two formats for advanced configuration objects - a longhand one:

.lankrc.js`
le.exports = [
module: "one" },
module: "two", tags: "foo" },           // tags can be strings ...
module: "three", tags: ["foo", "bar"] } // ... or an array of strings

and a shorthand object form:

.lankrc.js`
le.exports = {
e: {},
o: { tags: "foo" },
ree: { tags: ["foo", "bar"] }

Scoped Modules

If your project has a scoped published name like @org/foo, then your directory structure on disk must reflect that as well. So our example above may look something instead like:

ECTS/
rg/
one/
two/
three/

with a .lankrc.js file of:

le.exports = [
org/one",
org/two",
org/three"

Unfortunately, the directory path of the actual module name is required, @ symbol and all.

Also note that for .lankrc.js resolution, the rules would be something like:

 we're in `one`, look in CWD first.
ECTS/@org/one/.lankrc.js

 not, look **two** levels down since we're in a scoped project.
ECTS/.lankrc.js

This scheme allows you to link both scoped and non-scoped projects in the same directory structure.

Running a Project

Workflows with lank circle around a root “controlling” project that you will change directory into and run:

 /PATH/TO/PROJECTS/one
nk <commands>

lank will base working directory paths for all linked projects off this assumption, and it will error out if it does not find a directory structure that matches the configuration files.

Side Note – What is a “controlling” project?: We use lank to control 2+ interrelated projects that need simultaneous changes. The controlling project is usually the “most upstream” one that is not depended on by any other project. For example, this would likely be your application that then “controls” many linked library projects. Ultimately, it doesn't really matter that much as long as the lank commands are executed from within the root project directory of any linked project.

As we learned above, that project must have a .lankrc file in one of:

H/TO/PROJECTS/one/.lankrc.js
H/TO/PROJECTS/.lankrc.js

So now, turning to the projects at hand, the core issue is cross-project dependencies. In our example, they might look like this:

de_modules/
two
  node_modules/
    three           # this dependency may be flattened to higher level
three

de_modules/
three

So, in this case three has no dependencies on the other projects, two depends on three, and one (our control project) depends on at least two and three and possibly a second three as a transitive dependency of two. Yowza.

Linking Projects

So how do we “link” these projects together such that we can make a change in three and see it reflected live in one and two? There are a couple of approaches in the current ecosystem:

1. npm link the changed projects to each other.
2. Manually watch and recursively copy files from one project to another.
3. Manually npm pack and npm install packed tarballs on each change.

All of these approaches have drawbacks of some sort – either relying on symlinks, file watching / delayed copying, and/or manual steps. This adds up to incorrect behavior or irksome developer workflows when changing things across multiple project simultaneously.

lank takes a very different approach to this problem:

1. Find and delete all of the npm install-ed cross-project dependencies from each project's node_modules
2. Provide an execution wrapper that enhances the process environment such that the Node dependencies resolve to the “linked” live projects.

With this scheme in place, changes across projects are instantaneous because the real code in the linked project is used at the real path – no symlinks or file copying!

With that long introduction in place, we introduce lank link. What this command basically does is find and delete all of the cross-project dependencies. So, in our example, we first do a dry run to show what we would delete without actually deleting:

nk link --dry-run
k:link]              Found 4 directories to delete:
k:link]              - /PATH/TO/PROJECTS/one/node_modules/two
k:link]              - /PATH/TO/PROJECTS/one/node_modules/two/node_modules/three
k:link]              - /PATH/TO/PROJECTS/one/node_modules/three
k:link]              - /PATH/TO/PROJECTS/two/node_modules/three

lank has traversed the dependency trees in all linked projects and found the cross-dependencies. We then run lank link to actually perform the deletion.

If you want to undo the linking, simply reinstall all projects' dependencies with:

ncurrent yarn installs are finicky, so install one at a time in serial.
so "force" the install as manually deleted packages from `lank` won't be
cessarily reinstalled on a vanilla `yarn install` alone.
nk exec -s -- yarn install --force
. OR ...
nk exec -s -- npm install

Shell Commands

Once you have lank link-ed a project, all projects effectively have “holes” for cross-dependencies. We can use this to have the linked projects resolve to each other in source by running any commands in one project with an environment variables NODE_PATH that includes the value .. which means “look one directory below CWD to find additional dependencies”. (See our section on Node require resolution for a further explanation of this.)

To help with this environment enhancement and for similar multi-repository workflows, lank provides the exec command, which runs the same command in all linked projects:

nk exec -- SHELL_COMMAND

Here are some basic examples:

int CWD
nk exec -- pwd

t status
nk exec -- git status

stall deps
nk exec -s -- yarn install --force
nk exec -s -- npm install

Sometimes, you only want to exec a command in some projects. This is where the -t, --tags <tags> flag commes in handy to run based on arbitrary tags and the -m, --modules <modules> flag is useful to limit to a list of named projects.

ec in projects configured with a (1) "foo" tag, (2) a "foo" or "bar" tag.
nk exec -t foo -- pwd
nk exec -t foo,bar -- pwd

ec in specifically named projects
nk exec -m one -- pwd
nk exec -m one,two -- pwd

Output Buffering

By default, lank buffers all output and displays it once the underyling processes end. This is nice for processes that do end since you don't get random process output crossing streams in your terminal during execution. But, this scheme doesn't really work well for exec's that are meant to be long-lived or persistent processes, such as a file build watch.

In these cases, use the -u / --unbuffered flag to just have output splattered to stdout/stderr as it happens, with some helpful prefixes to indicate which project the output came from. For example:

nk exec -u -- npm run watch-files

Keeping Package Dependencies in Sync

Multiple repositories generally ends up with dependency skews across projects. lank provides a very convenient manner of harmonizing dependencies across all linked projects with:

nk deps -d  # Check with a dry run first.
nk deps

lank uses a simplistic algorithm of:

1. Only looking at deps that are of the forms 1.2.3, ~1.2.3, and ^1.2.3
2. If 2+ different versions exist in a package.json, lank chooses the latest or highest version string to win.

lank then writes out updates to actual project package.json files where applicable.

For a usual workflow, you'll want to update deps in linked projects, re-install dependencies (and new ones), then re-link the projects. Something like:

date
nk deps

install
nk exec -s -- yarn install --force
nk exec -s -- npm install

-link
nk link

Notes, Tips, and Tricks

Miscellaneous

• The name of the Node modules must correspond to the directory name of the project on disk. For example, if you are linking the foo project normally found in node_modules/foo, it now must be named foo on the local file system relative to the directories that lank controls.

Node require Resolution

lank depends on the actual details of Node's require resolution, which is a bit complicated and described in full detail at: https://nodejs.org/api/modules.html

Here's an abbreviated example for us:

o/index.js
le.exports = require("three");

s
e/index.js
t two = require("two");

"the transitive THREE command!");

The resolution of three from two has a large set of paths to traverse until it finds a match.

rst, try `${dirname}/node_modules` and up file system
e/user/projects/one/node_modules/two/node_modules/three
e/user/projects/one/node_modules/three
e/user/projects/node_modules/three
e/user/node_modules/three
e/node_modules/three
e_modules/three

w, look to `NODE_PATH`
DE_PATH}/three

nally, the global installs
ME}/.node_modules/three
ME}/.node_libraries/three
EFIX}/lib/node/three

What lank does in link-ing is to just delete the normal projects from node_modules and the in exec and related commands enhance the NODE_PATH variable so that lookup up until NODE_PATH fails to find the cross-referenced project. That then leaves NODE_PATH to get the live link-ed project instead and presto! we have first class Node require integration with our custom projects instead of what's installed via yarn|npm install.

TODO(INITIAL) Additional Sections

• TODO(INITIAL): Document - webpack
• TODO(INITIAL): eslint