Name: reaction-file-collections
Owner: Reaction Commerce
Description: Reaction File Collection packages
Created: 2018-02-09 15:52:13.0
Updated: 2018-04-26 07:21:53.0
Pushed: 2018-04-26 07:24:00.0
Homepage: null
Size: 318
Language: JavaScript
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Reaction FileCollections is a set of NPM packages that provide the ability to support file uploads, storage, and downloads in Node and Meteor apps, and in browser JavaScript. It is based on the no-longer-supported Meteor-CollectionFS project patterns, but greatly simplified and modernized.
Table of Contents generated with DocToc
Your app should depend on the @reactioncommerce/file-collections package:
i --save @reactioncommerce/file-collections
And then depend on a storage adapter package. Currently there is only GridFS:
i --save @reactioncommerce/file-collections-sa-gridfs
The overall steps are:
FileRecord.fromUrl.What follows is more detail for each of these steps. Check out server/main.js in the example Meteor app to see it all put together.
A store is where files are ultimately saved and retrieved from. In a simple scenario, you might have one store per FileCollection. Often, though, you need to process files and store multiple related copies. In this case, each copy goes into its own store, and the FileCollection record (the FileRecord) ties them all together. For example, you might transform all image uploads and save them to four different stores: large, med, small, thumbnail.
A store is created by making a new instance of a storage adapter class. For example, new GrisFSStore({ name: "primary" }) creates a new store backed by MongoDB GridFS and named “primary”.
You can mix and match storage adapters within the set of stores that you attach to a FileCollection. For example, you could store each file duplicated in GridFS and an S3 bucket. Or you could store image thumbnails in GridFS but store the full original image in an S3 bucket.
Each storage adapter package has its own constructor options unique to that package, but a few options are common to all storage adapters:
name: REQUIRED. Any name you want to use to identify the store. Keep in mind this appears in the URL for downloads from that store.fileKeyMaker: OPTIONAL. You should rarely need to use this. Provide a function that takes a FileRecord instance and returns an identifying key for saving a new copy of a file. This will override the storage adapter's default key-creation logic.transformWrite: OPTIONAL. This is where you can transform a file stream before saving. You will receive the FileRecord instance as the only argument, and you must return a ReadWrite stream that performs a transformation. Can optionally return a Promise.transformRead: OPTIONAL. This is where you can transform a file stream being read out of the store. You will receive the FileRecord instance as the only argument, and you must return a ReadWrite stream that performs a transformation. Can optionally return a Promise.In addition to the common options, GridFSStore requires that you pass in two dependencies: db, your MongoDB client, and mongodb, which is the default export from the mongodb NPM package.
Here is an example GridFS store that resizes an uploaded image if it's above a certain size, and converts it to JPEG format if it isn't already JPEG. It uses the sharp NPM package for transformation.
rt GridFSStore from "@reactioncommerce/file-collections-sa-gridfs";
rt mongodb from "mongodb";
rt sharp from "sharp";
t db = getDBSomehow(); // get your mongodb client from where you have it
r, if in a Meteor app, get db and mongodb this way:
t mongodb = MongoInternals.NpmModules.mongodb.module;
t { db } = MongoInternals.defaultRemoteCollectionDriver().mongo;
t imagesStore = new GridFSStore({
unkSize: 1 * 1024 * 1024, // You don't have to customize this, but you can
,
ngodb,
me: "images",
ansformWrite(fileRecord) {
// Need to update the content type and extension of the file info, too.
// The new size gets set correctly automatically.
fileRecord.type("image/jpeg", { store: "images" });
fileRecord.extension("jpg", { store: "images" });
// Resize keeping aspect ratio so that largest side is max 1600px, and convert to JPEG if necessary
return sharp().resize(1600, 1600).max().toFormat("jpg");
A TempFileStore instance defines where multi-part browser uploads will be stored temporarily while the upload is in progress. You can create multiple, but usually one is enough even if you have multiple FileCollections and/or multiple stores.
Example:
rt { TempFileStore } from "@reactioncommerce/file-collections";
t tempStore = new TempFileStore({
Optional. 20MB is the default
xFileSize: 1024 * 1024 * 2,
Optional. "uploads" is default. This is the OS file path relative to the
Node app's directory. It will be created if possible. Files are only stored
here temporarily during uploading from a browser.
FilePath: "my_upload_folder",
Required. Without this, all requests will receive Forbidden response.
Simply returning `true` will allow anyone to upload anything.
This is an example of checking the file type, which is available to you in
a req.uploadMetadata object (as is size). You could also examine the request
headers for an authorization token.
ouldAllowRequest(req) {
const { type } = req.uploadMetadata;
if (typeof type !== "string" || !type.startsWith("image/")) {
console.info(`shouldAllowRequest received request to upload file of type "${type}" and denied it`); // eslint-disable-line no-console
return false;
}
return true;
Then define a route for it using Express/Connect. In a Meteor app:
rt { WebApp } from "meteor/webapp";
pp.connectHandlers.use("/uploads", tempStore.connectHandler);
Uploads are handled by the tus library.
A FileCollection is where original files are tracked and mapped to their copies that exist in one or more stores. Most FileCollection methods accept and/or return FileRecord instances. A FileRecord is one document within a FileCollection, representing one original file, but potentially multiple copies of it in your linked stores.
The FileCollection class itself is generic and must be subclassed to define where the file document is actually stored. There are currently two included subclasses: MeteorFileCollection and MongoFileCollection.
A MongoFileCollection is a FileCollection that stores the documents in MongoDB.
Here's an example:
rt { MongoClient } from "mongodb";
rt { MongoFileCollection } from "@reactioncommerce/file-collections";
rt getRandomId from "./getRandomId";
t { DATABASE_NAME, DATABASE_URL } = process.env;
oClient.connect(DATABASE_URL, (error, client) => {
(error) throw error;
nsole.info("Connected to MongoDB");
= client.db(DATABASE_NAME);
nst Images = new MongoFileCollection("Images", {
// Just a normal Mongo `Collection` instance. Name it however you like.
collection: db.collection("ImagesFileCollection"),
// add more security here if the files should not be public
allowGet: () => true,
makeNewStringID: () => getRandomId(),
// See previous sections in the documentation for the definitions of imagesStore and tempStore
stores: [imagesStore],
tempStore
;
Do something with Images
A MeteorFileCollection is a FileCollection that stores the documents in MongoDB using Meteor's Mongo.Collection API. While you could just use MongoFileCollection in a Meteor app, MeteorFileCollection will set up DDP methods and security automatically, and exports a client API that uses them. This means you can insert, update, and remove file records from browser code. You will also get the built-in Tracker reactivity when you find in browser code.
Here's an example:
rt { Meteor } from "meteor/meteor";
rt { Mongo } from "meteor/mongo";
rt { check } from "meteor/check";
rt { MeteorFileCollection } from "@reactioncommerce/file-collections";
t Images = new MeteorFileCollection("Images", {
Optional, but you should pass it in if you use the audit-argument-checks Meteor package
eck,
Just a normal Meteor `Mongo.Collection` instance. Name it however you like.
llection: new Mongo.Collection("ImagesFileCollection"),
Usually you'll pass `Meteor` as the `DDP` option, but you could also
pass a different DDP connection. Methods will be set up on this connection
for browser instances of MeteorFileCollection to call.
P: Meteor,
add more security depending on who should be able to manipulate the file records
lowInsert: () => true,
lowUpdate: () => true,
lowRemove: () => true,
add more security here if the files should not be public
lowGet: () => true,
See previous sections in the documentation for the definitions of imagesStore and tempStore
ores: [imagesStore],
mpStore
This is straightforward. Pass in an array of all FileCollection instances that should be served by a single endpoint. You can optionally pass headers to set on all responses.
rt { FileDownloadManager } from "@reactioncommerce/file-collections";
t downloadManager = new FileDownloadManager({
llections: [Images],
aders: {
get: {
"Cache-Control": "public, max-age=31536000"
}
Then define a route for it using Express/Connect. In a Meteor app:
rt { WebApp } from "meteor/webapp";
pp.connectHandlers.use("/files", downloadManager.connectHandler);
This means that all files in the Images file collection will have URLs that begin with /files. The full URL will be /files/:collectionName/:fileId/:storeName/:filename.
A TempFileStoreWorker instance observes one or more MeteorFileCollections and triggers final storage after an upload from a browser is complete and all the chunks have been assembled into a single file in the temp store.
All provided FileCollections must be MeteorFileCollections rather than MongoFileCollections because Meteor's observe API is used. A variation of this could be created to use polling or some other method for detecting waiting temporary files.
Example:
rt { TempFileStoreWorker } from "@reactioncommerce/file-collections";
t worker = new TempFileStoreWorker({ fileCollections: [Images] });
er.start();
Keep in mind that a worker can run in a separate service from your main app, and you may wish to do it this way for scalability. However, there is not yet support for running multiple workers. (They will all try to work the same file record.) This could be solved easily enough if you are interested in submitting a pull request to add record locking.
Also, a TempFileStoreWorker must be running on the same machine (or container) where the related TempFileStore is running.
A RemoteUrlWorker instance observes one or more MeteorFileCollections and triggers final storage streaming from a remote URL, after you've inserted a FileRecord that you created using FileRecord.fromUrl. You'll need to pass in a reference to a fetch implementation.
All provided FileCollections must be MeteorFileCollections rather than MongoFileCollections because Meteor's observe API is used. A variation of this could be created to use polling or some other method for detecting waiting remote URL inserts.
Example:
rt fetch from "node-fetch";
rt { RemoteUrlWorker } from "@reactioncommerce/file-collections";
t worker = new RemoteUrlWorker({ fetch, fileCollections: [Images] });
er.start();
Keep in mind that a worker can run in a separate service from your main app, and you may wish to do it this way for scalability. However, there is not yet support for running multiple workers. (They will all try to work the same file record.) This could be solved easily enough if you are interested in submitting a pull request to add record locking.
There is much less setup necessary in browser code. First define the same FileCollection that you defined in server code:
rt { Meteor } from "meteor/meteor";
rt { Mongo } from "meteor/mongo";
rt { MeteorFileCollection } from "@reactioncommerce/file-collections";
t Images = new MeteorFileCollection("Images", {
The backing Meteor Mongo collection, which you must make sure is published to clients as necessary
llection: new Mongo.Collection("ImagesFileCollection"),
Usually you'll pass `Meteor` as the `DDP` option, but you could also
pass a different DDP connection. Methods will be called on this connection
if you perform inserts, updates, or removes in browser code
P: Meteor
Then set the default upload and download URLs:
rt { FileRecord } from "@reactioncommerce/file-collections";
hese need to be set in only one client-side file
Record.absoluteUrlPrefix = "https://my.app.com";
Record.downloadEndpointPrefix = "/files";
Record.uploadEndpoint = "/uploads";
These must match the Express/Connect routes you defined on the server. Set these just once, early in your browser code, before you call fileRecord.upload() or fileRecord.url() anywhere.
All the setup is done. Time for the easy part. Let the user pick a file, upload it, and then save the file record.
tion uploadAndInsertBrowserFile(file) {
Convert it to a FileRecord
nst fileRecord = FileRecord.fromFile(file);
Listen for upload progress events if desired
leRecord.on("uploadProgress", (info) => {
console.info(info);
;
Do the upload. chunkSize is optional and defaults to 5MB
leRecord.upload({ chunkSize: 1024 * 1024 })
.then((id) => {
console.log(`Temp ID is ${id}`);
// We insert only AFTER the server has confirmed that all chunks were uploaded
return Images.insert(fileRecord);
})
.then(() => {
console.log("FileRecord saved to database");
})
.catch((error) => {
console.error(error);
});
xample event handler for a type=file HTML input
tion handler(event) {
nst [file] = event.target.files;
loadAndInsertBrowserFile(file);
Uploading from Node should also work (for example a Blob), but this hasn't been tested. There is a FileRecord.fromBlob static method.
Either in a browser or in Node code, you can also store and insert a file record from a remote URL.
Make sure you have a file worker set up somewhere to do the actual download, transformation, and storage.
rt fetch from "node-fetch";
rt { FileRecord } from "@reactioncommerce/file-collections";
rt Images from "./Images"; // wherever you defined your FileCollection
c function insertRemoteFile(url) {
nst fileRecord = await FileRecord.fromUrl(url, { fetch });
turn Images.insert(fileRecord);
Each storage adapter should be its own NPM package. If you create one, you can submit a pull request to add it in the packages folder of this repo, or you can store it in your own repo and publish it yourself.
To make a storage adapter package:
@reactioncommerce/file-collections-sa-base package.StorageAdapter from that package.super: fileKeyMaker, name, transformRead, transformWrite_fileKeyMaker, _getReadStream, _getWriteStream, _removeFilethis.typeName to a string that identifies your type of storage adapter. This will be the storageAdapter property on each copy in a FileRecord, which is currently just informational.Refer to the GridFSStore class definition as a model.
When dealing with streams, trying to debug with inspect and breakpoints isn't always possible. All of the file-collections packages log plenty of information for debugging in the reaction-file-collections namespace. Run your Node or Meteor app with DEBUG=reaction-file-collections or DEBUG=reaction* to see it.
This NPM package is published automatically on every push to the master branch. Be sure to use proper Git commit messages so that the version will be bumped properly and release notes can be automatically generated.
[skip release] in your commit message.Install Meteor, and then run the following commands:
install
run bootstrap
xample-apps/meteor-blaze-app
or npm install
start
npm run bootstrap in project root.install
run bootstrap
test
CircleCI should publish all package changes to NPM automatically. If you need to publish manually and you have write access to the NPM packages, do so by running npx lerna publish --conventional-commits. It will determine appropriate version bumps automatically.