Name: emp
Owner: biocore
Description: Code repository of the Earth Microbiome Project.
Created: 2012-07-18 23:48:09.0
Updated: 2018-01-02 04:47:07.0
Pushed: 2017-12-20 02:42:46.0
Homepage: http://www.earthmicrobiome.org
Size: 1116429
Language: Jupyter Notebook
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
The Earth Microbiome Project (EMP) is a systematic attempt to characterize global microbial taxonomic and functional diversity for the benefit of the planet and humankind. Most of the data generated to this point are from 16S rRNA amplicon surveys, but the project also includes data from 18S and ITS amplicons, metagenomics, and metabolomics. For more information about the EMP – people, publications, news, protocols and standards, and more – please see the EMP website.
This GitHub repository describes the EMP catalogue and how to use it. The EMP dataset is generated from samples that individual researchers have compiled and contributed to the EMP. Samples from each group of researchers represent individual EMP studies. In addition to analyses being done by contributing researchers on the individual studies, we are performing cross-study meta-analyses. EMP 16S Release 1, a meta-analysis of the first 97 16S rRNA amplicon studies, has been published (article, preprint). The analysis code used for that manuscript is provided in this GitHub repository.
There are several ways to get involved with the EMP:
The EMP catalogue is a diverse and standardized set of thousands of microbiomes for use by the public. Here are some of the ways you can use this resource:
Download EMP Release 1 from our FTP site. EMP Release 1 contains merged and quality-filtered mapping files, BIOM tables, OTU/sequence information, and alpha/beta-diversity results for ~25,000 samples in 97 studies of the initial meta-analysis of the EMP. The FTP site contains README files about its contents, and the individual files are listed here.
Download individual studies from the Qiita EMP Portal. For each study, you can download metadata (mapping file), feature tables (BIOM file), and demultiplexed raw sequence files. Like the rest of Qiita, the EMP Portal requires the Google Chrome browser.
Merge your data with all or part of the EMP dataset. If you sequenced your sample using the EMP 16S rRNA primers and picked OTUs using either Deblur or closed-reference against Greengenes 13.8 or Silva 123, you can merge your BIOM table with the relevant merged EMP Release 1 BIOM table or one of the individual per-study BIOM tables from Qiita. Basic instructions for initial processing of your data are provided. You can then use QIIME1 or QIIME2 to merge the BIOM tables and mapping files.
Query the EMP catalogue using Redbiom. Redbiom is a command-line tool that allows users to query the Qiita database, including EMP studies. It allows you to find samples based on the sequences or taxa they contain or on sample metadata, and to export selected sample data and metadata. Once you have Redbiom installed, you can carry out queries such as those described here:
rst, summarize the contexts available. A context represents a partition by
ocessing parameters (e.g., closed-reference OTU picking) and preparation
.g., 16S V4).
iom summarize contexts | cut -f 1,2,3
eate a variable for the context. For this example, we will use the closed-
ference 16S V4 context by setting a local bash variable "ctx".
Pick_closed-reference_OTUs-illumina-16S-v4-66f541
ery 1: "Show me all the genera that were observed at pH > 8."
rst we search for samples with pH > 8, then select the features from those
mples, then summarize the taxonomy of those features, then grep for just
e genera and count them.
iom search metadata "where ph > 8" | redbiom select features-from-samples \
ntext $ctx | redbiom summarize taxonomy --context $ctx | grep g__ | wc -l
swer: There are 1423 genera found in samples with pH > 8.
ery 2: "Show me all sites where Pyrobaculum are found."
rst we search for features that are genus Pyrobaculum, then search for
mples containing those features, then fetch sample metadata for those
mples and output the metadata file, then grab the columns for latitude and
ngitude (note: these are not guaranteed to reside in columns 10 and 11).
iom search taxon --context $ctx g__Pyrobaculum | redbiom search features \
ntext $ctx | redbiom fetch sample-metadata --context $ctx \
tput g__Pyrobaculum_metadata.txt; cut g__Pyrobaculum_metadata.txt -f 10,11
This repository contains the following directories:
code IPython notebooks and scripts (Python, Java, R, Bash) developed for meta-analysis of EMP data; this code is used in the top-level directory methods.data Data files used for processing and downstream analysis.methods Instructions to generate the figures in the EMP 16S Release 1 meta-analysis (in press).legacy Early code, results, and website documents from the initial phase of the EMP (2010-2013).presentations Collection of presentations on the EMP.Some abbreviations used in this repository:
demux is shorthand for “demultiplexed”, which describes the fastq data after it is split into per-sample fastq files using barcodes.deblur refers to the exact-sequence de novo OTU picking method Deblur.cr refers to closed-reference OTU picking.or refers to open-reference OTU picking.refseqs refers to reference sequence collections that could be used in reference-based OTU picking.mc2 refers to minimum sequence count in an OTU to be included equals to 2.If you're looking for data generated and used for the ISME 14 EMP presentations, look here.