Name: FEL
Owner: Yahoo Inc.
Description: Fast Entity Linker Toolkit for training models to link entities to KnowledgeBase (Wikipedia) in documents and queries.
Created: 2016-04-19 23:37:17.0
Updated: 2018-01-09 08:53:22.0
Pushed: 2017-07-05 01:48:18.0
Size: 300
Language: Java
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Fast Entity Linker Core
This library performs query segmentation and entity linking to a target reference Knowledge Base (i.e., Wikipedia). In its current version it is tailored +towards query entity linking (alternatively, short fragments of text). The main goal was to have an extremely fast linker (< 1 or 2 ms/query on average on a standard laptop) that is completely unsupervised, so that more sophisticated approaches can work on top of it with a decent time budget available. A side effect of this is that the datapack used by the linker occupies <3GB making it suitable to run on the grid (and making the footprint on server machines very low).
Please install maven before you run this project. The project comes with a pom.xml which should install all dependencies when you run the command mvn install.
The library performs query and document entity linking. It implements different algorithms that return a confidence score (~log likelihood) that should be (more or less) comparable across pieces of text of different length so one can use a global threshold for linking. The program operates with two datastructures, one big hash and compressed word and entity vectors. The hash is generated out of a datapack that records counts of phrases and entities that co-occur together. These counts might come from different sources, for instance anchor text and query logs. In anchor text, whenever there is a link to a corresponding entity page we would store the anchor and the entity counts. In a query log whenever there is a click to an entity page, we would update the query and entity counts. The word and entity vector files are compressed vector representations that account for the contexts in which the word/entity appears. The library provides a way to learn the entity vectors. Word vectors can be generated using general tools like word2vec, or you can reuse pre-trained word vectors such as those available in Facebook's fastText project.
The library also comes with two different sets of tools for generating the hash and the word vector files.
If you use this library, please cite following papers:
roceedings{Blanco:WSDM2015,
dress = {New York, NY, USA},
thor = {Blanco, Roi and Ottaviano, Giuseppe and Meij, Edgar},
oktitle = {Proceedings of the Eight ACM International Conference on Web Search and Data Mining},
cation = {Shanghai, China},
mpages = {10},
blisher = {ACM},
ries = {WSDM 15},
tle = {Fast and Space-Efficient Entity Linking in Queries},
ar = {2015}
roceedings{Pappu:WSDM2017,
dress = {New York, NY, USA},
thor = {Pappu, Aasish, and Blanco, Roi, and Mehdad, Yashar and Stent, Amanda, and Thadani, Kapil},
oktitle = {Proceedings of the Tenth ACM International Conference on Web Search and Data Mining},
cation = {Cambridge, UK},
mpages = {10},
blisher = {ACM},
ries = {WSDM 17},
tle = {Lightweight Multilingual Entity Extraction and Linking},
ar = {2017}
There are a number of different rankers/linkers that use different conceptual models. The overall description of the algorithm with some implementation details is at:
Fast and space efficient entity linking for queries
The main class to use is
com.yahoo.semsearch.fastlinking.FastEntityLinker
The class can be called with --help to list the available options.
They provide interactive linking through stdin (edit the code or extend the class if you need a custom output format).
First download the dataset from webscope following the links provided below.
Example usage call:
exec:java -Dexec.mainClass=com.yahoo.semsearch.fastlinking.FastEntityLinker \
-Dexec.args="en/english-nov15.hash"
The CoherentEntityLinker class takes entity-mentions and n-best list of entity-links for each entity mention as input. It constructs a lattice from the n-best lists and runs Forward-Backward algorithm.
More coherency algorithms are under experimentation. They will be added in future versions of the code.
clean compile exec:java \
exec.mainClass=com.yahoo.semsearch.fastlinking.CoherentEntityLinkerWrapper \
exec.args="en/enwiki.wiki2vec.d300.compressed en/english-nov15.hash test.txt" \
exec.classpathScope=compile
You can include a mapping file in the entity linker arguments (below) that maps integral entity categories to human-readable entity categories.
The following command would run the linker on a Hadoop grid:
op jar FEL-0.1.0-fat.jar \
yahoo.semsearch.fastlinking.utils.RunFELOntheGrid \
pred.map.tasks=100 \
preduce.map.java.opts=-Xmx3g \
preduce.map.memory.mb=3072 \
pred.job.queue.name=adhoc \
es en/english-nov15.hash#hash, src/main/bash/id-type.tsv#mapping \
utfile> \
putfile>
The class reads files that have one query per line - it splits on
ty_type <TAB> query <TAB> modifier <TAB> entity_id
where
entity_type is given in the datapackquery is the original querymodifier is the query string when the entity alias is removeentity_id is the retrieved entityIn general you should rely on thresholding and possibly sticking to the top-1 entity retrieved but this depends on how you are going to use it.
This package also provides code to
More on this can be found in the w2v package.
The tool makes use of a datapack that stores counts and aliases (mentions) of entities from different sources. Originally, we used anchor text and query logs. The following describes how to mine and compute the anchor text from a public Wikipedia dump using a hadoop cluster (or if there is not one, you can use hadoop in a single machine). This is based on the code from the Cloud9 toolkit.
More on this can be found in the io package.
The datapack will contain two files: one with the per-entity counts and one with the entity to id mapping. Then, you can hash it using:
yahoo.semsearch.fastlinking.hash.QuasiSuccinctEntityHash \
<datapack_file> -e <entity2id_file> -o <output_file>
The following pre-trained models are provided to perform entity linking with the toolkit and are available through the Yahoo! webscope program for research purposes. These models are trained on Wikipedia and distributed using Creative Commons BY SA 4.0 license (see MODELS_LICENSE).