Name: rtika
Owner: rOpenSci
Description: R Interface to Apache Tika
Created: 2018-01-19 04:02:08.0
Updated: 2018-03-16 20:12:40.0
Pushed: 2018-03-12 22:22:42.0
Homepage: https://ropensci.github.io/rtika/
Size: 139359
Language: R
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Extract text or metadata from over a thousand file types.
Apache Tika is a content detection and analysis framework, written in Java, stewarded at the Apache Software Foundation. It detects and extracts metadata and text from over a thousand different file types, and as well as providing a Java library, has server and command-line editions suitable for use from other programming languages …
For most of the more common and popular formats, Tika then provides content extraction, metadata extraction and language identification capabilities. (From https://en.wikipedia.org/wiki/Apache_Tika, accessed Jan 18, 2018)
This is an R interface to the Tika software.
To start, you need R and Java 8 or OpenJDK 1.8. Higher versions work. To check your version, run the command java -version from a terminal. Get Java installation tips at http://openjdk.java.net/install/ or https://www.java.com/en/download/help/download_options.xml.
t devtools to easily install from github, until this is all on CRAN
requireNamespace('devtools')){
stall.packages('devtools',
repos = 'https://cloud.r-project.org')
requireNamespace('rtika')){
tools::install_github('ropensci/rtika')
ary('rtika')
ou need to install the Apache Tika .jar once.
his gets the development version with an important bug fix:
s.na(tika_jar())){
tall_tika()
The rJava package is not required.
Read an introductory article at https://ropensci.github.io/rtika/articles/rtika_introduction.html.
tika_text() to extract plain text.tika_xml() and tika_html() to get a structured XHMTL rendition.tika_json() to get metadata as .json, with XHMTL content.tika_json_text() to get metadata as .json, with plain text content.tika() is the main function the others above inherit from.tika_fetch() to download files with a file extension matching the Content-Type.Tika parses and extracts text or metadata from over one thousand digital formats, including:
.pdf).rtf).epub).jpeg, .png, etc.).mbox, Outlook).html).xml, etc.).gzip, .rar, etc.)For a list of MIME types, see: https://tika.apache.org/1.17/formats.html
The rtika package processes batches of documents efficiently, so I recommend batches. Currently, the tika() parsers take a tiny bit of time to spin up, and that will get annoying with hundreds of separate calls to the functions.
st files
h <- c(
stem.file("extdata", "jsonlite.pdf", package = "rtika"),
stem.file("extdata", "curl.pdf", package = "rtika"),
stem.file("extdata", "table.docx", package = "rtika"),
stem.file("extdata", "xml2.pdf", package = "rtika"),
stem.file("extdata", "R-FAQ.html", package = "rtika"),
stem.file("extdata", "calculator.jpg", package = "rtika"),
stem.file("extdata", "tika.apache.org.zip", package = "rtika")
tches are best, and can also be piped with magrittr.
<- tika_text(batch)
xt has one string for each document:
th(text)
1] 7
snippet:
substr(text[1], 54, 190))
ackage ?jsonlite?
une 1, 2017
ersion 1.5
itle A Robust, High Performance JSON Parser and Generator for R
icense MIT + file LICENSE
To learn more and find out how to extract structured text and metadata, read the vignette: https://ropensci.github.io/rtika/articles/rtika_introduction.html.
There is some overlap with many other text parsers, such as the R interface to antiword (See: https://github.com/ropensci/antiword). Listing all of them would take a huge amount of space, since Apache Tika processes over a thousand file types (See: https://tika.apache.org/). The main difference is that instead of specializing on a single format, Tika integrates dozens of specialist libraries from the Apache Foundation. Tika's unified approach offers a bit less control, and in return eases the parsing of digital archives filled with possibly unpredictable file types.
In September 2017, github.com user kyusque released tikaR, which uses the rJava package to interact with Tika (See: https://github.com/kyusque/tikaR). As of writing, it provided similar text and metadata extraction, but only xml output.
Back in March 2012, I started a similar project to interface with Apache Tika. My code also used low-level functions from the rJava package. I halted development after discovering that the Tika command line interface (CLI) was easier to use. My empty repository is at https://r-forge.r-project.org/projects/r-tika/.
I chose to finally develop this package after getting excited by Tika's new 'batch processor' module, written in Java. The batch processor has very good efficiency when processing tens of thousands of documents. Further, it is not too slow for a single document either, and handles errors gracefully. Connecting R to the Tika batch processor turned out to be relatively simple, because the R code is simple. It uses the CLI to point Tika to the files. Simplicity, along with continuous testing, should ease integration. I anticipate that some researchers will need plain text output, while others will want json output. Some will want multiple processing threads to speed things up. These features are now implemented in rtika, although apparently not in tikaR yet.
Please note that this project is released with a Contributor Code of Conduct. By participating in this project you agree to abide by its terms.