Name: crux
Owner: Lucidworks
Description: Crux parses Web pages to identify the crux of an article, minus all the fluff.
Created: 2018-01-25 22:24:50.0
Updated: 2018-04-01 21:27:15.0
Pushed: 2018-01-26 02:40:37.0
Size: 3996
Language: HTML
GitHub Committers
| User | Most Recent Commit | # Commits |
|---|
Other Committers
| User | Most Recent Commit | # Commits |
|---|
Crux parses Web pages to identify the crux of an article ? the very essential points ? minus all the fluff. The library consists of multiple independent APIs. You can pick and choose which ones you want to use. If you use Crux in an Android app, they are designed to be independent so that Proguard or other minification tools can strip out the parts you don?t use.
Continuous integration with unit tests and golden file tests.In a background thread, make a network request and obtain the rawHTML of the page you would like
to analyze.
ng url = "https://example.com/article.html";
ng rawHTML = "<html><body><h1>This is an article</h1></body></html>";
cle article = ArticleExtractor.with(url, rawHTML)
.extractMetadata()
.extractContent() // If you only need metadata, you can skip `.extractContent()`
.article();
On the UI thread:
se article.document, article.title, etc.
From a single DOM Element root, the Image URL API inspects the sub-tree and returns the best
possible image URL candidate available within it. It does this by scanning within the DOM tree
for interesting src & style tags.
All URLs are resolved as absolute URLs, even if the HTML contained relative URLs.
eUrlExtractor.with(url, domElement).findImage().imageUrl();
From a single DOM Element root, the Image URL API inspects the sub-tree and returns the best
possible link URL candidate available within it. It does this by scanning within the DOM tree
for interesting href tags.
All URLs are resolved as absolute URLs, even if the HTML contained relative URLs.
UrlExtractor.with(url, domElement).findLink().linkUrl();
This API examines a given URL (without connecting to the server), and returns heuristically-determined answers to questions such as:
This API also supports resolving redirects for certain well-known redirectors, with the precondition that the target URL be available as part of this candidate URL. In other words, this API will not be able to resolve redirectors that perform a HTTP 301 redirect.
URL cruxUrl = CruxURL.parse("https://example.com/article.html");
Url.resolveRedirects();
Url.isLikelyArticle(); // Returns true.
Url.isLikelyImage(); // Returns false.
Include Crux in your project, then see sample code for each API provided above.
Crux uses semantic versioning. If the API changes, then the major version will be incremented. Upgrading from one minor version to the next minor version within the same major version should not require any client code to be modified.
Project/build.gradle:
rojects {
positories {
maven { url "https://jitpack.io" }
Module/build.gradle:
From the Releases page, copy the latest version number & use it below:
ndencies {
mpile 'com.github.chimbori:crux:<version>'
Crux began as a fork of Snacktory with the goal of making it more performant on Android devices, but it has quickly gained several new features that are not available in Snacktory.
Snacktory (and thus Crux) borrow ideas and test cases from Goose and JReadability.
Copyright 2016, Chimbori, makers of Hermit, the Lite Apps Browser.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.