Name: watson-document-classifier
Owner: International Business Machines
Description: Augment IBM Watson Natural Language Understanding APIs with a configurable mechanism for text classification, uses Watson Studio.
Created: 2017-07-06 18:38:40.0
Updated: 2018-05-18 19:19:12.0
Pushed: 2018-03-22 01:40:57.0
Homepage: https://developer.ibm.com/code/patterns/extend-watson-text-classification/
Size: 1150
Language: Jupyter Notebook
GitHub Committers
User | Most Recent Commit | # Commits |
---|
Other Committers
User | Most Recent Commit | # Commits |
---|
Read this in other languages: ???.
Data Science Experience is now Watson Studio. Although some images in this code pattern may show the service as Data Science Experience, the steps and processes will still work.
In this code pattern we will use Jupyter notebooks in Watson Studio to augment IBM Watson Natural Language Understanding API output through configurable mechanism for text classification.
When the reader has completed this code pattern, they will understand how to:
The intended audience for this code pattern is developers who want to learn a method for augmenting classification metadata obtained from Watson Natural Language Understanding API, in situations when there is a scarcity of historical data. The traditional approach of training a Text Analytics model yields less than expected results. The distinguishing factor of this code pattern is that it allows a configurable mechanism of text classification. It helps give a developer a head start in the case of text from a specialized domain, with no generally available English parser.
IBM Watson Studio: Analyze data using RStudio, Jupyter, and Python in a configured, collaborative environment that includes IBM value-adds, such as managed Spark.
IBM Cloud Object Storage: An IBM Cloud service that provides an unstructured cloud data store to build and deliver cost effective apps and services with high reliability and fast speed to market.
Watson Natural Language Understanding: A IBM Cloud service that can analyze text to extract meta-data from content such as concepts, entities, keywords, categories, sentiment, emotion, relations, semantic roles, using natural language understanding.
Follow these steps to setup and run this code pattern. The steps are described in detail below.
Sign up for IBM's Watson Studio. By creating a project in Watson Studio a free tier Object Storage
service will be created in your IBM Cloud account. Take note of your service names as you will need to select them in the following steps.
Note: When creating your Object Storage service, select the
Free
storage type in order to avoid having to pay an upgrade fee.
Create the following IBM Cloud service and name it wdc-NLU-service:
Create notebook
to create a notebook.Assets
tab, select the Create notebook
option.From URL
tab.Create
button.My Projects > Default
page, Use Find and Add Data
(look for the 10/01
icon)
and its Files
tab.browse
and navigate to this repo watson-document-classifier/data/sample_text.txt
browse
and navigate to this repo watson-document-classifier/configuration/sample_config.txt
Note: It is possible to use your own data and configuration files. If you use a configuration file from your computer, make sure to conform to the JSON structure given in
configuration/sample_config.txt
.
If you use your own data and configuration files, you will need to update the variables that refer to the data and configuration files in the Jupyter Notebook.
In the notebook, update the global variables in the cell following 2.3 Global Variables
section.
Replace the sampleTextFileName
with the name of your data file and sampleConfigFileName
with your configuration file name.
Select the cell below 2.1 Add your service credentials from IBM Cloud for the Watson services
section in the notebook to update the credentials for Watson Natural Langauage Understanding.
Open the Watson Natural Language Understanding service in your IBM Cloud Dashboard and click on your service, which you should have named wdc-NLU-service
.
Once the service is open click the Service Credentials
menu on the left.
In the Service Credentials
that opens up in the UI, select whichever Credentials
you would like to use in the notebook from the KEY NAME
column. Click View credentials
and copy username
and password
key values that appear on the UI in JSON format.
Update the username
and password
key values in the cell below 2.1 Add your service credentials from IBM Cloud for the Watson services
section.
2.2 Add your service credentials for Object Storage
section in the notebook to update the credentials for Object Store.Find and Add Data
(look for the 10/01
icon) and its Files
tab. You should see the file names uploaded earlier. Make sure your active cell is the empty one below 2.2 Add...
Insert to code
(below your sample_text.txt).Insert Crendentials
from drop down menu.credentials_1
.When a notebook is executed, what is actually happening is that each code cell in the notebook is executed, in order, from top to bottom.
IMPORTANT: The first time you run your notebook, you will need to install the necessary packages in section 1.1 and then
Restart the kernel
.
Each code cell is selectable and is preceded by a tag in the left margin. The tag
format is In [x]:
. Depending on the state of the notebook, the x
can be:
*
, this indicates that the cell is currently executing.There are several ways to execute the code cells in your notebook:
Play
button in the toolbar.Cell
menu bar, there are several options available. For example, you
can Run All
cells in your notebook, or you can Run All Below
, that will
start executing from the first cell under the currently selected cell, and then
continue executing all cells that follow.Schedule
button located in the top right section of your notebook
panel. Here you can schedule your notebook to be executed once at some future
time, or repeatedly at your specified interval.sample_text_classification.txt
file using select box to the left of the file listingSelectAction
button and use the Download File
drop down menu to download sample_text_classification.txt
file.After running each cell of the notebook under Classify text
, the results will display.
The configuration json controls the way the text is classified. The classification process is divided into stages - Base Tagging and Domain Tagging. The Base Tagging stage can be used to specify keywords based classification, regular expression based classification, and tagging based on chunking expressions. The Domain Tagging stage can be used to specify classification that is specific to the domain, in order to augment the results from Watson Natural Language Understanding.
We can modify the configuration json to add more keywords or add regular expressions. In this way, we can augment the text classification without any changes to the code. We can add more stages to the configuration json if required and enhance the text classification results with code modifications.
It can be seen from the classification results that the keywords and regular expressions specified in the configuration have been correctly classified in the analyzed text that is displayed.