nodesource/poblado

Name: poblado

Owner: NodeSource

Description: Parses and processes JSON that is written to it in chunks on a background thread.

Created: 2017-12-15 16:09:52.0

Updated: 2017-12-20 22:15:47.0

Pushed: 2017-12-22 17:24:30.0

Homepage: null

Size: 22

Language: C++

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

poblado

Parses and processes JSON that is written to it in chunks on a background thread.

Table of Contents generated with DocToc

• Dependencies
• API
• void onparsedToken(rapidjson_writable::SaxHandler& handler)
• void processParsedData()
• void onprocessingComplete()
• Overriding ASSERT
• Logging Diagnostics via POBLADO_LOG
• Example
• Output
• LICENSE

Dependencies

Poblado depends on the following Headers

• rapidjson
• rapidjson-writable
• libuv

Make sure you are including all those headers when building and in the case of libuv you need to either link to the libuv library as well or make it part of your build.

API

In order to use poblado you'll extend the Poblado class and override the following methods.

lude "poblado.h"

s MyPoblado : public Poblado {
id onparsedToken(rapidjson_writable::SaxHandler& handler);
id processParsedData();
id onprocessingComplete();

The methods are invoked either on the parser/processor (background) thread or the main thread (same as libuv's event loop).

poblado guarantees that it will not invoke any background thread methods after invoking onprocessingComplete on the main thread. Therefore users of the API don't have to use any thread synchronization as it is all handled under the hood by poblado.

void onparsedToken(rapidjson_writable::SaxHandler& handler)

• invoked on background thread.
• compare the handler.type to a rapidjson_writable::JsonType and from that decide how to process the respective values.
• for more information have a look at the rapidjson_writable::SaxHandler implementation
• store the extracted data in fields on your class so they are available in the following processing steps

void processParsedData()

• invoked on background thread.
• here the user can perform further processing on the data collected via onparsedToken

void onprocessingComplete()

• invoked on main thread.
• this method is invoked when both background processing methods have completed
• at this point the processing methods will not be called anymore and thus the process result can safely be accessed on the main thread

Overriding ASSERT

poblado uses a built in ASSERT method which can be overridden.

ine POBLADO_ASSERT MY_ASSERT
lude "poblado.h"

Logging Diagnostics via POBLADO_LOG

By default poblado is quiet, but has built in support to log diagnostic messages. All you have to do is supply a POBLADO_LOG method that takes a char* as an input.

ine POBLADO_LOG MY_LOG
lude "poblado.h"

Have a look at the top of test/test_dump.cc to see an example of that feature.

Example

Below is a sample implementation of the Poblado class. For a more complete example please read test/test_dump.cc or test/test_capitalize.cc.

s ExtractKeysProcessor : public Poblado {
blic:
ExtractKeysProcessor(uv_loop_t& loop) : Poblado(loop) {}

ivate:
ackground thread {

// @override
void onparsedToken(rapidjson_writable::SaxHandler& handler) {
  TEST_LOG("[processor] parsed a token");
  if (handler.type != rapidjson_writable::JsonType::Key) return;
  keys_.push_back(scopy(handler.stringVal.c_str()));
}

// @override
void processParsedData() {
  for (auto& key : keys_) {
    TEST_LOG(key);
    processedKeys_.push_back(string_toupper(key));

    // simulate that processing keys is slow
    uv_sleep(200);
  }
}

 background thread

ain thread {

// @override
void onprocessingComplete() {
  if (hasError()) {
    TEST_LOG("[collector] found error");
    fprintf(stderr, "%s\n", &error());
    return;
  }

  for (auto& key : processedKeys_) {
    TEST_LOG("[collector] printing key");
    TEST_LOG(key);
  }
}

 main thread

std::vector<const char*> keys_;
std::vector<const char*> processedKeys_;

Output

Note that the output is using a log method to provide diagnostic messages in order to demonstrate how things work under the hood.

The output format is as follows:

e](file:line)[thread id] [topic] message

/bin/test_dump test/fixtures/capitals.json
n] processing test/fixtures/capitals.json
00](test_dump.cc:157)[0x7fff88639340] [main] starting loop
04](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
04](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
04](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
04](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
04](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
26](test_dump.cc:85)[0x7fff88639340] [ticker] tick
26](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
26](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
26](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
26](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
26](test_dump.cc:39)[0x700004f9f000] [processor] parsed a token
26](poblado.h:89)[0x700004f9f000] [processor] parser complete, processing ...
26](test_dump.cc:47)[0x700004f9f000] colombia
29](test_dump.cc:47)[0x700004f9f000] australia
29](test_dump.cc:47)[0x700004f9f000] germany
32](test_dump.cc:47)[0x700004f9f000] united states
37](poblado.h:141)[0x700004f9f000] [processor] signaling processing is complete
37](poblado.h:123)[0x7fff88639340] [collector] found processing is complete, reading results
37](test_dump.cc:68)[0x7fff88639340] [collector] printing key
37](test_dump.cc:69)[0x7fff88639340] COLOMBIA
37](test_dump.cc:68)[0x7fff88639340] [collector] printing key
37](test_dump.cc:69)[0x7fff88639340] AUSTRALIA
37](test_dump.cc:68)[0x7fff88639340] [collector] printing key
37](test_dump.cc:69)[0x7fff88639340] GERMANY
37](test_dump.cc:68)[0x7fff88639340] [collector] printing key
37](test_dump.cc:69)[0x7fff88639340] UNITED STATES
57](test_dump.cc:85)[0x7fff88639340] [ticker] tick

LICENSE

MIT