screepers/utf15

Name: utf15

Owner: screepers

Description: Library (codec generator) for packing integers to JavaScript UTF-16 strings

Created: 2017-11-29 10:57:37.0

Updated: 2017-12-05 18:49:45.0

Pushed: 2017-12-01 16:22:45.0

Homepage: null

Size: 10

Language: JavaScript

GitHub Committers

User	Most Recent Commit	# Commits

Other Committers

User	Email	Most Recent Commit	# Commits

README

utf15

Library (codec generator) for packing integers to JavaScript UTF-16 strings

The main use cases of the library:

• Converting integer arrays to UTF-16 string: in case you know the exact bit depth (“maximum value”) of numbers, you can save them with reduced bit depth to a few UTF-16 code units instead of much larger string by JSON.stringify. Example:

  [1337,1337,1337] + depth:15 => "???"

• Converting array with variadic bit depths to UTF-16 string: values can be effectively packed even if they have different bit depths. Example:

  [0xFF, 0xAA, 0xFFFF, 0xDEAD] + depths:[8,8,16,16] => "???"

• Converting single integers to UTF-16 string: you can customize codecs as you wish to create any kind of serialization/deserialization protocol.

Features:

• Output code points range is [0x0030, 0x8030) (with length = 2^15 = 32768), so control characters [0x0000, 0x0020) and surrogate code points [0xD800, 0xDFFF] will never appear in the output, which could be critical or ineffective while serialization/deserialization.
• Dense binary packing: for example, 3 integer values with bit_depth=15 will occupy exactly 3 UTF-16 code units, and 10 values with bit_depth=3 will occupy exactly 2 UTF-16 code units.
• Arrays serialization/deserialization can be performed with variadic bit depths for each element.
• Meta information can be stored within the encoded string for future decoding without knowing a particular encoding codec.
• Lightweight, acceptable performance (see TravisCI output for measurement results).

What the utf15.js is not:

• It's not an lz-string analogue: it won't compress your data in terms of binary size (4 bytes UInt32 value may become 6 bytes UTF-16 string).
• It's not an archiver: see previous point.
• It's not a JSON.stringify replacement: utf15.js can process unsigned integers ? [0,Number.MAX_SAFE_INTEGER], arrays of such integers, and only them.

Examples:

Here some useful examples of library use cases, see more in tests.js:examples() section.

Global RoomPosition packing

t values = [3, 101, 97, 24, 42];    // room W101N97 x:24 y:42
t depths = [2,   7,  7,  6,  6];    // max = [4,128,128,64,64]

odec for position coding, oh, exploitable!
t map_codec = new Codec({ depth:depths, array:1 });

= map_codec.encode(values);
= map_codec.decode(enc);
rt(arr_eq(values, dec));
ole.log(values, `[str.length=${enc.length}]`, dec);

> [ 3, 101, 97, 24, 42 ] '[str.length=2]' [ 3, 101, 97, 24, 42 ]

Directions packing

t values = [0,1,2,3,4,5,6,7,6,5];
t depth = 3; // 3 bits, max 8 values, range [0,7]

t dir_codec = new Codec({ depth, array:1 });

= dir_codec.encode(values);
= dir_codec.decode(enc);
rt(arr_eq(values, dec));
ole.log(values, `[str.length=${enc.length}]`, dec);

> [ 0, 1, 2, 3, 4, 5, 6, 7, 6, 5 ] '[str.length=3]' [ 0, 1, 2, 3, 4, 5, 6, 7, 6, 5 ]

Single integer packing

t value = 1337;

epth === 11 bits, value < 2^depth, enough
t codec = new Codec({ depth:11 });
= codec.encode(value);
= codec.decode(enc);
rt(value === dec);
ole.log(value, enc, dec);

> 1337 '?' 1337

ID packing

t
id = '5a216a3df33b4e435ca8c5ab',    // object.id example
values = [];                        // array of hex numbers

let i = 0, len = id.length; i < len; ++i) {
const point = parseInt(id[i], 16);
values.push(point);


t id_codec = new Codec({ depth:4, array: 1 });
= id_codec.encode(values);
= id_codec.decode(enc);

t decoded_id = dec.reduce((acc, x)=> (acc + x.toString(16)), '');

rt(id === decoded_id);
ole.log(`${id}.len=${id.length}`, `[str.len=${enc.length}]`, decoded_id);

> 5a216a3df33b4e435ca8c5ab.len=24 [str.len=8] 5a216a3df33b4e435ca8c5ab