Text

· · to read

MessagePack and Cap’n Proto are two different serialization tools. MessagePack is a format for serialization. Cap’n Proto creates different formats for serialization, using a .capnp specification. The .capnp specification has a format that is understood by the capnp command-line tool, which is implemented in C++, but Cap’n Proto itself doesn’t come with a built-in universal serialization format. This is similar to Protocol Buffers. MessagePack, on the other hand, has client libraries with the equivalents of JSON.parse() and JSON.stringify().

Cap’n Proto is known for making it possible to skip the serialization/deserialization step, and to access the raw data in memory. This could also be done with MessagePack, though many libraries don’t support it.

For the use case of sending a bunch of JSON data, the default way of using MessagePack will involve sending each key in a JSON key/value pair N times. For this use case in Cap’n Proto, it prescribes creating a struct, which makes it so it won’t need to be sent at all with the data. It will only need to be included in the code that uses the protocol. This can be got around in MessagePack by using extensions or just by changing the data layout, perhaps by using two dimensional arrays like a CSV file, with a header.

With MessagePack you can have a client simply use a MessagePack library. With Cap’n Proto, you need to ship a serializer/deserializer. That is because it’s a self-describing format, like JSON.

Cap’n Proto pays attention to alignment, while MessagePack uses just enough bytes for the data. It’s worth looking at the spec – numbers from -32 to 127 only need a byte a piece. The byte alignment of Cap’n Proto makes it so an array of the same type of number could presumably be stored efficiently, but a array of mixed values would likely take more space. However, the byte alignment makes in-memory access of raw Cap’n Proto data very fast.

The creators of each are very accomplished software engineers. The author of MessagePack founded Treasure Data, which was acquired by Arm. The author of Cap’n Proto founded Sandstorm and now works at Cloudflare.

MessagePack is similar to tnetstrings. It was used by Socket.io but they stopped using it, using JSON and a minimal custom netstrings-like format for efficiently storing binary data. That shows that the techniques can be used without the tools.

I think that MessagePack and Cap’n Proto are very good for teaching some different serialization techniques. They also should be easy to use them together, because unlike JSON, both can store binary data efficiently. With JSON you have the overhead of converting it to and from base64. Also it can be nested efficiently. If you put JSON data inside a JSON string, all the double quotes need to be escaped, and this can result in inflated sizes if it’s nested deeply. So if you make a library that uses MessagePack but some want to use it with Cap’n Proto, I say keep using MessagePack, and vice versa. However, currently for me the edge goes to Cap’n Proto because it is so fast, and is more space efficient under typical use. However, Cap’n Proto’s efficient formats could possibly be partially recreated in MessagePack and might be preferable to many.

Text

· · to read

A poorly estimated programming task doesn’t just take longer than the estimated time. It also tends to takes longer than it would have taken if it were properly estimated, and is much more likely to not get done properly or to not get done at all.

Text

· · to read

Encoding to binary:

'hello'.replace(/./g, function(c) {
function pad(n) { return (n > 0) ? ('0' + pad(n - 1)) : ''; }
var s = c.charCodeAt(0).toString(2);
return pad(8 - s.length) + s;
});

Decoding binary to a string:

'011101110110111101110111'.replace(/.{8}/g, function(s) {
return String.fromCharCode(parseInt(s, 2));
});
Text

· · to read

I like building up a project from small parts but sometimes it’s good to try using a full framework to see what parts I’m missing. Today I tried starting a mean.js app with yeoman. It comes with a lot preconfigured, including five different authentication options, and since I chose it, an articles model. I removed four of the five authentication modules and changed the name of Article to Bin. In the process I found where things are stored in mean.js. It seems to do a good job of following the rails mantra of Convention Over Configuration.

  • The Code (unfinished, and most of it isn’t mine, the commits show what I did)

I want to return to this experiment and get it working the rest of the way and deployed on Heroku. The basic idea is a minimalistic jsfiddle.

Text

· · to read

I used d3 to make a map. It was good practice. I had forgotten how d3?s chaining API works. When a function mutates an element, it returns the same element back. When a function appends something to an element, it returns the new element back.

I also tried using patterns in svg. I still have to learn some more to understand them and to get the results I want, and I also need to brush up on biezer curves.

Update: I adjusted the vertical offset of the pattern to remove gaps between states.

doge-usa