Case Study: Micro-services using Erlang

At the Erlang User Conference 2017 (EUC17) in Stockholm Sweden, I presented our experience of how Alert Logic moved to a new micro-services architecture using our preferred programming language: Erlang.

Alert Logic had been using Erlang for years before, but we really did not have a holistic platform for building applications. A small team of us went off and build a new product for the company, and in doing so build a new application platform using micro-services patterns and written in the Erlang programming language.

You can watch a recording of the presentation below, as well as review the slide deck.


Rapid Prototyping with MacroFab, Part 2

As I described in Part 1, I am using MacroFab for rapid prototyping of my jeepbot concept project. In this post, I’ll describe the next phase of my project and how MacroFab has made it easy to get there.

As I thought about what jeepbot should be, the concept evolved into several different ideas. What I wanted to build first was a control board that directly controlled a simple relay box, like the one described by Hooz’s DIY project post. The idea was to focus on the control circuitry first, rather than the relay circuitry. The control board would replace a bank of switches, so easily plug-and-play. This would allow me to fine-tune the design over time, while retaining a good, simple relay box in the vehicle. I also renamed the project concept switchbot to go with the idea of being vehicle independent.

For my first board, I designed a circuit that contained a Atmel ATmega328P micro-controller, a MCP2515 CAN controller, a MCP2551 CAN transceiver, and a 74HC595D 8-bit shift register. I only designed in 4 switch circuits for this initial board. Each switch circuit is protected by a LTV816 optocoupler, to isolate the micro-controller’s digital logic from the actual switches. These output lines can in turn be used to drive other circuits, such as a relay board.

To round out the board, I added a two-port terminal connector for +12v power and ground, a two-port terminal connector for CAN high and low connections, and a DB9 connector for easily interfacing into automobile OBD-II connectors. I used a 6-port Molex connector for output for power, ground, and the 4 switch channels. A reset button and a 2×3 header for in-system chip programming (ICSP) interface to program the ATmega micro-controller was important to make sure I could download firmware later.

The board also has a simple +12v to +5v power supply regulation circuit and protection diode, but not with significant automotive transient protection. That circuitry will likely be needed in future versions.

You can find the Eagle files for this version of the board on github at

Like with my previous board, I used the MacroFab web interface to upload my Eagle design files. This board had a lot more parts than my previous one, and my experience with the previous board taught me I needed to be more careful in double-checking my part selection. Even with my caution, the engineers at MacroFab found several discrepancies for me to clarify, mostly with resistor values.

While the board was being produced, I discovered the diode value mistake I made with my previous board. I found the same mistake in this board, so I asked MacroFab if it could be replaced – yep! And they did so easily, since this was before assembly of the board had actually begun.


During the final assembly of the board, MacroFab found the two-port connector parts I selected for my CAN and power interfaces were wrong for my board design. I asked them not to populate those connectors as I had many of the correct ones in my workshop.

MacroFab also found during assembly that the crystal oscillator part I had selected would not  match the pins exactly on my board. The size was correct, but the orientation was wrong. I recall that selecting the oscillator part was difficult, so I was not too surprised. A little bit more research from the parts houses and I found what I thought would be a better part. MacroFab agreed and ordered it for me. This extra order only added a couple of days to the overall board assembly time.

IMG_2448A couple of weeks after ordering, my boards arrived. I ordered two of the boards this time, as I wanted to be able to leave one in my Jeep and use another for bench testing. As soon as they arrived, I hand soldered in the two-port connectors I had (I have a nice supply of them from SparkFun as they are so commonly used on Arduino-based boards).

The next step was to power up the board and see what happens. No smoke; it lives! Without a program downloaded to the micro-controller nothing much happened, but the solid power-on LED and the all 4 switch LEDs being on indicated to me that the core of the system was alive.


Next I used an AVR ICSP programmer to download software using the Arduino IDE, which front-ends the avrdude tool. I picked the Arduino blink sketch to see if it would download… success! I do not have an LED on pin 13 of the ATmega so I couldn’t see the sketch actually running, so next I dowloadeded my shiftreg test sketch. This sketch will cycle through all 4 switches and leave each one active for a few milliseconds. It will also display anything it receives over the CAN bus to its own serial output, although I did not make it easy to show serial output on this board.


Surprisingly to me, not only did the sketch download, it actually worked. The switch indicator LEDs lit up in turn, and with a meter I could measure the signal voltage on each output line on the Molex connector. Success!

I now have the basis for real hardware testing in my vehicle. I’ll post a follow-up on further progress with this project.

The most amazing thing about this experience so far has been the ease of use of the MacroFab interface, their quick turn-around time, and the pricing. For these two boards, the price was approximately $76 each. For a hobbyist, this pricing puts real low-volume projects in easy each. At higher quantities, the price drops dramatically. For 1,000 units, for example, pricing approaches $15 a board for this design.

Stay tuned…


CAN Bus Hacking with the Arduino and Raspberry Pi

In December of 2013, I published a set of videos on YouTube discussing how to do CAN bus hacking using an Arduino and/or Raspberry Pi. These videos were made in conjunction with my Jeep hacking projects.

Several viewers have asked for a little bit more information on getting an Arduino and Raspberry Pi to talk to one another over a CAN bus, so I thought I’d provide a bit of written information here to accompany the videos. In addition, two years of time have provided some improvements that make everything easier.

For both the Arduino and the Raspberry Pi, I used daughter boards that use Microchip’s MCP2515 CAN controller and MCP2551 CAN transceiver. These are extremely common integrated circuits for CAN and are great to integrate with as they use an Serial Peripheral Interface (SPI).

The Videos

The CAN Bus

A CAN bus has to be properly terminated on each end of the bus. The easiest way to do this on a bench is using a breadboard. Use two 120 Ω resistors on each end of one of the power rails to terminate the bus properly. You can then connect your nodes to any location within the power rail and have them join the bus properly.

Be sure to create a twisted pair out of each set of wires coming from each node. CAN bus is highly resilient to electrical noise when each node follows certain rules, and twisted pairs of wires is one of those rules.

A CAN bus setup on a breadboard
A CAN bus setup on a breadboard

The Arduino Setup

Sparkfun sells a CAN bus shield designed by SK Pang. This shield is great for developing automotive-focused applications, and is extremely easy to use from a software standpoint. Once built, the shield plugs into an Arduino with no further configuration.

For software, the following should be downloaded:

  1. My AVRDebug library –
  2. My “hello, world” application –
  3. The library for the Sparkfun / SK Pang shield –

Once built and downloaded, the project will broadcast a message to the CAN bus every 500ms. The message will contain an ASCII string value of a monotonically increasing integer value.

If you run the project without another node connected to the CAN bus, expect to start seeing errors after a few messages are queued up. The sender requires another node on the network to acknowledge the message electrically. This can trip you up if you are first starting out with the CAN bus as it is non-obvious why you might be getting an error.

The Raspberry Pi Setup

The easiest way to get CAN support on your Raspberry Pi is to buy a daughter board from SK Pang: the PICAN board. You can build your own configuration using mcp2515 and mcp2551 ICs, but for the price the PICAN is an easy bet.

Today the Raspberry Pi setup is much easier than ever before. Pick the 2015-02-16 or newer version of Raspian and you will not have to compile any kernel modules.

See this blog post on SK Pang site on setting up your complete system.

On the RPi, the regular SK Pang setup they have documented works great. On mine, the end of my /boot/config.txt looks like:

and then my /etc/network/interfaces file looks like:

auto can0
iface can0 inet manual
    pre-up ip link set $IFACE type can bitrate 500000 listen-only off triple-sampling on
    up /sbin/ifconfig $IFACE up
    down /sbin/ifconfig $IFACE down
The ip command can give you a big clue on state if you use this command:
root@raspberrypi:~# ip -details -statistics link show can0
3: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 10
    can <TRIPLE-SAMPLING> state ERROR-ACTIVE restart-ms 0
    bitrate 500000 sample-point 0.875
    tq 125 prop-seg 6 phase-seg1 7 phase-seg2 2 sjw 1
    mcp251x: tseg1 3..16 tseg2 2..8 sjw 1..4 brp 1..64 brp-inc 1
    clock 8000000
re-started bus-errors arbit-lost error-warn error-pass bus-off
     0          0          0          0          0          0     
RX: bytes  packets  errors  dropped overrun mcast
     632288     79036    0       103     0       0     
TX: bytes  packets  errors  dropped carrier collsns
     0          0        0       0       0       0
The couple of things I bolded are key. The clock should be 1/2 of what the oscillator setting in config.txt is (weird quirk). The state is the actual MCP2515 state – so error-active is what you usually want to see. Error-passive or Bus-off are bad. See the mcp2515 datasheet for details there.
You’ll want can-utils downloaded and installed:
Then, if you run the Arduino project so it’s sending data every 500 ms, and then run this command on the RPi, you should see some similar output:
pi@raspberrypi ~ $ candump -t a -c -c -a can0
 (1452184714.419422)  can0  1FF   [8]  31 34 32 35 33 00 00 00   '14253...'
 (1452184714.939332)  can0  1FF   [8]  31 34 32 35 34 00 00 00   '14254...'
 (1452184715.459257)  can0  1FF   [8]  31 34 32 35 35 00 00 00   '14255...'
You can use the can-utils for just about everything. cansend and canplayer are useful for sending messages, candump and cansniffer are the two used for watching data (cansniffer is especially awesome for figuring out what’s really happening). If you want to write custom software that uses CAN on the RPi, then you can use these as examples. It’s effectively just using the socket system calls.
So that’s pretty much it. You have to have the bus speed match, you have to have the correct wiring, but otherwise that’s all there is to it. I’ve got a setup with an Arduino as shown below, the RPi as shown below, another RPi with a different CAN board (still mcp25xx), and a breadboard cased ATMEGA setup without the Arduino hardware. All works great.

Jeep Bot – Proof of Concept #1

I finished the 1st proof-of-concept for using the CAN-Bus data to control auxiliary relays. It worked great. Attached is a block diagram of what I used, and a longish video of how the testing went.

Jeep Bot Block Diagram


If you want to skip all the bench testing, the actual in-car testing happens at the 11:15 mark of the video.

Code for the demo can be found at

A few hours after I finished, a bluetooth-low-energy board I ordered showed up. The next step will be to throw that on there so that I can use the smartphone to configure each switch and optionally control them by hand.

Each switch will have the following different possible control states:

  • always on
  • manual only
  • on when interior lights are on
  • on when high-beams are on

Right after that, I’ll start working on a prototype PCB and housing so I can start testing real versions of this system.

I’ll post the design, schematics and code in progress on my blog so anyone can offer feedback as it gets built.

JavaScript Evaluation for Erlang Applications

If you are an Erlang developer you are likely familiar with the Riak storage engine, and how the MapReduce feature can evaluate arbitrary JavaScript expressions. Can you use this same code to do your own JavaScript evaluations for your Erlang projects? Yes, and it turns out to be surprisingly easily.

Getting your development environment running erlang_js is the first order of business. I’ve done all of my testing on a Mac OS X system, both Lion (10.7) and Mavericks (10.9). I use Erlang R14B04 for these tests, since that version is what we primarily target at the office.

If you want to manage multiple versions of Erlang on your OS X system, I highly recommend the erlbrew utility as a quick and easy way to maintain multiple versions. We’ve recently made it support R14 builds on OS X Mavericks, so there’s no excuse not to use it!

Assuming you have a working Erlang and OS X command-line development tools active, execute the following to checkout the erlang_js project and build it locally:

git clone git://
cd erlang_js; make all test

One of the most compelling uses of a JavaScript engine from another language like Erlang is the ability to pass arbitrarily deep objects as JSON so they can be evaluated. This allows your application to use JavaScript as a general-purpose expression language even if the objects are backed by an Erlang term. This is effectively what you get with Riak when using the MapReduce feature.

As an example, let’s imagine a deeply nested data structure that represents a server configuration. We’d like to execute a JavaScript expression against that data to find the first active network interface. Here’s an example of how you’d do it using erlang_js:

% Start the erlang_js application and driver.
ok = application:start(erlang_js).
{ok, JSDriver} = js_driver:new().
% Create a deeply-nested data structure to evaluate
ServerObj = {struct, [
    {&lt;&lt;"interfaces"&gt;&gt;, [
        {struct, [
        {struct, [
        {struct, [
% Evaluate our object with an arbitrary JavaScript expression
JSFun = &lt;&lt;"function is_any_interface_enabled(Server) { for (var i = 0; i &lt; Server.interfaces.length; i++) { if (Server.interfaces[i].enabled) return true; } return false; }"&gt;&gt;.
js:call(JSDriver, JSFun, [ServerObj]).

The real trick is building your Erlang data structure properly so that the custom version of mochijson2 can encode the object as JSON. As you see from the examples, use tuples with a struct atom as the first member to specify an object. Arrays are just arrays.

I have noticed that it is very easy to confuse a single instance of the js_driver object and it is hosed until a new one is created, so consider that in how you structure your driver usage.