Malformed Core Dumps – Episode 2 – Mark Allen Interview (Part 2 of 2)

In Episode 2, I continue the interview with Mark Allen.

Episode 2 Notes

  • Starting out as a systems administrator and working for Southwestern Bell
  • Working in Silicon Valley during the dot-com boom, and then surviving the following crash
  • Open Source origins and projects at work
    • Mark is maintaining kerl, an Erlang installation manager at https://github.com/kerl/kerl
    • Mark is now maintaining AWS Beam, an AWS API client for Erlang and Elixir at https://github.com/aws-beam
    • Mark would love to work more directly on large distributed-systems problems, especially time-series databases
    • Mark worked at Basho, an open-source company that was known for Riak but is now defunct
      • riak-core – https://github.com/basho/riak_core
      • lager – https://github.com/basho/lager
      • machi – https://github.com/basho/machi
  • Target, career changes, and leaving Minnesota
  • Industry Ethics
  • Quick Questions
    • Favorite BBQ is Roegels! http://www.roegelsbarbecue.com
    • Favorite place to visit: Santa Fe, NM!
    • Favorite programming language: Erlang
    • What would do you if money was no object? A ranch in New Mexico sounds nice…

Malformed Core Dumps – Episode 1 – Mark Allen Interview (Part 1 of 2)

Podcast Introduction

Welcome everyone to the Malformed Core Dumps podcast! This is your host, Chad Gibbons, and this is our very first episode of this podcast, so a big thank you for checking us out.

So, what’s this podcast all about? In short, it’s about personal story-telling from technology industry practitioners about their career journey.

We’ll have interviews with folks about their own experiences, and will likely go off on tangents about technology topics we find interesting.

My goal for this podcast isn’t a secret: I’m hoping to take the artificial barriers to entry out of our industry, both for people thinking about joining the field, and those who have already been involved for a long time.

Episode 1 Notes

 

 

Case Study: Micro-services using Erlang

At the Erlang User Conference 2017 (EUC17) in Stockholm Sweden, I presented our experience of how Alert Logic moved to a new micro-services architecture using our preferred programming language: Erlang.

Alert Logic had been using Erlang for years before, but we really did not have a holistic platform for building applications. A small team of us went off and build a new product for the company, and in doing so build a new application platform using micro-services patterns and written in the Erlang programming language.

You can watch a recording of the presentation below, as well as review the slide deck.

 

Rapid Prototyping with MacroFab, Part 2

As I described in Part 1, I am using MacroFab for rapid prototyping of my jeepbot concept project. In this post, I’ll describe the next phase of my project and how MacroFab has made it easy to get there.

As I thought about what jeepbot should be, the concept evolved into several different ideas. What I wanted to build first was a control board that directly controlled a simple relay box, like the one described by Hooz’s DIY project post. The idea was to focus on the control circuitry first, rather than the relay circuitry. The control board would replace a bank of switches, so easily plug-and-play. This would allow me to fine-tune the design over time, while retaining a good, simple relay box in the vehicle. I also renamed the project concept switchbot to go with the idea of being vehicle independent.

For my first board, I designed a circuit that contained a Atmel ATmega328P micro-controller, a MCP2515 CAN controller, a MCP2551 CAN transceiver, and a 74HC595D 8-bit shift register. I only designed in 4 switch circuits for this initial board. Each switch circuit is protected by a LTV816 optocoupler, to isolate the micro-controller’s digital logic from the actual switches. These output lines can in turn be used to drive other circuits, such as a relay board.

To round out the board, I added a two-port terminal connector for +12v power and ground, a two-port terminal connector for CAN high and low connections, and a DB9 connector for easily interfacing into automobile OBD-II connectors. I used a 6-port Molex connector for output for power, ground, and the 4 switch channels. A reset button and a 2×3 header for in-system chip programming (ICSP) interface to program the ATmega micro-controller was important to make sure I could download firmware later.

The board also has a simple +12v to +5v power supply regulation circuit and protection diode, but not with significant automotive transient protection. That circuitry will likely be needed in future versions.

You can find the Eagle files for this version of the board on github at https://github.com/dcgibbons/switchbot/releases/tag/v1.0_alpha

Like with my previous board, I used the MacroFab web interface to upload my Eagle design files. This board had a lot more parts than my previous one, and my experience with the previous board taught me I needed to be more careful in double-checking my part selection. Even with my caution, the engineers at MacroFab found several discrepancies for me to clarify, mostly with resistor values.

While the board was being produced, I discovered the diode value mistake I made with my previous board. I found the same mistake in this board, so I asked MacroFab if it could be replaced – yep! And they did so easily, since this was before assembly of the board had actually begun.

IMG_2447

During the final assembly of the board, MacroFab found the two-port connector parts I selected for my CAN and power interfaces were wrong for my board design. I asked them not to populate those connectors as I had many of the correct ones in my workshop.

MacroFab also found during assembly that the crystal oscillator part I had selected would not  match the pins exactly on my board. The size was correct, but the orientation was wrong. I recall that selecting the oscillator part was difficult, so I was not too surprised. A little bit more research from the parts houses and I found what I thought would be a better part. MacroFab agreed and ordered it for me. This extra order only added a couple of days to the overall board assembly time.

IMG_2448A couple of weeks after ordering, my boards arrived. I ordered two of the boards this time, as I wanted to be able to leave one in my Jeep and use another for bench testing. As soon as they arrived, I hand soldered in the two-port connectors I had (I have a nice supply of them from SparkFun as they are so commonly used on Arduino-based boards).

The next step was to power up the board and see what happens. No smoke; it lives! Without a program downloaded to the micro-controller nothing much happened, but the solid power-on LED and the all 4 switch LEDs being on indicated to me that the core of the system was alive.

IMG_2450

Next I used an AVR ICSP programmer to download software using the Arduino IDE, which front-ends the avrdude tool. I picked the Arduino blink sketch to see if it would download… success! I do not have an LED on pin 13 of the ATmega so I couldn’t see the sketch actually running, so next I dowloadeded my shiftreg test sketch. This sketch will cycle through all 4 switches and leave each one active for a few milliseconds. It will also display anything it receives over the CAN bus to its own serial output, although I did not make it easy to show serial output on this board.

yqbnj

Surprisingly to me, not only did the sketch download, it actually worked. The switch indicator LEDs lit up in turn, and with a meter I could measure the signal voltage on each output line on the Molex connector. Success!

I now have the basis for real hardware testing in my vehicle. I’ll post a follow-up on further progress with this project.

The most amazing thing about this experience so far has been the ease of use of the MacroFab interface, their quick turn-around time, and the pricing. For these two boards, the price was approximately $76 each. For a hobbyist, this pricing puts real low-volume projects in easy each. At higher quantities, the price drops dramatically. For 1,000 units, for example, pricing approaches $15 a board for this design.

Stay tuned…

 

CAN Bus Hacking with the Arduino and Raspberry Pi

In December of 2013, I published a set of videos on YouTube discussing how to do CAN bus hacking using an Arduino and/or Raspberry Pi. These videos were made in conjunction with my Jeep hacking projects.

Several viewers have asked for a little bit more information on getting an Arduino and Raspberry Pi to talk to one another over a CAN bus, so I thought I’d provide a bit of written information here to accompany the videos. In addition, two years of time have provided some improvements that make everything easier.

For both the Arduino and the Raspberry Pi, I used daughter boards that use Microchip’s MCP2515 CAN controller and MCP2551 CAN transceiver. These are extremely common integrated circuits for CAN and are great to integrate with as they use an Serial Peripheral Interface (SPI).

The Videos

The CAN Bus

A CAN bus has to be properly terminated on each end of the bus. The easiest way to do this on a bench is using a breadboard. Use two 120 Ω resistors on each end of one of the power rails to terminate the bus properly. You can then connect your nodes to any location within the power rail and have them join the bus properly.

Be sure to create a twisted pair out of each set of wires coming from each node. CAN bus is highly resilient to electrical noise when each node follows certain rules, and twisted pairs of wires is one of those rules.

A CAN bus setup on a breadboard
A CAN bus setup on a breadboard

The Arduino Setup

Sparkfun sells a CAN bus shield designed by SK Pang. This shield is great for developing automotive-focused applications, and is extremely easy to use from a software standpoint. Once built, the shield plugs into an Arduino with no further configuration.

For software, the following should be downloaded:

  1. My AVRDebug library – https://github.com/dcgibbons/AVRDebug
  2. My “hello, world” application – https://github.com/dcgibbons/CANBusHello
  3. The library for the Sparkfun / SK Pang shield – https://github.com/sparkfun/SparkFun_CAN-Bus_Arduino_Library

Once built and downloaded, the project will broadcast a message to the CAN bus every 500ms. The message will contain an ASCII string value of a monotonically increasing integer value.

If you run the project without another node connected to the CAN bus, expect to start seeing errors after a few messages are queued up. The sender requires another node on the network to acknowledge the message electrically. This can trip you up if you are first starting out with the CAN bus as it is non-obvious why you might be getting an error.

The Raspberry Pi Setup

The easiest way to get CAN support on your Raspberry Pi is to buy a daughter board from SK Pang: the PICAN board. You can build your own configuration using mcp2515 and mcp2551 ICs, but for the price the PICAN is an easy bet.

Today the Raspberry Pi setup is much easier than ever before. Pick the 2015-02-16 or newer version of Raspian and you will not have to compile any kernel modules.

See this blog post on SK Pang site on setting up your complete system.

On the RPi, the regular SK Pang setup they have documented works great. On mine, the end of my /boot/config.txt looks like:
dtparam=spi=on
dtoverlay=mcp2515-can0-overlay,oscillator=16000000,interrupt=25
dtoverlay=spi-bcm2835-overlay
dtoverlay=spi-dma-overlay

and then my /etc/network/interfaces file looks like:

auto can0
iface can0 inet manual
    pre-up ip link set $IFACE type can bitrate 500000 listen-only off triple-sampling on
    up /sbin/ifconfig $IFACE up
    down /sbin/ifconfig $IFACE down
The ip command can give you a big clue on state if you use this command:
root@raspberrypi:~# ip -details -statistics link show can0
3: can0: <NOARP,UP,LOWER_UP,ECHO> mtu 16 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 10
    link/can
    can <TRIPLE-SAMPLING> state ERROR-ACTIVE restart-ms 0
    bitrate 500000 sample-point 0.875
    tq 125 prop-seg 6 phase-seg1 7 phase-seg2 2 sjw 1
    mcp251x: tseg1 3..16 tseg2 2..8 sjw 1..4 brp 1..64 brp-inc 1
    clock 8000000
re-started bus-errors arbit-lost error-warn error-pass bus-off
     0          0          0          0          0          0     
RX: bytes  packets  errors  dropped overrun mcast
     632288     79036    0       103     0       0     
TX: bytes  packets  errors  dropped carrier collsns
     0          0        0       0       0       0
The couple of things I bolded are key. The clock should be 1/2 of what the oscillator setting in config.txt is (weird quirk). The state is the actual MCP2515 state – so error-active is what you usually want to see. Error-passive or Bus-off are bad. See the mcp2515 datasheet for details there.
You’ll want can-utils downloaded and installed: https://github.com/linux-can/can-utils
Then, if you run the Arduino project so it’s sending data every 500 ms, and then run this command on the RPi, you should see some similar output:
pi@raspberrypi ~ $ candump -t a -c -c -a can0
 (1452184714.419422)  can0  1FF   [8]  31 34 32 35 33 00 00 00   '14253...'
 (1452184714.939332)  can0  1FF   [8]  31 34 32 35 34 00 00 00   '14254...'
 (1452184715.459257)  can0  1FF   [8]  31 34 32 35 35 00 00 00   '14255...'
You can use the can-utils for just about everything. cansend and canplayer are useful for sending messages, candump and cansniffer are the two used for watching data (cansniffer is especially awesome for figuring out what’s really happening). If you want to write custom software that uses CAN on the RPi, then you can use these as examples. It’s effectively just using the socket system calls.
So that’s pretty much it. You have to have the bus speed match, you have to have the correct wiring, but otherwise that’s all there is to it. I’ve got a setup with an Arduino as shown below, the RPi as shown below, another RPi with a different CAN board (still mcp25xx), and a breadboard cased ATMEGA setup without the Arduino hardware. All works great.