Case Study: Micro-services using Erlang

At the Erlang User Conference 2017 (EUC17) in Stockholm Sweden, I presented our experience of how Alert Logic moved to a new micro-services architecture using our preferred programming language: Erlang.

Alert Logic had been using Erlang for years before, but we really did not have a holistic platform for building applications. A small team of us went off and build a new product for the company, and in doing so build a new application platform using micro-services patterns and written in the Erlang programming language.

You can watch a recording of the presentation below, as well as review the slide deck.


You Don’t Need Flash for Rich Graphs on a Web Page

Adobe’s Flash has a lot of uses, but one of the most impressive to me has been the the creation of interactive graphs on a web page. One just has to visit Google Finance to see a great example of this in action; it’s fast, effective and fits seemlessly within the rest of the page.

Many times, however, Flash isn’t an appropriate technology to use. If you’re an open-source product like Zenoss, Flash presents a licensing issue. If you’re targeting mobile platforms like the iPhone, Flash isn’t available. And, sometimes, you may just not like Flash; it does have it’s own security problems and overhead, for example. What’s a web developer to do? Enter HTML5 to the rescue…

HTML5 is not yet an approved standard, but it’s well on its way and surprisingly well-supported by the browser community already. One of the nice new features provided in HTML5 is the canvas element which allows for two-dimensional drawing functionality. Between this new feature and JavaScript, we should be able to create a rich graph display.

My Zenoss Development Environment – Part 3

In Part 1 of this series we discussed getting an initial Zenoss environment checked out and running on a Mac OS X or Ubuntu system. In Part 2 we discussed how to configure Eclipse to use the Zenoss source. In this part, we’ll discuss how to handle day-to-day operations such as branch management and working with multiple versions.

Branch-Based Development

At Zenoss we do all development, including bug fixes and small maintenance tasks, in a private development branch within the subversion repository. This allows us to independently work-on changes, check-them into the repository for safe-keeping, and then perform code reviews with team members without having to share files or using pastebin style tools (even though we do both at times).

  1. Create a branch within your user’s sandbox. In this example, I’ve decided to name the sandbox new-widget to identify what I’m working on. If I were fixing a defect, I’d use the defect number from the defect tracking system. Create the branch by copying the trunk folder to the sandbox branch. In Subversion this is a low-overhead operation and doesn’t actually copy files.
    svn copy -m " * Copying trunk to sandbox branch."
  2. Switch your working directory to use the new sandbox branch. You can do this either from the command-line or using Eclipse. From the command line, you’d do the following:
    cd $HOME/zenoss/core
    svn switch
    From within Eclipse, secondary-click on the core project and choose Switch to Another Branch/Tag/Revision… option from the Team menu. On the dialog that appears, enter in the sandbox URL. After switching, your Eclipse will show the new location next to the core project item.
    svn switch

Once your development environment has been switched, you can make changes and commit to the Subversion repository as desired. If you’re unsure if you are in the right branch, you can always use the svn info command to see which directory is being used.

Merging Branches

Once you have completed changes in a branch and have had them reviewed by a peer, it is time to merge them into trunk (or another branch, if using a maintenance release). Merging can be tricky, but a consistent process can make it much easier to handle.

  1. Change your working directory to a checked-out and clean version of the branch you want to merge into. For example, I keep a $HOME/zenoss/clean-trunk directory that I never make changes to, except for merging.
  2. Determine the base working revision of your working branch. There are a variety of ways to do this, but one of the best is to view the revision log graph within the Trac system directly. For example, for the branch discussed above we can browse to and see that revision 15513 is the base.
  3. Perform a dry-run on the merge to get a general idea of what the changes into the branch will be. You should see your expected changes, plus any conflicts from changes to the other branch while you have been working in the sandbox branch.
    svn merge --dry-run --revision 15513:HEAD .
  4. If the merge results look satisfactory, rerun the command without the dry-run argument.
  5. Look at the final merge results using svn status and svn diff, and once you’re ready, issue an svn commit.

Multiple Branches and Zenoss Configuration

As you switch between branches you will often render your Zenoss configuration useless.  Resetting your database after each branch switch is usually a good practice, and being able to quickly recreate any test data you may need makes this process less painful.

After switching a branch, my process is usually the following:

  1. Shutdown zenoss and restart only zeo.
    zenoss stop
    zeoctl start
  2. Run the zenwipe script from the inst source directory.
    $HOME/zenoss/inst/ --no-prompt
  3. Run zenmigrate to install any database changes available within the current branch.

Depending upon the task at hand, I may install additional ZenPacks and add new devices through the command-line if those are needed. Helper scripts, such as to install of the Windows ZenPacks and create several local test devices in the instance, are useful tools to have for your typical configurations.

My Zenoss Development Environment – Part 2

In Part 1 of this series we discussed getting an initial Zenoss environment checked out and running on a Mac OS X or Ubuntu system. In this part, we’ll discuss configuring Eclipse to use this environment.

Mac OS X Prerequisites

  1. Install Eclipse Classic 3.5.x Mac Cocoa 32-bit.

Ubuntu Prerequisites

  1. Install the Sun Java JDK. Why? Eclipse is a Java application and requires a full-fledged Java environment to run properly.
    sudo apt-get -y install sun-java6-jdk
  2. Install Eclipse Classic 3.5.x Linux 32-bit.

Eclipse Configuration

  1. In part one, we created a Zenoss root project directory of $HOME/zenoss – use that directory, or the one you created, for the rest of these steps.
  2. Launch Eclipse and configure it to use the Zenoss root project directory. Why? Eclipse needs a workspace directory to keep track of configuration settings for a group of related projects. By placing the workspace directory inside of the Zenoss root workspace, we can separate the requirements for a Zenoss workspace from other projects you may be using.eclipse-workspace
  3. Install the PyDev plug-in for Eclipse. Why? PyDev provides Python language support to Eclipse.
    1. Go to the Help Menu and choose Install New Software.
    2. In the Available Software dialog, choose Add and enter in as the Location and close the dialog. Eclipse will return to the Available Software dialog.PyDev
    3. Software matching PyDev will appear in the dialog; pick the PyDev option but do not pick PyDev Mylyn Integration. Click Next and install the plug-in.


  4. Install the Subclipse plug-in. Why? Subclipse allows you to work with the Subversion version control system directly from within Eclipse.
    1. In the Available Software dialog add the Subclipse plug-in update site:
    2. In the Available Software dialog, choose the Subclipse, Subversion Client Adapter, Subversion JavaHL Native Library Adapter and the Subversion Revision Graph items.Subclipse
  5. Go to the Window menu, choose Open Perspective and then SVN Repository Exploring (you may have to choose Other… to see this option).
  6. Choose the New Repository Location button in the SVN Repositories panel. Add the Zenoss SVN site at
  7. Open the SVN repository and select the trunk folder. Secondary-click the folder and choose the Checkout… option. Be sure to change the Depth option to be Only this item so that we don’t check out any of the sub-folders of trunk just yet (many of the folders within trunk are not needed for development, but we want to keep the directory structure). In the next dialog you will be asked to choose a Project Wizard. Open the Pydev tree item and select the Pydev Project option.svn checkout
  8. Create the project in the Pydev Project dialog:
    1. Use core as the project name and use the default location. This should create a core project at $HOME/zenoss/core.
    2. Make sure the Python project type is selected.
    3. Select 2.4 as the Python Grammar version.
    4. Uncheck the Create default ‘src’ folder... option.
    5. Click the Click here to configure an interpreter not listed… option in order to add the python interpreter built by the Zenoss installation scripts.
      1. In the Preferences Dialog, choose the Interpreter – Python item underneath Pydev and select the New… button to add a new Python interpreter.
      2. Browse to the $ZENHOME/bin directory and choose the python2.4 executable from that directory.
      3. Name the interpreter python-2.4, zenoss-python or some other variant.
      4. Picture 13After the new Python interpreter has been added, return to the Pydev Project dialog and choose that interpreter from the list and then click Finish to create the project.

    6. Update the core folder from the command-line SVN client so you can selectively pull the sub-folders of the core project and not all of them:
      cd $HOME/zenoss/core
      svn update Products
    7. Move the Products directory the installation checked out and create a symbolic link to the one updated above. Why? This allows the Products source tree to be worked on from Eclipse and then also used by the Zenoss run-time.
      cd $ZENHOME
      mv Products Products.old
      ln -s $HOME/zenoss/core/Products Products
    8. Return to Eclipse and choose the Refresh option from the File menu so that Eclipse notices the updated directory and builds necessary dependencies.
    9. Secondary-click on the core project folder in Eclipse and choose Properties. Choose the PyDev – PYTHONPATH item and add source folders so PyDev can reference the project from within itself.
      1. In the Source Folders tab choose the Add source folder button and pick the root core folder from the provided list.Picture 2
      2. In the External Libraries tab choose the Add source folder button and choose the $ZENHOME/lib/python directory.Picture 3

    At this point, your Eclipse project should allow you to navigate between dependencies within the Zenoss project. You can also simultaneously switch between using the Team feature within Eclipse to update and manage Subversion or do so using the command-line svn client.


    In Part 3 of this series, we’ll discuss how to manage day-to-day activities such as creating sandbox branches for changes and dealing with multiple versions are done from within the same environment.

My Zenoss Development Environment – Part 1

Over the past 18 months the developers at Zenoss have used a variety of development environments and methods to productively work with Zenoss, but there are a lot of best practices that have emerged out of this diversity.

I develop Zenoss primarily on Mac OS X, with some work done on Ubuntu when necessary. I normally use Eclipse as my development editor, although I’ll often just use vim when doing quick changes or bug fixes. In either case, careful setup of the Zenoss environment is key to being able to productively work with both the new development version and with the shipping versions that require maintenance.

The environment you get by default when you check out the source version of Zenoss from the source repository, or from a source tarball, is not necessarily setup in the best way to develop productively using tools like Eclipse.

The rest of this post will document how both my Mac OS X and Ubuntu environments are initially configured so that a working source Zenoss installation is realized.

Mac OS X Prerequisites

These prerequisite instructions assume Mac OS X 10.5 Leopard; 10.6 Snow Leopard will not be able to compile Zenoss’s third-party dependencies, so an additional work-around is required for that platform until Zenoss moves to Python 2.6.

  1. Install Xcode. Why? Installs the GNU C/C++ compiler and other command-line development tools needed to build dependencies used by Zenoss.
  2. Install Universal Subversion 1.6.x from CollabNet Community.Why? Leopard only comes with Subversion 1.4.x. This version is not compatible with the Subclipse plug-in for Eclipse that will be used later. In order to be able to use both the command-line and Eclipse Subversion versions simultaneously on the same checked our source, the release of subversion should match. This installation will place the Subversion binaries in /opt/subversion and should automatically add it to your PATH.
  3. Install MySQL Community Edition 5.1 32-bit. Why? Zenoss needs MySQL for storage of event data.

Ubuntu Prerequisites

These prerequisite instructions assume Ubuntu 9.04 32-bit Desktop Edition. Installing the server edition or one with different options may require additional prerequisites to be installed.

  1. Install Subversion. Why? Working with the Zenoss product in source form requires Subversion to access the source repositories (it is also possible to build directly from a source tarball, but this is not discussed here).
    sudo apt-get -y install subversion
  2. Install MySQL. Why? Zenoss needs MySQL for storage of event data.
    sudo apt-get -y install mysql-client mysql-server libmysqlclient15-dev
  3. Install additional development environment tools. Why? Zenoss third-party dependencies require several binaries to be built from source.
    sudo apt-get -y install patch make vim gcc g++ autoconf
  4. Install SNMP support. Why? Zenoss requires SNMP libraries for monitoring, and having a local SNMP agent is useful for testing.
    sudo apt-get -y install libsnmp-base snmp snmpd
  5. Install Liberation TrueType fonts. Why? Graphs generated by RRDtool will not contain the correct glyphs without this font package.
    sudo apt-get -y install ttf-liberation

Environment Configuration

Configuring Eclipse will require determining where you want to work with your Zenoss installation, and installing Eclipse plug-ins to provide the features required for Python and Subversion support.

  1. Create your Zenoss root directory:
    mkdir $HOME/zenoss
  2. Create and run a script that will configure needed environment variables for zenoss. This script can be started from your log in profile if desired.
    cd $HOME/zenoss
    cat > <<EOF
    chmod +x
    mkdir $ZENHOME
  3. Checkout the Zenoss source installation tree. Why? This tree is used to bootstrap the installation from the source repository and create a running Zenoss installation. We’ll need this before we modify it to fit the needs of our development environment.
    svn co inst
  4. Run the installation script to checkout, compile, and configure a zenoss environment. If you need to customize your MySQL configuration at all do not use the --no-prompt argument.
    cd inst
    ./ --no-prompt
  5. Modify the zensocket file to be setuid root. Why? Some of the Zenoss daemons use a privileged port and making the file owned by root and accessible by the Zenoss user allows the daemons to be run as a non-priviledged user but still use the privileged port.
    sudo chown root:`id -g` $ZENHOME/bin/zensocket
    sudo chmod 04750 $ZENHOME/bin/zensocket

At this point, you should have running zeo and zope processes and be able to log on to the local Zenoss instance. Your Zenoss root directory should look similar to the following:
zenoss workspace


In Part 2 of this series, we’ll download and configure the Eclipse IDE.