My Zenoss Development Environment – Part 3

In Part 1 of this series we discussed getting an initial Zenoss environment checked out and running on a Mac OS X or Ubuntu system. In Part 2 we discussed how to configure Eclipse to use the Zenoss source. In this part, we’ll discuss how to handle day-to-day operations such as branch management and working with multiple versions.

Branch-Based Development

At Zenoss we do all development, including bug fixes and small maintenance tasks, in a private development branch within the subversion repository. This allows us to independently work-on changes, check-them into the repository for safe-keeping, and then perform code reviews with team members without having to share files or using pastebin style tools (even though we do both at times).

  1. Create a branch within your user’s sandbox. In this example, I’ve decided to name the sandbox new-widget to identify what I’m working on. If I were fixing a defect, I’d use the defect number from the defect tracking system. Create the branch by copying the trunk folder to the sandbox branch. In Subversion this is a low-overhead operation and doesn’t actually copy files.
    svn copy -m " * Copying trunk to sandbox branch."
  2. Switch your working directory to use the new sandbox branch. You can do this either from the command-line or using Eclipse. From the command line, you’d do the following:
    cd $HOME/zenoss/core
    svn switch
    From within Eclipse, secondary-click on the core project and choose Switch to Another Branch/Tag/Revision… option from the Team menu. On the dialog that appears, enter in the sandbox URL. After switching, your Eclipse will show the new location next to the core project item.
    svn switch

Once your development environment has been switched, you can make changes and commit to the Subversion repository as desired. If you’re unsure if you are in the right branch, you can always use the svn info command to see which directory is being used.

Merging Branches

Once you have completed changes in a branch and have had them reviewed by a peer, it is time to merge them into trunk (or another branch, if using a maintenance release). Merging can be tricky, but a consistent process can make it much easier to handle.

  1. Change your working directory to a checked-out and clean version of the branch you want to merge into. For example, I keep a $HOME/zenoss/clean-trunk directory that I never make changes to, except for merging.
  2. Determine the base working revision of your working branch. There are a variety of ways to do this, but one of the best is to view the revision log graph within the Trac system directly. For example, for the branch discussed above we can browse to and see that revision 15513 is the base.
  3. Perform a dry-run on the merge to get a general idea of what the changes into the branch will be. You should see your expected changes, plus any conflicts from changes to the other branch while you have been working in the sandbox branch.
    svn merge --dry-run --revision 15513:HEAD .
  4. If the merge results look satisfactory, rerun the command without the dry-run argument.
  5. Look at the final merge results using svn status and svn diff, and once you’re ready, issue an svn commit.

Multiple Branches and Zenoss Configuration

As you switch between branches you will often render your Zenoss configuration useless.  Resetting your database after each branch switch is usually a good practice, and being able to quickly recreate any test data you may need makes this process less painful.

After switching a branch, my process is usually the following:

  1. Shutdown zenoss and restart only zeo.
    zenoss stop
    zeoctl start
  2. Run the zenwipe script from the inst source directory.
    $HOME/zenoss/inst/ --no-prompt
  3. Run zenmigrate to install any database changes available within the current branch.

Depending upon the task at hand, I may install additional ZenPacks and add new devices through the command-line if those are needed. Helper scripts, such as to install of the Windows ZenPacks and create several local test devices in the instance, are useful tools to have for your typical configurations.

My Zenoss Development Environment – Part 2

In Part 1 of this series we discussed getting an initial Zenoss environment checked out and running on a Mac OS X or Ubuntu system. In this part, we’ll discuss configuring Eclipse to use this environment.

Mac OS X Prerequisites

  1. Install Eclipse Classic 3.5.x Mac Cocoa 32-bit.

Ubuntu Prerequisites

  1. Install the Sun Java JDK. Why? Eclipse is a Java application and requires a full-fledged Java environment to run properly.
    sudo apt-get -y install sun-java6-jdk
  2. Install Eclipse Classic 3.5.x Linux 32-bit.

Eclipse Configuration

  1. In part one, we created a Zenoss root project directory of $HOME/zenoss – use that directory, or the one you created, for the rest of these steps.
  2. Launch Eclipse and configure it to use the Zenoss root project directory. Why? Eclipse needs a workspace directory to keep track of configuration settings for a group of related projects. By placing the workspace directory inside of the Zenoss root workspace, we can separate the requirements for a Zenoss workspace from other projects you may be using.eclipse-workspace
  3. Install the PyDev plug-in for Eclipse. Why? PyDev provides Python language support to Eclipse.
    1. Go to the Help Menu and choose Install New Software.
    2. In the Available Software dialog, choose Add and enter in as the Location and close the dialog. Eclipse will return to the Available Software dialog.PyDev
    3. Software matching PyDev will appear in the dialog; pick the PyDev option but do not pick PyDev Mylyn Integration. Click Next and install the plug-in.


  4. Install the Subclipse plug-in. Why? Subclipse allows you to work with the Subversion version control system directly from within Eclipse.
    1. In the Available Software dialog add the Subclipse plug-in update site:
    2. In the Available Software dialog, choose the Subclipse, Subversion Client Adapter, Subversion JavaHL Native Library Adapter and the Subversion Revision Graph items.Subclipse
  5. Go to the Window menu, choose Open Perspective and then SVN Repository Exploring (you may have to choose Other… to see this option).
  6. Choose the New Repository Location button in the SVN Repositories panel. Add the Zenoss SVN site at
  7. Open the SVN repository and select the trunk folder. Secondary-click the folder and choose the Checkout… option. Be sure to change the Depth option to be Only this item so that we don’t check out any of the sub-folders of trunk just yet (many of the folders within trunk are not needed for development, but we want to keep the directory structure). In the next dialog you will be asked to choose a Project Wizard. Open the Pydev tree item and select the Pydev Project option.svn checkout
  8. Create the project in the Pydev Project dialog:
    1. Use core as the project name and use the default location. This should create a core project at $HOME/zenoss/core.
    2. Make sure the Python project type is selected.
    3. Select 2.4 as the Python Grammar version.
    4. Uncheck the Create default ‘src’ folder... option.
    5. Click the Click here to configure an interpreter not listed… option in order to add the python interpreter built by the Zenoss installation scripts.
      1. In the Preferences Dialog, choose the Interpreter – Python item underneath Pydev and select the New… button to add a new Python interpreter.
      2. Browse to the $ZENHOME/bin directory and choose the python2.4 executable from that directory.
      3. Name the interpreter python-2.4, zenoss-python or some other variant.
      4. Picture 13After the new Python interpreter has been added, return to the Pydev Project dialog and choose that interpreter from the list and then click Finish to create the project.

    6. Update the core folder from the command-line SVN client so you can selectively pull the sub-folders of the core project and not all of them:
      cd $HOME/zenoss/core
      svn update Products
    7. Move the Products directory the installation checked out and create a symbolic link to the one updated above. Why? This allows the Products source tree to be worked on from Eclipse and then also used by the Zenoss run-time.
      cd $ZENHOME
      mv Products Products.old
      ln -s $HOME/zenoss/core/Products Products
    8. Return to Eclipse and choose the Refresh option from the File menu so that Eclipse notices the updated directory and builds necessary dependencies.
    9. Secondary-click on the core project folder in Eclipse and choose Properties. Choose the PyDev – PYTHONPATH item and add source folders so PyDev can reference the project from within itself.
      1. In the Source Folders tab choose the Add source folder button and pick the root core folder from the provided list.Picture 2
      2. In the External Libraries tab choose the Add source folder button and choose the $ZENHOME/lib/python directory.Picture 3

    At this point, your Eclipse project should allow you to navigate between dependencies within the Zenoss project. You can also simultaneously switch between using the Team feature within Eclipse to update and manage Subversion or do so using the command-line svn client.


    In Part 3 of this series, we’ll discuss how to manage day-to-day activities such as creating sandbox branches for changes and dealing with multiple versions are done from within the same environment.

My Zenoss Development Environment – Part 1

Over the past 18 months the developers at Zenoss have used a variety of development environments and methods to productively work with Zenoss, but there are a lot of best practices that have emerged out of this diversity.

I develop Zenoss primarily on Mac OS X, with some work done on Ubuntu when necessary. I normally use Eclipse as my development editor, although I’ll often just use vim when doing quick changes or bug fixes. In either case, careful setup of the Zenoss environment is key to being able to productively work with both the new development version and with the shipping versions that require maintenance.

The environment you get by default when you check out the source version of Zenoss from the source repository, or from a source tarball, is not necessarily setup in the best way to develop productively using tools like Eclipse.

The rest of this post will document how both my Mac OS X and Ubuntu environments are initially configured so that a working source Zenoss installation is realized.

Mac OS X Prerequisites

These prerequisite instructions assume Mac OS X 10.5 Leopard; 10.6 Snow Leopard will not be able to compile Zenoss’s third-party dependencies, so an additional work-around is required for that platform until Zenoss moves to Python 2.6.

  1. Install Xcode. Why? Installs the GNU C/C++ compiler and other command-line development tools needed to build dependencies used by Zenoss.
  2. Install Universal Subversion 1.6.x from CollabNet Community.Why? Leopard only comes with Subversion 1.4.x. This version is not compatible with the Subclipse plug-in for Eclipse that will be used later. In order to be able to use both the command-line and Eclipse Subversion versions simultaneously on the same checked our source, the release of subversion should match. This installation will place the Subversion binaries in /opt/subversion and should automatically add it to your PATH.
  3. Install MySQL Community Edition 5.1 32-bit. Why? Zenoss needs MySQL for storage of event data.

Ubuntu Prerequisites

These prerequisite instructions assume Ubuntu 9.04 32-bit Desktop Edition. Installing the server edition or one with different options may require additional prerequisites to be installed.

  1. Install Subversion. Why? Working with the Zenoss product in source form requires Subversion to access the source repositories (it is also possible to build directly from a source tarball, but this is not discussed here).
    sudo apt-get -y install subversion
  2. Install MySQL. Why? Zenoss needs MySQL for storage of event data.
    sudo apt-get -y install mysql-client mysql-server libmysqlclient15-dev
  3. Install additional development environment tools. Why? Zenoss third-party dependencies require several binaries to be built from source.
    sudo apt-get -y install patch make vim gcc g++ autoconf
  4. Install SNMP support. Why? Zenoss requires SNMP libraries for monitoring, and having a local SNMP agent is useful for testing.
    sudo apt-get -y install libsnmp-base snmp snmpd
  5. Install Liberation TrueType fonts. Why? Graphs generated by RRDtool will not contain the correct glyphs without this font package.
    sudo apt-get -y install ttf-liberation

Environment Configuration

Configuring Eclipse will require determining where you want to work with your Zenoss installation, and installing Eclipse plug-ins to provide the features required for Python and Subversion support.

  1. Create your Zenoss root directory:
    mkdir $HOME/zenoss
  2. Create and run a script that will configure needed environment variables for zenoss. This script can be started from your log in profile if desired.
    cd $HOME/zenoss
    cat > <<EOF
    chmod +x
    mkdir $ZENHOME
  3. Checkout the Zenoss source installation tree. Why? This tree is used to bootstrap the installation from the source repository and create a running Zenoss installation. We’ll need this before we modify it to fit the needs of our development environment.
    svn co inst
  4. Run the installation script to checkout, compile, and configure a zenoss environment. If you need to customize your MySQL configuration at all do not use the --no-prompt argument.
    cd inst
    ./ --no-prompt
  5. Modify the zensocket file to be setuid root. Why? Some of the Zenoss daemons use a privileged port and making the file owned by root and accessible by the Zenoss user allows the daemons to be run as a non-priviledged user but still use the privileged port.
    sudo chown root:`id -g` $ZENHOME/bin/zensocket
    sudo chmod 04750 $ZENHOME/bin/zensocket

At this point, you should have running zeo and zope processes and be able to log on to the local Zenoss instance. Your Zenoss root directory should look similar to the following:
zenoss workspace


In Part 2 of this series, we’ll download and configure the Eclipse IDE.

Connecting to Apple Push Notification Services using Python & Twisted

One of the exciting features of the upcoming iPhone 3.0 operating system is the ability for a device to receive a notification message from your own application that is relayed by Apple’s network cloud. This feature allows the iPhone to channel all out-of-band communication into a narrow channel that is low impact on the device from a power, security and usability point-of-view.

As an application developer, you are responsible for two pieces to make this new notification service work: the actual application that runs on the iPhone OS, and a provider service that both communicates with the device and directly to Apple’s cloud.

In a typical flow, the application on an iPhone device issues an API call to request notifications for the identifier that the developer has assigned to the application. In response to that API call, Apple’s servers provide the device with a binary token value that uniquely identifies the device & installation combination (for security reasons, the device token is reset if the device is reset, so it is not a one-to-one mapping to the actual hardware). Next, the application must connect to the developer’s provider servers and provide it with the device token it has received from apple. The provider can now use the device token when communicating back to the Apple Push Notification Service (APNS) cloud.

A provider implementation has very little work to do to inform APNS of a device update. Most of the effort in a provider will be in the actual framework of the application itself, and managing what information should be sent to a device. A common implementation will be for a provider to both communicate with the APNS server cloud and also to the application’s own network services and effectively become a protocol translator. Such a heavily network based application like that is a perfect match for the Python language and the Twisted framework.

A provider to APNS link is very simple: a secure-socket connection (SSL) is made to a APNS server and then simple messages are sent by the provider whenever a notification needs to be sent to the device. The APNS server does not respond or acknowledge the message. Every so often (Apple recommends once an hour), the provider server should connect to a web server that will dump a list of device tokens that are no longer valid (as in the device has uninstalled the application). This is the provider’s only feedback mechanism outside of communicating directly with a device.

The first step is to setup an SSL connection to the APNS servers. By following the directions on Apple’s developer portal, you can create a certificate and private key that are assigned to your application identifier. This certificate & private key must then be used when connecting to the APNS servers as an authentication mechanism.

One caveat  – the Mac OS X Keychain Access application does not directly export certificates and private keys in Private Enhanced Mail (.pem)  format, which is what the OpenSSL implementation we use with Twisted will want, but luckily there’s an easy mechanism to convert if you export the files as Personal Information Exchange (.p12) format. The following two commands can be used to convert the .p12 files into .pem files using the built-in openssl command on Mac OS X or most Linux distributions:

openssl pkcs12 -in cred.p12 -out certkey.pem -nodes -clcerts
openssl pkcs12 -in pkey.p12 -out pkey.pem -nodes -clcerts

Now that we have our SSL files in the proper format, we can build our ClientContextFactory object that will be used by Twisted to initiate a connection to the APNS servers:

class APNSClientContextFactory(ClientContextFactory):
    def __init__(self):
        self.ctx = SSL.Context(SSL.SSLv3_METHOD)
def getContext(self):
    return self.ctx

We’ll also need a ClientFactory object that will be used to build the Protocol object whenever a connection is established to a server:

class APNSClientFactory(ClientFactory):
    def buildProtocol(self, addr):
        print "Connected to APNS Server %s:%u" % (, addr.port)
        return APNSProtocol()
    def clientConnectionLost(self, connector, reason):
        print "Lost connection. Reason: %s" % reason
    def clientConnectionFailed(self, connector, reason):
        print "Connection failed. Reason: %s" % reason

Next, we’ll need the actual Protocol object that will format messages and queue them to the server. For APNS, we don’t receive any responses so that half of the protocol goes unused.

class APNSProtocol(Protocol):
    def sendMessage(self, deviceToken, payload):
        # notification messages are binary messages in network order
        # using the following format:
        # &lt;1 byte command&gt; &lt;2 bytes length&gt; &lt;2 bytes length&gt;
        fmt = "!cH32sH%ds" % len(payload)
        command = '\x00'
        msg = struct.pack(fmt, command, 32, deviceToken,
                          len(payload), payload)

And finally, we’ll need the main code to kick everything off for the test:

if __name__ == '__main__':

From here, the Python and Twisted combination can be easily extended to handle multiple protocols and convert between them as needed. For example, one of the commonly expected use cases of the new push feature will be instant messaging (IM) clients that will need to provide a proxy between the device and the IM servers and the APNS servers in order to notify a device of outstanding messages.

The full Python script containing the above excerpts can be downloaded here:


One Year at Zenoss

Yesterday marked my one-year anniversary at Zenoss. When I decided to take the job at Zenoss I knew it would be a challenging position, and time has proven this assumption correct. The new (to me, anyway) technology, the remote work environment and the realities of a startup company with limited resources have all added their own challenges to the job.

I found the Zenoss product itself to be more mature than I had expected. While it still has plenty of rough edges, especially in the user interface, it is a rather feature-rich application with a very large user base in both the free community and paid enterprise versions. There is a tremendous amount of work left to do on the application, but with each version we see a significant step forward and an expanded user interest because of the changes.

When I came to Zenoss, my new co-workers and I brought with us a culture of using agile development methodologies and processes to the company. We expected, and were asked by company management, to implement those processes at Zenoss. Over the past year, we have seen the right things happen from this shift in engineering culture:

  • development time has become more predictable
  • planning and priority selection take a much more important role than ever before, and not just in engineering (in fact, most importantly not in engineering, but at the executive level)
  • quality assurance is now a fundamental part of the engineering process, not a luxury for later
  • code and design quality has improved dramatically as the engineering team begins to understand how much control they really do have over the process

There still remain many challenges, but these are the same ones I have seen with all teams that struggle with the discipline needed for agile methodologies. Effectively breaking down large development tasks into two-week iterations remains a hurdle; I suspect at a fundamental level many of the developers do not fully believe this is possible, which is a long, long discussion. There was also a reluctance, until recently, to fully engage with teammates on performing effective code and design reviews. These will both improve over time as the team matures.

When I first started at Zenoss there were 5 developers in Annapolis, 2 developers in Austin and me , usually working remote in Houston. Since then, the team dynamics have changed so that I and one other developer are remote most of the time, 3 developers are full-time in Austin and 1 in Annapolis. This team dynamic change has made the infrastructure needed to support remote development much more important, but at the same time it has made the remote developers somewhat more isolated since a bit more now happens only in one location. Since I travel to Austin frequently I can avoid the worst of this unavoidable trend, but it will likely always remain a challenge as long as we have remote employees.

On a personal level, working remotely full-time for this long has been much more difficult than I expected. I last did this much remote work in the mid-90s, and it was difficult to do then, as well. On the surface, I spend all of my work hours focused on some problem on the computer, but in reality much of my job is very social and without face time it starts to wear on me very quickly. Frequent visits to Austin help tremendously, but in reality this will likely always be an issue until I relocate.

I’ve encountered almost all of the traditional problems people have while working at home, but especially keeping focus and balancing the time spent between work and home life. Focus is purely a self-discipline issue, but it is remarkable how much of this is provided by the routine of commuting to an office. Maintaining a good work-life balance is also tremendously hard when working at home, unless you dedicate a specific place in the house for work. It took a few months to really learn this lesson but eventually I bought a second desk and made myself an office separate from my normal home office. Before then, I found it way too easy to bring my laptop into the living room and I’d continue to check e-mail and work on problems throughout the evening. Sometimes, I still do, but not at least it is not a mindless occurrence.

Zenoss is a very small company, and naturally we have very limited resources. Of course, today’s economic environment makes this true for almost every single company. But more specifically, I find ourselves working around what effectively is a lack of resources in our IT department. I often have to build my own virtual machines at home, or buy little widgets to help productivity because I am remote. At some level, I don’t mind this, since it’s nice being able to actually work somewhere you can do this, rather than being told you can’t.

After a year, I’m still not a huge fan of the Python programming language, but I’ve come to accept it and I can be productive with it. Working with it feels a whole lot like I stepped back in time 15 years, and saying that is a great way to start an argument, so I’ll just leave it at that.

One great surprise from the past year is that I became the guru for Zenoss’s Windows monitoring technology. We have a custom implementation of MS-RPC that is based upon the Samba 4 source tree. On top of this layer, we have provided our own implementations of the DCOM, WMI, and Windows registry interfaces so we can call these services directly on any Windows device. This is a great feature, as it allows us to communicate directly to Windows devices from our Linux based product, without requiring a Windows device to be running our software just to provide Windows-based connectivity, unlike some other systems management products.

Over the next few months I will be taking on more and more of the implementation of the new Zenoss user interface, which is a great project that should dramatically improve the user acceptance of quality of the application. It also gives me more time to work on user interfaces, which is always an area of software development I enjoy but rarely get a chance to do.