My Zenoss Development Environment – Part 3

In Part 1 of this series we discussed getting an initial Zenoss environment checked out and running on a Mac OS X or Ubuntu system. In Part 2 we discussed how to configure Eclipse to use the Zenoss source. In this part, we’ll discuss how to handle day-to-day operations such as branch management and working with multiple versions.

Branch-Based Development

At Zenoss we do all development, including bug fixes and small maintenance tasks, in a private development branch within the subversion repository. This allows us to independently work-on changes, check-them into the repository for safe-keeping, and then perform code reviews with team members without having to share files or using pastebin style tools (even though we do both at times).

  1. Create a branch within your user’s sandbox. In this example, I’ve decided to name the sandbox new-widget to identify what I’m working on. If I were fixing a defect, I’d use the defect number from the defect tracking system. Create the branch by copying the trunk folder to the sandbox branch. In Subversion this is a low-overhead operation and doesn’t actually copy files.
    svn copy -m " * Copying trunk to sandbox branch."
  2. Switch your working directory to use the new sandbox branch. You can do this either from the command-line or using Eclipse. From the command line, you’d do the following:
    cd $HOME/zenoss/core
    svn switch
    From within Eclipse, secondary-click on the core project and choose Switch to Another Branch/Tag/Revision… option from the Team menu. On the dialog that appears, enter in the sandbox URL. After switching, your Eclipse will show the new location next to the core project item.
    svn switch

Once your development environment has been switched, you can make changes and commit to the Subversion repository as desired. If you’re unsure if you are in the right branch, you can always use the svn info command to see which directory is being used.

Merging Branches

Once you have completed changes in a branch and have had them reviewed by a peer, it is time to merge them into trunk (or another branch, if using a maintenance release). Merging can be tricky, but a consistent process can make it much easier to handle.

  1. Change your working directory to a checked-out and clean version of the branch you want to merge into. For example, I keep a $HOME/zenoss/clean-trunk directory that I never make changes to, except for merging.
  2. Determine the base working revision of your working branch. There are a variety of ways to do this, but one of the best is to view the revision log graph within the Trac system directly. For example, for the branch discussed above we can browse to and see that revision 15513 is the base.
  3. Perform a dry-run on the merge to get a general idea of what the changes into the branch will be. You should see your expected changes, plus any conflicts from changes to the other branch while you have been working in the sandbox branch.
    svn merge --dry-run --revision 15513:HEAD .
  4. If the merge results look satisfactory, rerun the command without the dry-run argument.
  5. Look at the final merge results using svn status and svn diff, and once you’re ready, issue an svn commit.

Multiple Branches and Zenoss Configuration

As you switch between branches you will often render your Zenoss configuration useless.  Resetting your database after each branch switch is usually a good practice, and being able to quickly recreate any test data you may need makes this process less painful.

After switching a branch, my process is usually the following:

  1. Shutdown zenoss and restart only zeo.
    zenoss stop
    zeoctl start
  2. Run the zenwipe script from the inst source directory.
    $HOME/zenoss/inst/ --no-prompt
  3. Run zenmigrate to install any database changes available within the current branch.

Depending upon the task at hand, I may install additional ZenPacks and add new devices through the command-line if those are needed. Helper scripts, such as to install of the Windows ZenPacks and create several local test devices in the instance, are useful tools to have for your typical configurations.

My Zenoss Development Environment – Part 2

In Part 1 of this series we discussed getting an initial Zenoss environment checked out and running on a Mac OS X or Ubuntu system. In this part, we’ll discuss configuring Eclipse to use this environment.

Mac OS X Prerequisites

  1. Install Eclipse Classic 3.5.x Mac Cocoa 32-bit.

Ubuntu Prerequisites

  1. Install the Sun Java JDK. Why? Eclipse is a Java application and requires a full-fledged Java environment to run properly.
    sudo apt-get -y install sun-java6-jdk
  2. Install Eclipse Classic 3.5.x Linux 32-bit.

Eclipse Configuration

  1. In part one, we created a Zenoss root project directory of $HOME/zenoss – use that directory, or the one you created, for the rest of these steps.
  2. Launch Eclipse and configure it to use the Zenoss root project directory. Why? Eclipse needs a workspace directory to keep track of configuration settings for a group of related projects. By placing the workspace directory inside of the Zenoss root workspace, we can separate the requirements for a Zenoss workspace from other projects you may be using.eclipse-workspace
  3. Install the PyDev plug-in for Eclipse. Why? PyDev provides Python language support to Eclipse.
    1. Go to the Help Menu and choose Install New Software.
    2. In the Available Software dialog, choose Add and enter in as the Location and close the dialog. Eclipse will return to the Available Software dialog.PyDev
    3. Software matching PyDev will appear in the dialog; pick the PyDev option but do not pick PyDev Mylyn Integration. Click Next and install the plug-in.


  4. Install the Subclipse plug-in. Why? Subclipse allows you to work with the Subversion version control system directly from within Eclipse.
    1. In the Available Software dialog add the Subclipse plug-in update site:
    2. In the Available Software dialog, choose the Subclipse, Subversion Client Adapter, Subversion JavaHL Native Library Adapter and the Subversion Revision Graph items.Subclipse
  5. Go to the Window menu, choose Open Perspective and then SVN Repository Exploring (you may have to choose Other… to see this option).
  6. Choose the New Repository Location button in the SVN Repositories panel. Add the Zenoss SVN site at
  7. Open the SVN repository and select the trunk folder. Secondary-click the folder and choose the Checkout… option. Be sure to change the Depth option to be Only this item so that we don’t check out any of the sub-folders of trunk just yet (many of the folders within trunk are not needed for development, but we want to keep the directory structure). In the next dialog you will be asked to choose a Project Wizard. Open the Pydev tree item and select the Pydev Project option.svn checkout
  8. Create the project in the Pydev Project dialog:
    1. Use core as the project name and use the default location. This should create a core project at $HOME/zenoss/core.
    2. Make sure the Python project type is selected.
    3. Select 2.4 as the Python Grammar version.
    4. Uncheck the Create default ‘src’ folder... option.
    5. Click the Click here to configure an interpreter not listed… option in order to add the python interpreter built by the Zenoss installation scripts.
      1. In the Preferences Dialog, choose the Interpreter – Python item underneath Pydev and select the New… button to add a new Python interpreter.
      2. Browse to the $ZENHOME/bin directory and choose the python2.4 executable from that directory.
      3. Name the interpreter python-2.4, zenoss-python or some other variant.
      4. Picture 13After the new Python interpreter has been added, return to the Pydev Project dialog and choose that interpreter from the list and then click Finish to create the project.

    6. Update the core folder from the command-line SVN client so you can selectively pull the sub-folders of the core project and not all of them:
      cd $HOME/zenoss/core
      svn update Products
    7. Move the Products directory the installation checked out and create a symbolic link to the one updated above. Why? This allows the Products source tree to be worked on from Eclipse and then also used by the Zenoss run-time.
      cd $ZENHOME
      mv Products Products.old
      ln -s $HOME/zenoss/core/Products Products
    8. Return to Eclipse and choose the Refresh option from the File menu so that Eclipse notices the updated directory and builds necessary dependencies.
    9. Secondary-click on the core project folder in Eclipse and choose Properties. Choose the PyDev – PYTHONPATH item and add source folders so PyDev can reference the project from within itself.
      1. In the Source Folders tab choose the Add source folder button and pick the root core folder from the provided list.Picture 2
      2. In the External Libraries tab choose the Add source folder button and choose the $ZENHOME/lib/python directory.Picture 3

    At this point, your Eclipse project should allow you to navigate between dependencies within the Zenoss project. You can also simultaneously switch between using the Team feature within Eclipse to update and manage Subversion or do so using the command-line svn client.


    In Part 3 of this series, we’ll discuss how to manage day-to-day activities such as creating sandbox branches for changes and dealing with multiple versions are done from within the same environment.

My Zenoss Development Environment – Part 1

Over the past 18 months the developers at Zenoss have used a variety of development environments and methods to productively work with Zenoss, but there are a lot of best practices that have emerged out of this diversity.

I develop Zenoss primarily on Mac OS X, with some work done on Ubuntu when necessary. I normally use Eclipse as my development editor, although I’ll often just use vim when doing quick changes or bug fixes. In either case, careful setup of the Zenoss environment is key to being able to productively work with both the new development version and with the shipping versions that require maintenance.

The environment you get by default when you check out the source version of Zenoss from the source repository, or from a source tarball, is not necessarily setup in the best way to develop productively using tools like Eclipse.

The rest of this post will document how both my Mac OS X and Ubuntu environments are initially configured so that a working source Zenoss installation is realized.

Mac OS X Prerequisites

These prerequisite instructions assume Mac OS X 10.5 Leopard; 10.6 Snow Leopard will not be able to compile Zenoss’s third-party dependencies, so an additional work-around is required for that platform until Zenoss moves to Python 2.6.

  1. Install Xcode. Why? Installs the GNU C/C++ compiler and other command-line development tools needed to build dependencies used by Zenoss.
  2. Install Universal Subversion 1.6.x from CollabNet Community.Why? Leopard only comes with Subversion 1.4.x. This version is not compatible with the Subclipse plug-in for Eclipse that will be used later. In order to be able to use both the command-line and Eclipse Subversion versions simultaneously on the same checked our source, the release of subversion should match. This installation will place the Subversion binaries in /opt/subversion and should automatically add it to your PATH.
  3. Install MySQL Community Edition 5.1 32-bit. Why? Zenoss needs MySQL for storage of event data.

Ubuntu Prerequisites

These prerequisite instructions assume Ubuntu 9.04 32-bit Desktop Edition. Installing the server edition or one with different options may require additional prerequisites to be installed.

  1. Install Subversion. Why? Working with the Zenoss product in source form requires Subversion to access the source repositories (it is also possible to build directly from a source tarball, but this is not discussed here).
    sudo apt-get -y install subversion
  2. Install MySQL. Why? Zenoss needs MySQL for storage of event data.
    sudo apt-get -y install mysql-client mysql-server libmysqlclient15-dev
  3. Install additional development environment tools. Why? Zenoss third-party dependencies require several binaries to be built from source.
    sudo apt-get -y install patch make vim gcc g++ autoconf
  4. Install SNMP support. Why? Zenoss requires SNMP libraries for monitoring, and having a local SNMP agent is useful for testing.
    sudo apt-get -y install libsnmp-base snmp snmpd
  5. Install Liberation TrueType fonts. Why? Graphs generated by RRDtool will not contain the correct glyphs without this font package.
    sudo apt-get -y install ttf-liberation

Environment Configuration

Configuring Eclipse will require determining where you want to work with your Zenoss installation, and installing Eclipse plug-ins to provide the features required for Python and Subversion support.

  1. Create your Zenoss root directory:
    mkdir $HOME/zenoss
  2. Create and run a script that will configure needed environment variables for zenoss. This script can be started from your log in profile if desired.
    cd $HOME/zenoss
    cat > <<EOF
    chmod +x
    mkdir $ZENHOME
  3. Checkout the Zenoss source installation tree. Why? This tree is used to bootstrap the installation from the source repository and create a running Zenoss installation. We’ll need this before we modify it to fit the needs of our development environment.
    svn co inst
  4. Run the installation script to checkout, compile, and configure a zenoss environment. If you need to customize your MySQL configuration at all do not use the --no-prompt argument.
    cd inst
    ./ --no-prompt
  5. Modify the zensocket file to be setuid root. Why? Some of the Zenoss daemons use a privileged port and making the file owned by root and accessible by the Zenoss user allows the daemons to be run as a non-priviledged user but still use the privileged port.
    sudo chown root:`id -g` $ZENHOME/bin/zensocket
    sudo chmod 04750 $ZENHOME/bin/zensocket

At this point, you should have running zeo and zope processes and be able to log on to the local Zenoss instance. Your Zenoss root directory should look similar to the following:
zenoss workspace


In Part 2 of this series, we’ll download and configure the Eclipse IDE.

Using Zenoss Core to Watch the Home Network

Zenoss is an open-source infrastructure management product. Normally used by institutions to watch their networking and server infrastructure, it also is used in smaller, less mission-critical scenarios. Scenarios like mine: I want to monitor my home & home office infrastructure since it has grown over time to contain a fair number of devices.

Without open-source alternatives like Zenoss, monitoring an extensive home or small office network is cost prohibitive so it often just isn’t done.

Since I also work for Zenoss, I’m often using various versions of Zenoss on my development systems to watch the home network. But these installations are not stable and long-running. One alternative I’ve considered is to install a new Virtual Machine on my home workstation that is a Zenoss appliance, but the issue here is that my home workstation is not always completely stable. I want my Zenoss install to be stable and relatively free from interference from my other activities. A dedicated server is the answer.

Normally I have enough spare computer parts in the closet that I could cobble together a working machine, but no such luck this time. Since I was building a server from scratch, I really did not want to go and build another large machine that was power-hungry, loud or took up a lot of space. Intel has come to the rescue with the new Atom-based systems that are very low power and use the mini-ITX form factor.

I wound up choosing the Intel D945GCLF2 main-board that comes with a dual-core Intel Atom 330 already installed. I configured it with 2GB of DDR2 667 memory. I picked an Apex MI-100 case with 250W power supply for it all to live in. Since this case is passively cool the only noise in the system is the small fan on the main-board’s memory controller (the CPU is passively cooled!) and any noise from the hard drive. I did have some spare hard drives laying around, so I picked an older Western Digital 150GB Raptor to help with system speed. Total out of pocket cost since I already had the drive? $171.22 shipped from newegg.


As you can see from the relative size of the hard drive the whole system is tiny. I don’t need an optical drive so if I wanted to put another 3.5″ hard drive in the case for mirroring purposes I could easily do so. Chances are better than I’ll eventually put a low-cost solid-state drive in the system as a 16 or 32 GB size would be plenty.


I put Ubuntu server on the system and used the network-based install to get it installed so I didn’t need to put an optical drive in the system. Rather than mess around with a network boot (which is great when it works, but I never have a working server setup to host the boot files) I just put the boot image on a USB memory stick and booted the core installation from there. The only issue here was a drive number reordering problem once the system was rebooted and the memory stick removed, but that was easily fixed.

Once running the system is effectively like any other x86 based Linux server. It’s not even slow, as the 1.6 GHz dual-core Atom supplies plenty of computing power. It’s clearly not as fast as current or previous generation desktop or server chips, but for building Zenoss I don’t find it any slower than the virtual machines in Zenoss’s development VMware farm. In many ways, I like having a slower system here to use because it helps keep me mindful of not ever user has plenty of computing power to throw at our software.

I decided to run Zenoss using a source-based install, but instead of running off the trunk I picked the 2.3.x branch to start with. This branch is from the version we just released which is proving to be our best release yet. I’ll switch to new releases using the source install but avoid the trunk given the large amount of turmoil that can occur there as new features are integrated into the product.

After getting Zenoss running I let it discover my home network. As expected, it found nearly all of the devices I have (13 out of the 16). There will be some configuration to do on some of the devices to enable true monitoring, but this is a great start.


net-snmp snmpd failure in CentOS 5

I was recently testing Zenoss on a CentOS 5 system when I discovered that the snmpd component of net-snmp would not start. There were no error messages in /var/log/snmpd.log so this made diagnosis a bit tricky given I had never used the tool before. 🙂

Running the daemon manually with snmpd -f showed the following error:

snmpd: symbol lookup error: snmpd: undefined symbol: smux_snmp_select_list_get_length

A quick Google search found the following bug based upon that error:

This bug indicates that the net-snmp-libs package was not being updated when the net-snmp package was updated using CentOS’s built-in update manager. A quick check validated this:

[root@cgibbons-dev CentOS]# rpm --query net-snmp
[root@cgibbons-dev CentOS]# rpm --query net-snmp-libs

And then a quick update of the net-snmp-libs package solved the issue:

[root@cgibbons-dev ~]# yum update net-snmp-libs
Loading "installonlyn" plugin
Setting up Update Process
Setting up repositories
base                      100% |=========================| 1.1 kB    00:00     
updates                   100% |=========================|  951 B    00:00     
addons                    100% |=========================|  951 B    00:00     
extras                    100% |=========================| 1.1 kB    00:00     
Reading repository metadata in from local files
Resolving Dependencies
--> Populating transaction set with selected packages. Please wait.
---> Downloading header for net-snmp-libs to pack into transaction set.
net-snmp-libs-5.3.1-19.el 100% |=========================|  27 kB    00:00     
---> Package net-snmp-libs.i386 1:5.3.1-19.el5_1.4 set to be updated
--> Running transaction check

Dependencies Resolved

 Package                 Arch       Version          Repository        Size 
 net-snmp-libs           i386       1:5.3.1-19.el5_1.4  updates           1.2 M

Transaction Summary
Install      0 Package(s)         
Update       1 Package(s)         
Remove       0 Package(s)         

Total download size: 1.2 M
Is this ok [y/N]: Y
Downloading Packages:
(1/1): net-snmp-libs-5.3. 100% |=========================| 1.2 MB    00:00     
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating  : net-snmp-libs                ######################### [1/2] 
  Cleanup   : net-snmp-libs                ######################### [2/2]

Updated: net-snmp-libs.i386 1:5.3.1-19.el5_1.4