everal years ago I evaluated CiscoWorks as a network management platform, and was impressed. CiscoWorks is rife with features. In particular it had a few key features that I found to be immensely useful. Unfortunately, CiscoWorks is rather expensive, so I ended up “rolling my own” similar features. One key CiscoWorks feature is the capability of storing Cisco configurations in a versioning-system. This feature lets you save your configurations and compare current configurations to previous ones. CiscoWorks also uses SNMP traps, or system-generated message, so when someone makes a configuration change to a router, the system grabs and stores the new configuration, providing an automated change-tracking system.

I decided to use Subversion and some simple scripts to mimic the CiscoWorks functionality. Subversion is a control versioning system that aims to replace CVS. It’s primarily HTTP-based, which makes it a convenient platform for storing configurations that are accessible from any Web browser. I have an additional Web-based interface, called ViewCVS on my Cisco configuration repository, making it easy to view running router configurations while in a meeting, and useful for simply evaluating what has changed on my network recently. But an additional advantage to this type of storage is that you can automatically create up-to-date configurations for disaster recovery scenarios.

Software Requirements
This article focuses primarily on setting up the configuration repository on a Unix-like system. You could most certainly follow a similar process to set the repository up on Windows, although the examples in this article don’t cover that operating system. Most Unix-like operating systems have some sort of package management for the software discussed here. I’d recommend you use those package systems because they’re the simplest to install; however, because such packages vary from OS to OS, I’m simply going to give the software requirements, and leave the installation up to you.

Briefly, you need to install the following software packages:

• Apache 2, (currently at version 2.0.50). If you already have an earlier version of Apache installed, be aware that one common problem arises when people use Apache 1.x and 2.x on the same system. While that’s possible, you can’t run them on the same port. The simplest solution to the problem is to run Apache 2 on a different port such as port 8080 instead of the standard port 80.
• Subversion. Subversion is part of the core software of the configuration repository. It’s designed to work with Apache 2.0, using the WebDAV protocol, so be sure to install both Apache and Subversion on the same machine. Apache needs direct access to the Subversion database to perform its updates. For more detailed information, try the excellent book Version Control with Subversion. As of this writing, the current version of Subversion is 1.0.5. If you’re planning to build the application from the source, make sure your BerkleyDB libraries are new enough (in the 4.1.x realm) and that the configure script uses the flags ?with-apache or ?with-axps to build Apache support into Subversion when running the configuration script.
• The WebDav protocol (mod_dav module). Subversion needs the mod_dav module to function, so make sure to install this as well.
• SSL (recommended). If you’d like the additional security of putting your repository behind an SSL-enabled site, you can build mod_ssl for your Apache installation. In addition, you’ll need to comment out two of the directives discussed below to force Apache to use SSL for the directories.
• ViewCVS (recommended). Subversion ships with a very limited Web-based front end for browser-based HTTP clients; however a much more advanced front end?ViewCVS?is fairly straightforward to install from source, or install as a package. The current version as of this writing is 0.9.2.
• Perl 5. The script that gathers the configurations is written in Perl, so you’ll need Perl 5. The scripts for this article were tested with Perl 5.8.3. The script uses a Perl module called Net::Telnet::Cisco. To install it, run the command perl -MCPAN -e ‘install Net::Telnet::Cisco, or use your package manager to install the module.

The next two packages are optional, depending on your configuration. The Configuration Repository can work in two different ways. The first is just to go and grab the Cisco configurations once a week. The second way is to configure them to be downloaded whenever an SNMP trap is sent. The first method is very simple to set up, the second is a little trickier, and requires some additional software packages.

• Net-SNMP. This package has a program that listens for SNMP traps. So, you’ll need to install this to update your configurations when they change. The current version as of this writing is version 5.1.1.
• SNMPTT. Finally, you’ll need to install SNMPTT, which processes the incoming SNMP traps and runs scripts on certain events. The current version as of this writing is 0.9. SNMPTT requires some additional Perl modules?Text::Parsewords, Getopt::Long, Config::IniFil, Time::HiRes and the Net-SNMP module. You should configure SNMPTT in “daemon” mode as well.

A Bit About Subversion

Figure 1. Subversion: Subversion is a revision control system that lets you easily track and compare changes to files.

Subversion (see Figure 1) is a revision control system that lets you easily track and compare changes to files. Typically, people use Subversion to track changes to source code. When a developer “checks out” a folder containing code files, Subversion ensures that nobody else can make changes to the code in that folder. As developers finish making changes to the source, they “commit” their changes to the source, which updates the “master copy” of the folder, but maintains a list of the changes that occurred.

But you don’t have to limit Subversion’s use to source code files; in this context, you can use it to track changes to your router configurations easily. The same basic theory applies to saving changes for Cisco configurations; however, you essentially have only a single source for changes. You can’t check out a single file in Subversion. Instead, you check out the entire folder containing the file you want to. Because of this behavior, you need to do a little work-around at the end. Don’t worry, though, because Subversion is only going to update the configurations that have actually changed. For example, if you’re updating weekly, and your routers haven’t had any configuration changes in the past week, then Subversion won’t update anything. You’ll need to make one change to your Cisco configurations if you plan on using these stored configurations for disaster recovery purposes. If you drop a working configuration onto a router, IOS automatically puts all interfaces into the “shut” state. Therefore, you must make sure to modify the configurations to be put into a “no shut” state when doing disaster recovery.

Configuring Subversion and Apache
You need to create the configuration repository in Subversion. First, create the following directory tree:

   gonzo:~ # mkdir /opt   gonzo:~ # mkdir /opt/svn   gonzo:~ # mkdir /opt/svn/user_access

Then, create the initial configuration repository.

   gonzo:~ # svnadmin create /opt/svn/depot

Next, set the permissions on the new configuration repository. Here’s where you need the user and group that Apache is configured to use that you found above:

   chown --R : /opt/svn/depot

Finally, create some users. Use the following command and then press Enter.

   htpasswd -cm /opt/svn/user_access/passwd 

Then, type in the password for that user.

   gonzo:/opt # htpasswd -cm /opt/svn/user_access/passwd someuser   New password:    Re-type new password:    Adding password for user someuser

Now that you’ve created the basic repository, you need to enable mod_dav and Subversion in your Apache configuration. Check your httpd.conf file for the following lines and add them if they’re not there:

   LoadModule dav_svn_module modules/mod_dav_svn.so   LoadModule dav_fs_module modules/mod_dav_fs.so

You need to add the Subversion information to httpd.conf as well, as shown below.

         DAV svn      SVNPath /srv/svn/depot         # Limit write permission to list of valid users.               # Require SSL connection for password protection.         # SSLRequireSSL            AuthType Basic         AuthName "Authorization Realm"         AuthUserFile /opt/svn/user_access/passwd         Require valid-user         

After making these modifications, run the command apachectl configtest to check the Apache configuration for errors. You should see the following on your screen:

   gonzo:/etc/apache2/conf.d # apache2ctl configtest   Syntax OK

If the configuration test doesn’t return Syntax OK, it’ll return errors instead, and you’ll need to fix these before you can continue. Assuming that your configuration is OK, restart Apache using the command apachectl restart.

Now, open your favorite Web browser, and go to the Apache site we just created. In the location bar, enter http:///depot. The Web page should look something like:

   Revision 0: /      Powered by Subversion version 1.0.5 (dev build).

As you can see, the repository is empty?which is to be expected because you haven’t imported any configurations yet. You’re going to create a basic structure to store configurations now. First, you need to “check out” the existing repository. Create a “tmp” directory inside your home directory.

   conrad@gonzo:~> cd tmp   conrad@gonzo:~/tmp> svn co http://localhost/depot   Checked out revision 4.

Testing Your Installation
You’re going to create a simple file to make sure that everything is working properly. Create a file called README.txt inside ~/tmp/depot, and put whatever content you’d like inside it.

   conrad@gonzo:~/tmp/depot> echo "This is a Router Configuration Repository" > ~/tmp/depot/README.txt

Next, add that file to the repository. Without this step, the repository won’t know about the new file, so this is important.

   conrad@gonzo:~/tmp/depot> svn add ~/tmp/depot/README.txt   A         /home/conrad/tmp/depot/README.txt

Commit the file. The repository keeps a log of changes as part of the commit process. You need to define a text editor for this. Make sure your EDITOR shell environmental is set for your favorite editor. Use echo $EDITOR to display this variable. If it is not set, use EDITOR=vi; export EDITOR or setenv EDITOR vi depending on your shell. Because tastes in editors vary, simply change “vi” to reflect the launch command for your favorite editor.

By default, Subversion will send your username as your login name to the server to perform a commit. It’ll simply ask for a password.

   conrad@gonzo:~/tmp/depot> svn commit   Adding         README.txt   Transmitting file data .   Committed revision 5.

After typing svn commit, your text editor will launch so you can put some notes in the log for the update. Now that you know commits are working properly, here’s a look at some of the basic things you can do with Subversion. First, remove the old ~/tmp/depot directory. Use rm?rf ~/tmp/depot to do this. Then, check out the repository again. The following command checks out the README.txt file you just created.

   conrad@gonzo:~/tmp> svn co http://localhost/depot   A  depot/README.txt   Checked out revision 5.

You can view the repository log:

   conrad@gonzo:~/tmp/depot> svn log   ------------------------------------------------------   r1 | conrad | 2004-07-11 16:26:32 -0400 (Sun, 11 Jul 2004) | 2 lines      Initial import      ------------------------------------------------------

You can also list the files that are in the repository:

   conrad@gonzo:~/tmp/depot> svn ls   README.txt

Now, add some more content to README.txt

   conrad@gonzo:~/tmp/depot> echo "Subversion is loads of fun" >> README.txt

You can tell that the file’s been modified, by running svn status:

   conrad@gonzo:~/tmp/depot> svn status   M      README.txt

Now, we’re going to create a basic configuration repository structure inside of Subversion. This could be as elaborate or as simple as you want. I’m simply creating different directories for router configurations and switch configurations.

   conrad@gonzo:~/tmp/depot> svn mkdir routers   A         routers   conrad@gonzo:~/tmp/depot> svn mkdir switches   A         switches

The svn status command now shows us that README.txt has been modified, and these two directories have been added:

   conrad@gonzo:~/tmp/depot> svn status   A      routers   A      switches   M      README.txt

Now, update the repository by running svn update, and then commit your changes.

   conrad@gonzo:~/tmp/depot> svn commit   Sending        README.txt   Adding         routers   Adding         switches   Transmitting file data .   Committed revision 6.

Just to show a few more features, go to ~/tmp, and check out the repository again:

   conrad@gonzo:~/tmp> svn co http://localhost/depot   Checked out revision 6.

Now, svn log shows not only the initial import, but the subsequent entries as well:

   conrad@gonzo:~/tmp/depot> svn log   ------------------------------------------------------   r6 | conrad | 2004-07-11 19:24:02 -0400 (Sun, 11 Jul 2004) | 3 lines      Added basic directory structure.         ------------------------------------------------------   r5 | conrad | 2004-07-11 19:11:57 -0400 (Sun, 11 Jul 2004) | 2 lines      Initial checkout      ------------------------------------------------------

One of Subversion’s nicest features is the ability to show what’s changed, using svn diff.

   conrad@gonzo:~/tmp/depot> svn diff -r 5:6 README.txt    Index: README.txt   ======================================================   --- README.txt  (revision 5)   +++ README.txt  (revision 6)   @@ -1 +1,2 @@    This is a Router Configuration Repository   +Subversion is loads of fun

In the diff output shown above, you can see the changes made from revision 5 to revision 6 of the README.txt file. The plus sign (+) next to the second line (“Subversion is loads of fun”) shows that that is the new line. Now, when you access the repository via a regular Web browser, it should look like this:

   Revision 6: /          * README.txt       * routers/       * switches/      Powered by Subversion version 1.0.5 (dev build).

Find a Better Front End
ViewCVS was originally written for CVS. However, because CVS and SVN are so similar in functionality, it works just as well for SVN. You can run most of the above examples easily from ViewCVS, making it a convenient, portable way to view files.

Install ViewCVS, and set up Apache to use the viewcvs.cgi as a path. You’ll need to add these lines to your httpd.conf file:

      ScriptAlias /viewcvs /path/to/your/viewcvs/cgi/viewcvs.cgi

Next, you need to make sure that the viewcvs path has permissions to run properly:

         AllowOverride None      Options +ExecCGI      Order allow,deny      Allow from all        AuthType Basic        AuthName "Authorization Realm"        AuthUserFile /srv/svn/user_access/passwd        Require valid-user        # Require SSL connection for password protection.        # SSLRequireSSL   

In the installation root of ViewCVS, there’s a file called viewcvs.conf. Make sure that cvs_roots is commented out, and the following directives are enabled:

   svn_roots:           base_depot : /opt/svn/depot      svn_parent_path = /opt/svn/depot   default_root = base_depot

Finally, you’ll need to run apachectl configtest and then apachectl restart. Now, go to http:///viewcvs on your Web browser, and a nice Web-based version (see Figure 2) of some of what you’ve previously done should appear.

Figure 2. Repository View: By navigating to the viewcvs page of your server, you’ll see a nice browser-based view of the items in the repository.

Populating the Repository
As I mentioned previously, there are two major ways that the repository can be updated. The first method is fairly simple?simply run a cron job at regular intervals. The downside to this method is that the routers may not have the absolute most up-to-date revision of the configuration; however, it’s very easy to configure. The second method is to set up your Cisco router to send SNMP traps to a host whenever configuration changes happen. The host will then retrieve the latest configuration and upload it into the repository automatically.

Cron-Based updates
You’ll need to download the sample script (snmp-conf.pl), and configure your routers towards the bottom of the script. Next, you’ll need to paste the following code into a script at /usr/local/bin/router-cron.sh, and then run the command chmod 755 /usr/local/bin/router-cron.sh.

   #!/bin/sh      # set these for your environment   SVN=/usr/local/bin/svn   NOW=`date`   CMD=/usr/local/bin/snmp-conf.pl      # check out the configs   cd /tmp; $SVN co http:///depot/routers      # edit the following line to contain the    # IP addresses of your    routers   for ROUTER in '10.0.0.1 10.0.1.1 10.0.2.1'   do           $CMD -t ${ROUTER} -o /tmp/routers   done      # update the repository   cd /tmp/routers; $SVN update   # commit the changes   cd /tmp/routers; $SVN commit --message       "Auto update on $NOW"

Then, set up the cron job. The following example causes the configurations to be backed up every Thursday at midnight:

   0 0 * * 5 /usr/local/bin/router-cron.sh

If you’ve decided to use the cron method, ignore the next section of this article and skip to the section titled “Configuring the Initial Repository.” You need to take some steps to ensure the proper functionality of the Subversion repository.

Configuring SNMP
You can configure a Cisco router to send SNMP traps when certain events happen. A SNMP trap, simply put, is a message triggered by a system event, such as “Hey! A link just went down!” Typically, some sort of network management software that understands SNMP traps (such as IBM Tivoli Netview or Nagios) correlates these messages with the events. The Net-SNMP package also has a program that understands how to deal with SNMP events.

First, start up the SNMP trap daemon, by running snmptrapd from the Net-SNMP package. Run snmptrapd as the root user. You’ll also need to verify that any devices that might block SNMP traffic between the router and the SNMP host, such as firewalls, have been configured to allow this traffic through.

Next, you’ll need to log onto your router and configure it for SNMP traps. As a bare minimum, you need to run SNMP, and you need to send traps to the SNMP host.

   SomeRouter(config)# snmp-server host           SomeRouter(config)# snmp-server enable traps config

SNMP will more or less randomly decide which interface to source the SNMP packets from, but you can simplify maintenance if you configure the device to use one specific port. The example below uses the FastEthernet0/0 port; however, you can use whatever port you like. It’s important that your snmp-conf.pl script is configured to use this IP address as the identifier of the router.

   SomeRouter(config)# snmp-server trap-source FastEthernet 0/0

Now you can test the configuration. On the SNMP host, run a tail?f command on the System log. Under Solaris, it’s in /var/adm, on most Linux-like systems, it’s under /var/log. On the router, type config t, and then exit. The result should look something like this:

   Jul 18 22:42:32 netflowc snmptrapd[20718]: [ID 702911    daemon.warning] 10.0.37.31: Enterprise Specific Trap    (.1) Uptime: 111 days, 6:49:44.81,    .1.3.6.1.4.1.9.9.43.1.1.6.1.3.145 = INTEGER: 1,    .1.3.6.1.4.1.9.9.43.1.1.6.1.4.145 = INTEGER: 2,    .1.3.6.1.4.1.9.9.43.1.1.6.1.5.145 = INTEGER: 3   Jul 18 22:42:32 netflowc snmptrapd[20718]: [ID 702911    daemon.warning] 10.0.37.31 [10.0.37.31]: Trap ,    .1.3.6.1.2.1.1.3.0 = Wrong Type (should be Timeticks):    INTEGER: 961498481, .1.3.6.1.6.3.1.1.4.1.0 = OID:    .1.3.6.1.4.1.9.9.43.2.0.1,    .1.3.6.1.4.1.9.9.43.1.1.6.1.3.145 = INTEGER: 1,    .1.3.6.1.4.1.9.9.43.1.1.6.1.4.145 = INTEGER: 2,    .1.3.6.1.4.1.9.9.43.1.1.6.1.5.145 = INTEGER: 3

Now that you know SNMP is working, you need to configure SNMPTT. First, create a snmptt.conf file. You need to tell SNMPTT how to deal with the configuration traps, so add the following lines to snmptt.conf:

   EVENT ciscoConfigManEvent .1.3.6.1.4.1.9.9.43.2.0.1       "Status Events" Normal   FORMAT Notification of a configuration management       event as $*   EXEC /usr/local/bin/snmp-conf.pl -t $ar -a $1 -b $2 -c       $3 -o /tmp/routers |       /usr/local/bin/router-commit.sh   SDESC   Notification of a configuration management event as   recorded in ccmHistoryEventTable.   EDESC

You’ll need to download the script snmp-conf.pl and put it in /usr/local/bin, then run the command chmod 755 /usr/local/bin/snmp-conf.pl to make it executable. After doing that, you’ll need to configure your router information in the script. Finally, create another helper script to manage the SVN updates. Paste the following code into a script, and then run chmod 755 /usr/local/bin/router-commit.sh.

   #!/bin/sh   # set SVN for your environment   SVN=/usr/local/bin/svn   NOW=`date`      cd /tmp/routers; $SVN update   cd /tmp/routers; $SVN commit --message       "Auto-triggered update on: $NOW"      cd /tmp/; $SVN co http:///depot/routers

Finally, you’ll need to configure Net-SNMP’s snmptrapd by adding the following line to the snmptrapd.conf file.

   traphandle default /usr/local/sbin/snmptthandler

Configuring the Initial Repository
You need to upload some files into the repository for it to actually work. To do that, go to /tmp and check out the configurations.

   conrad@gonzo:~> cd /tmp   conrad@gonzo:/tmp> svn co http://localhost/depot/routers   Checked out revision 6.

Now, run the snmp-conf.pl script on each of your routers:

   /usr/local/bin/snmp-conf --t  -o    /tmp/routers

This will populate the repository with basic files. You’ll also need to run the following command on each of your routers:

   conrad@gonzo:/tmp/routers> svn add 10.0.0.1.txt   A         10.0.0.1.txt

After adding them all, you’ll need to run an update and then a commit:

   conrad@gonzo:/tmp/routers> svn update   At revision 6.   conrad@gonzo:/tmp/routers> svn commit   Adding         10.0.0.1.txt   Transmitting file data .   Committed revision 7.

You need to check out the configurations again for things to work properly:

   conrad@gonzo:/tmp> svn co http://localhost/depot/routers   Checked out revision 7.

Testing Everything Out
If you configured your repository to be updated via cron-based updates, run the /usr/local/bin/router-cron.sh script now. It should automatically update your repository. Then, skip the next paragraph to view your results.

If you configured your repository to be updated from SNMP trap triggers, there are a few more steps. You need to restart snmptrapd to use snmptt. Kill snmptrapd if it’s running, and re-start it using the command snmptrapd?On. Go to one of the routers you’ve configured, and make a basic configuration change. For example, you can simply change the hostname.

   SomeRouter(config)# hostname AnotherRouter   AnotherRouter(config)# exit   AnotherRouter# wr

Now, you should be able to go to http:///viewcvs, and view the configuration that was just changed, and see what changed about it. If your updates were configured via cron jobs, you are finished! Otherwise, there are just a few more steps that need to happen.

Finishing Up
Subversion needs to have the configurations checked out for updates to happen properly, so you need to set them up so they’re always in the checked-out state.

Create the following script as /usr/local/bin/router-maint.sh, and run chmod 755 /usr/local/bin/router-maint.sh to make it executable.

   #!/bin/sh      # set these for your environment   SVN=/usr/local/bin/svn   NOW=`date`      cd /tmp/routers; $SVN update   # commit any current changes   # this will throw an error the first time it is run    # You can safely ignore the error.   cd /tmp/routers; $SVN commit --message       "Auto update on $NOW"   # check out the configs   cd /tmp; $SVN co       http://gonzo.timconrad.org/depot/routers

Create a nightly cron job to do any missed updates and check out a new revision of the repository:

   0 0 * * * /usr/local/bin/router-maint.sh

When I initially set up a configuration repository, I did it primarily for disaster recovery purposes. Over time, I’ve found that it’s also very useful when I quickly need to look at a configuration for a given network device, or I’m curious as to what’s changed on the router over a certain amount of time. One extension to this that would be helpful in environments with multiple network administrators, would be the capability to keep track of which user makes which change to the devices, however, that would take some additional development. Hopefully, you’ll find the configuration repository as useful as I have!