HiddenOne

The underlying goal at work this month is to get some much needed insight and auditing abilities into our network. I have looked at Rancid before, and after doing some research I finally decided to take the plunge and implement it into our network.

The steps below assume that you are running Ubuntu Server 8.04.3 LTS i386 with a standard LAMP installation. They may work with other Debian based distributions, but your mileage may vary. For this installation I will be using Rancid version 2.3.2, which is the current stable release at the time that I wrote this article. I encourage you to check the Shrubbery web site and get the latest release that is available. Additionally, this article will go into a bit more detail about using Rancid with various HP Procurve equipment.

It would be a lie to say I figured out all of these steps on my own. Some of it was trial and error, other information I found in various places on the Internet. Tweets from Jeremy over at Evil Routers gave me the needed insight on editing the hrancid file to support some of the newer Procurve equipment.

Rancid Installation

After your base LAMP system is installed, you need to install some packages that are required to install and run rancid. You can combine these commands into one line, but I broke them out into two lines to show that the second line begins the actual Rancid dependencies.

apt-get install gcc make libc6-dev
apt-get install expect cvs

Next, create a user that will run Rancid. This user’s home directory will be used as the install path and CVS repository for Rancid. For simplicity’s sake I have used the user name “rancid”, but you can use anything.

adduser –home /home/rancid rancid

Download the latest rancid source archive using wget.

wget ftp://ftp.shrubbery.net/pub/rancid/rancid-2.3.2.tar.gz

Copy the archive to /usr/src and extract the contents.

cp rancid-2.3.2.tar.gz /usr/src
cd /usr/src
tar xvfz rancid-2.3.2.tar.gz
cd rancid-2.3.2

Configure and install Rancid. Note that we are providing arguments for specific installation directories.

./configure -prefix=/home/rancid -localstatedir=/home/rancid/var/rancid
make install

Since the previous commands were issued by the root user, we need to change ownership of /home/rancid back to the rancid user.

chown -R rancid:rancid /home/rancid

Rancid Configuration

Edit /home/rancid/etc/rancid.conf and look for the following line (which is probably commented out):

LIST_OF_GROUPS=”"

There will probably be text between the quotes which you will delete. The group names you provide will be your CVS groups, and will translate into folders when viewing the repository with CVSweb. In my case I choose to create a group for each building I manage. Once everything is running, each building’s folder will contain the network devices in that building as well as their configuration files. Below is an example of a list of groups. Each group is seperated from the previous group name with a space.

LIST_OF_GROUPS=”Philadelphia Boston Miami Burbank”

Edit the /home/rancid/.cloginrc file. This contains the login information for your switches. If you use AAA/Radius to authenticate users that manager network equipment, your job is easy. All you have to do is provide a username and password for every device like so:

add user * {ranciduser}
add password * {ranciduserpassword}
add autoenable * 1

The first two lines are self-explanatory. The asterisk is a wild card meaning all or every device. The last line tells Rancid that the authentication information provided automatically drops the user into enable mode.

Since the cloginrc file contains passwords, you want to change its permissions to ensure only the rancid user can access this file.

chmod 600 .cloginrc
chown rancid:rancid .cloginrc

Now we run the rancid-cvs command, which will create the CVS groups based on the “LIST_OF_GROUPS” we created earlier. It is important that you run this command as the rancid user.

su – rancid
/home/rancid/bin/rancid-cvs

Next, edit the router.db files for each group. There is a seperate router.db file for each group. The groups are located in the /home/rancid/var/rancid directory (each group has its own folder). The router.db files tell rancid the IP address or hostname of the device, the device manufacturer, and the status of the device. For example

192.168.1.2:hp:up

If you use a hostname instead of an IP address, be sure the server running Rancid is able to resolve the hostnames. For newer HP Procurve devices, you will set the manufacturer to “hp”. This works for 5400zl and 8200zl models as well as older model stackables (3400, 2524, etc). If you have a switch that was manufactured by Foundry, such as the Procurve 9308, set the manufacturer to “foundry”. Rancid will only process the device if the status is set to “up”. Any other status will cause the device to be skipped.

Edit hrancid File

The newer Procurve models no longer support the “show system-information” command, instead this command has been changed to “show system”. Luckily this command is specific enough that when it is issued on older switches that still use “show system-information”, it translates properly. In order to support the newer switches, we need to edit the “hrancid” file located in /home/rancid/bin. Once you have the file open, look for the following block of code:

@commandtable = (
{’show version’                 => ‘ShowVersion’},
{’show flash’                   => ‘ShowFlash’},
{’show system-information’      => ‘ShowSystem’},
{’show system information’      => ‘ShowSystem’},
{’show module’                  => ‘ShowModule’},
{’show stack’                   => ‘ShowStack’},
{‘write term’                   => ‘WriteTerm’}
);

You need to change “show system-information” to “show system”. The “show system information” line does not need to be modified.

Run Rancid

Now we will run Rancid for the first time. Again, we are going to need to do this as the rancid user. Be patient, as this command can take some time to complete if you have a large number of devices.

su – rancid
/home/rancid/bin/rancid-run

Install & Configure CVSweb

CVSweb will provide a web interface where you can view the configuration files for your network devices in addition to performing diffs on the files to see what has changed. Installing CVSweb is simple.

apt-get install cvsweb

After the installation is complete, edit the /etc/cvsweb/cvsweb.conf file to point CVSweb to your Rancid CVS repository.

@CVSrepositories = (
#’local’ => ['Local Repository', '/var/lib/cvs'],
‘MNSD’ => ['MNSD Devices', '/home/rancid/var/rancid/CVS'],
);

I usually comment out the “local” line with a pound, just to make navigation a bit easier (since in my case the local repository is not used.

Create a link to in the www directory that points to the location of the CVSweb icons.

ln -s /usr/share/cvsweb /var/www/cvsweb

You can now access your repository using the following URL: http://YOUR_SERVER/cgi-bin/cvsweb

Rancid Automation

Now that everything is working properly, we want to make sure Rancid runs automatically every so often. We will do this by editing the /etc/crontab file and adding the following lines.

1 12,23 * * *    rancid    /home/rancid/bin/rancid-run
50 23 * * *    rancid    /usr/bin/find /home/rancid/var/rancid/logs -type f -mtime +2 -exec rm {} \;

The first line runs Rancid at 12:00 PM and 12:00 AM. Carefully choose your intervals, because the number of devices you are running Rancid against and the size of the config on each device will increase the run time. The second line periodically clears the configuration differ log files.

There you have it! You should now have a working version of Rancid!

HP Procurve, Network Tools

At work we recently converted from a network monitoring package that was provided by our switching vendor, to Orion Network Performance Monitor from Solarwinds. Shortly after installing and configuring Orion, I configured a Linux server running Rancid for configuration management. In my eyes, both Orion and Rancid fall under the network monitoring/management category, so I wanted to make accessing both as easy as possible.

Orion allows you to create and add custom links to your menu bars, but adding a typical link is a bit “clunky” since the link either takes the user away from the Orion page or opens in a new window. The steps below describe how to create a custom link that will retain the Orion header and menu barat the top of the page. This has made navigating between the two much easier and has eliminated the need for me to have both Orion and Rancid open in separate browser tabs.

Creating a Custom Link

First, make sure you log into Orion with credentials that have administrator access. After you do that, click the “Admin” link that appears in the Orion menu bar.

On the admin page, click the “Customize Menu Bars” link under the Customize section.

On the “Customize Menu Bars” page, I choose to edit the admin menu bar by clicking the “Edit” button under the menu bar. If you want different groups of users to see this link, you may have to edit different or multiple menu bars.

On the edit menu bar page, click the “Add” button that appears towards the bottom of the page.

This will open the dialog box shown below.

Fill in the name and description fields as you see fit. Next, uncheck the “Open in a New Window” option box. In the URL field type the following:

http://YOUR_SERVER/Orion/External.aspx?Title=Rancid&URL=

After the “URL=” argument, type the URL of the web page you wish the link to display. When you are finished, click the “OK” button.

Back on the edit menu bar page, you will notice your new link in the “Available Items” list. Click and drag the link from this list to the “Selected Items” list.

When finished, click the “Submit” button and enjoy the fruits of your 30 seconds of labor!

This is a very simple process, but it came in handy for me and makes day to day management a bit easier.

Network Tools

There are many tools that can make the life of a network administrator easier. The following is a list of programs that I use on a daily basis to make management of the network easier. As a bonus, all of these programs are available free of charge.

A few of these tools I use mainly for organization, as I am anal retentive about being organized and making commonly used information easily accessible. You may find some of these tools more or less useful depending on your habits.

KeePass – This a great tool to organize and securely store password information. It has multiple methods of authentication, including password and key based. KeePass allows you to create groups as well as nested groups to easily organize information. The entire database is also searchable to further make accessing information easier. The included password generator can help you come up with passwords other than the usual P@$$word!

Wireshark  - Wireshark is a popular traffic capture program. A traffic capture program comes in handy when issues arise and you need to see communication information between hosts. It has helped my countless times when troubleshooting port authentication issues. Wireshark has a plethora of other features that can aide in finding the cause of any issues.

Tftpd32 - You’re going to need a TFTP server as some point in your career. It may be to upload a new software image, backup a config, or some other reason. Tftpd32 takes your run of the mill TFTP server and adds a few more features including: Tftp Client, DHCP server, and Syslog server. Even with these features, the program remains light and quick.

Network Stumbler - If you’re using the bars displayed in the Windows wireless connection manager to determine wireless signal strength, you’re not getting an accurate picture. Net Sumbler will give you a real-time graph of the signal to noise ratio. It will also provide various other pieces of information about the wireless networks it discovers.

PuTTY Connection Manager - PuTTY has been my personal choice for managing network devices from the terminal (Console, Telnet, etc). PuTTY CM is a separate piece of software that uses PuTTY for connections to devices. The biggest advantage of PuTTY CM, for me, is the ability to build a database of Telnet/SSH configurations to network devices. This allows me to quickly connect to a device and automate various tasks at login.

Network Tools

In a perfect world every MDF and IDF would have its own dedicated room with limited access. Since most of us live in the real world, compromises must be made.

About one year ago the organization I work for went through a complete network infrastructure upgrade in one of our buildings. Since space was limited and building renovations were not in the scope of the project, we ended up using locking 45U server cabinets to house the switching equipment as well as patch panels. Most of these cabinets were put in shared storage rooms, and only one was placed in a room the received steady traffic.
With the renovation of another one of our buildings this year, we had to make a decision on if we wanted to use locking cabinets or dedicate closets specifically for networking. My past experience and current opportunity allowed me to reflect one the pros and cons of both. Here is a quick breakdown that I came up with:

Network Cabinets:

• Allows flexible placement of distribution frames to maintain cabling standards
• Provides relative security when placed in trafficked areas

Network Closets:

• Security by obscurity
• Generally allows more flexibility for cable routing
• Easy access without interrupting others

These lists could be much longer and could contain many more technical arguments. In this case, I choose to look at the less technical reasons for choosing one over the other.

In my case, I believe that a dedicated network closet is superior to a locking cabinet in a public area. Placing anything in the public eye will by nature draw more attention to that object. Add some whirly fans along with blinking lights and even a grown adult will be drawn to a network cabinet. This attention may come with some unwanted consequences such as vandalism or even accidental damage. I strongly believe in security by obscurity. What’s out of sight is out of mind, and therefore people are less likely to tamper with the equipment.

Network closets are only superior if they are dedicated and secure. Closets that are shared will quickly become a place for others to store their junk. This increases the risk that someone might cause an outage or otherwise damage equipment. I often see custodians move their cleaning carts into these spaces, bringing with them all sorts of liquids and chemicals you don’t want on your equipment!

As with any project take your requirements into close consideration. Also ensure that you plan for the future and not just present needs. A properly designed network closet can easily scale to future needs, whereas with a cabinet you are more or less stuck between the posts.

Design

As I finish reading DNS & Bind, I started looking through my Amazon wishlist for a few more titles that I could add to my collection. I choose two titles that I think will tie well into some projects that I am involved with at work.

The first is Campus Network Design Fundamentals, a Cisco Press book. My hope is that this book will help me strengthen my network design skills as well as provide a good starting point for integrating VOIP into a network.

The second book is Network Warrior. This book intreagued me because of that statement on the cover that said: “Everything you need to know that wasn’t on the CCNA exam”. This should be interesting!

 

Be sure to check out my Bookshelf for a full list of network related books that you may enjoy.

Site Updates

In my previous article, I covered how to configure NAT overload, or PAT, on a Fortigate firewall. This time around I will cover how to configure static NAT.

Static NAT is used to map one IP address to another. This is typically used to perform translations from public address space to private address space. Many organizations do not put servers directly on the internet. Instead the servers may sit on a private LAN, or even in a DMZ. Static NAT mappings are used in these cases to allow servers that are sitting behind a firewall to extend services to the Internet.

Static NAT Configuration

Fortigate firewalls call the static NAT function “Virtual IP Mapping”. These virtual IP mappings can be used for static or dynamic NAT; however I will only be covering static NAT with a one to one mapping relationship.
A new virtual IP mapping is created on the “Virtual IP” tab under Firewall -> Virtual IP. On the “Virtual IP” tab, click on the “Create New” button. This will bring you to the “Add New Virtual IP Mapping” page as shown below:

Fill in the following information:

• Name – This is a name that will uniquely identify this mapping. It may be easy to use the name of the server that mapping is for just to keep things simple.
• External Interface – This will almost always be set to “external” unless you have a different configuration.
• External IP Address/Range – Here you can enter either the first and last IP addresses in a range, or just a single IP address (Leaving the second text box empty). Remember to use the public IP address here.
• Internal IP Address/Range – Same as the external IP, except this time you will use the private IP address of the server.
• Port Forwarding – Only use this if people will be sending requests from the outside on one port, while your server will be listening on another. For example if people will be requesting web pages on port 80, but your server’s web server runs on port 8080.

Once you have configured the virtual IP mapping it is time to create the corresponding policy. This will be done by going to Firewall -> Policy and then the “Policy” tab. Click on the “Create New” button to create a new policy.

This time we are going to create a policy for traffic entering our network, so the source interface will be “external” and the destination interface will be “internal” (or DMZ depending on your configuration).

The next crucial step is to set the “Destination Address. If you expand the drop down menu, you will see a section titled “Virtual IP”, under which you should see the name of the “Virtual IP Mapping” that was just created. It’s also good practice to only accept traffic for the services that the server will be running. There is no use in accepting incoming HTTP for a server that only runs SMTP services, plus it’s a good security practice.

Finally, in this case you do not need to select “NAT”. The virtual IP mapping will take care of all of the NAT work.

Now you should be armed to configure all of the basic NAT services on a Fortigate.

Firewalls

At work we recently migrated to a different Internet Service Provider. This migration brought many changes, but a big one for me as the network administrator was the fact that we would be handling our own address space and network address translation (NAT). Although Fortigate support was great at confirming the NAT configuration process for me, I found that the documentation of NAT Overload/PAT configuration was lacking. My hope is that someone out there will find this document useful in their work environment.

I will be using the Fortigate web interface for all configuration steps. The unit I am using is a Fortigate 800 running software 4.0.2,build0099. Other versions or Fortigate models may use a different process to configure NAT.

Before starting, ensure that your Fortigate is in NAT mode. This can be accomplished by going to the “Operation” tab under System -> Config. Keep in mind that you will lose many, if not all, of your policies and settings by making this change. Be sure to backup your configuration file first so you have something to fall back on if things go south.

NAT Overload/PAT Configuration

NAT overload, also called Port Address Translation (PAT), is a process used to allow many computers to access the Internet using one or more IP addresses. This is used when you have a large number of computers on your LAN, but your public IP address space does not allow for a one to one mapping of public IP addresses for each computer. Without going into detail, PAT uses one public IP address with a TCP port number appended to the IP address. The port number is what is used to identify which session maps to which computer.

First a public IP address must be assigned to the external interface on your firewall. This can be accomplished by going to System -> Network -> Interface tab. On the Interface tab, click the edit icon for the external interface. You will see the screen below:

Under “Addressing mode”, click “Manual” and enter the IP address and subnet mask. The IP address and subnet mask should be separated by a forward slash like this: “176.16.1.2/255.255.255.0”. Click “Ok” to save your changes.

Next, we need to make sure the firewall knows how to get where it needs to go. This is done through routes under the Router -> Static -> Static Routes tab. Here, you’re probably going to need at least two routes. The first route you will need is a default route, which is represented using “0.0.0.0/0.0.0.0”. Here the gateway address should be set to the next hop of your Internet bound routing equipment.
Now your firewall needs to know how to get back to your LAN subnets. It’s up to you how you configure this, but for simplicity sake we will use a blanket route. Say all of the IP addresses on your LAN subnets start with “10.70.xxx.xxx”, you can simply create a route for all of the subnets that start with “10.70” using the following IP address and subnet mask combination: “10.70.0.0/255.255.0.0”. You may choose to be more specific with your routes.

Below is an example route configuration. This is used on a production Fortigate, so I have blocked out some information to protect the innocent.

Now that the routing has been setup we can create a policy to allow outbound traffic and enable NAT. This is accomplished by going to the “Policy” tab under Firewall -> Policy. Once there, click the “Create New” tab. Here you will be presented with a list of options, but we will focus on a select few.

The “Source Interface” and “Destination Interface” options are self explanatory. We are going to set the source to “Internal” and the destination to “External” since we want to configure internal to external translation (the translation table will automatically handle the reverse mapping). The next crucial step is to ensure that the “NAT” checkbox is selected to enable NAT for this policy. You can create an “allow all” policy for now just to test if NAT is working. Various “what’s my IP” web sites should be able to confirm that the public IP address of the external interface on the firewall is being used for NAT.

One note about the NAT checkbox; it needs to be checked for any internal to external policy that you wish to allow traffic. At work I have a policy that blocks outbound RDP access; however there is one vendor’s server to which we need to permit outbound RDP access. If you create the outbound policy and place it in the appropriate spot on the policy list, but forget to check the NAT option the policy will not work. If things don’t seem to be working as they should, or is seems that the policy is being ignored, always check the Nat checkbox!

That covers NAT overload/PAT configuration on the Fortigate 800. In a follow-up article I will cover static NAT configuration, which is a less involved process.

Firewalls

IP Telephony is something that every network administrator is going to run into at some point in time. Its many features really do help reduce costs across the board. Having a unified network that carries voice and other data services reduces cabling, equipment, and maintenance costs just to name a few. Even with the popularity of IP telephony there are many organizations out there they have yet to take the plunge. This further increases chances that you will find yourself working as part of a telephony project at some point.

If your company has a network with all Cisco devices and a Cisco IP telephony system has been chosen, everything seems straightforward. Cisco is going to make sure their products work well together; otherwise they’re going to have a tough time marketing their telephony system to current Cisco customers. Concerns arise when you have a mixed vendor environment. Will it work? How do the switches need to be configured?

If you have an HP Procurve based network, Procurve has you covered with their interoperability guides. These guides detail how to get phones from various vendors to work with Procurve networking equipment. The guides are very detailed covering configuration of the phones, the switching equipment, and even Call Manager (In the case of Cisco). Each section of the guide includes screen shots of any web interfaces as well as command prompt output you may encounter along the way.

If you are planning to run an IP telephony system on top of HP Procurve gear, these guides will be your source of knowledge more than a few times!

Thanks to “procurvehelp” on Twitter for posting a link to these documents!

HP Procurve, Voice (VOIP)

In some cases you can use a RADIUS server to perform port authentication based on the MAC address of the connecting device. In other cases the port usage may remain static, but you still may want to lock down the ports without going through the process of configuring a RADIUS server. The ‘port-security’ command can be used to lock down ports without the need for a RADIUS server or even knowing the MAC address of the computers that will connect to the port.

Where would this be useful? At work I use this to secure ports in our computer labs. In the labs the desktops stay in one place. In this situation, seeing multiple MAC addresses through any port (except for the uplink port) would indicate that either someone setup a rouge switch/hub or a game of musical computers is being played. This particular command allows me to set the maximum amount of MAC addresses allowed though a port to one.

The command that makes all of this happen is ‘port-security’. Here is an example of a command that I use in our computer labs at work:

port-security 1-20 address-limit 1 learn-mode static action send-disable

Breaking the command down into sections makes it easier to understand:

‘port-security 1-20’ – Apply the command and all of the following options to ports 1 thru 20. A single port or group of ports can be specified in addition to a range.

‘address-limit 1’ – Set the amount of MAC addresses that can gain access through the switch port(s). Most HP switches support an address limit of up to about 38.

‘learn-mode static’ – This configures how the switch will learn or know the MAC addresses that are allowed to gain access through the port(s).

‘action send-disable’ – This sets the action to be taken once the threshold defined by ‘address-limit’ is crossed. ‘send-disable’ sends an error message and disables the port.

My preferred option for ‘learn-mode’ is static. In this case I do not predefine any MAC addresses, which leaves many people wondering how the switch knows which MAC addresses to allow though which ports. When the above command is first issued, the switch doesn’t know that information. With the ‘learn-mode static’ option the switch will take the first source MAC address is sees on a port and assign that MAC address as an authorized MAC address for that particular port. This process is repeated for each port in this case. The picture below illustrates this process so that it is easier to understand.

Our basic training has told us that when you first take a switch out of the box it is a blank slate. For the sake of explanation we are going to assume this for our scenario. We are also going to assume that the previously discussed ‘port-security’ command has been applied to interfaces one thru twenty. You connect Computer A to interface one of the switch. Soon after that Computer A sends a packet which enters the switch through interface one. The switch reviews the packet and one of the steps it takes is to add the source MAC address (AAAAAA-AAAAAA) to its MAC address table. This accomplishes multiple things at once. First it now tells the switch which interface Computer A is attached to, which allows more efficient forwarding of packets destined for Computer A. This MAC address table is also used by the ‘port-security’ command to determine if another computer should be allowed to gain access to the network through that port.

If some black hat comes by and connects their laptop to interface one, the switch will now see a new source MAC address attached to the packets. In this case the ‘address-limit’ was set to one, and the switch already has one MAC address associated with port one in its MAC address table. This means that the switch will take the ‘send-disable’ action, disabling the interface.

There you have it! This is a quick and easy way to add another layer of security to your network! I caution to use this command wisely as using it in the wrong situation could make you one unpopular network administrator!

As always consult the manual for your HP Procurve switch for the exact command syntax.

HP Procurve

I use many different tools to monitor, diagnose, and test my network on a daily basis. One tool that is often overlooked is a cell phone. Cell phones have enjoyed an explosive growth in popularity over the years. PDAs in particular have bridged a major gap, going from a device for business people to a device for the every day consumer. With the popularity of PDAs comes the likelihood that most, if not all, of these phones have a data plan associated to them.

When making changes to any network it is often important to test all services from inside and outside the network. With tightening budgets, you can consider yourself lucky if your employer springs for an Internet connection strictly for testing. That’s ok, since you can use that fancy phone that’s attached to your hip for testing.

At work we recently changed Internet service providers. Along with this change came changes to our address space, DNS zones, firewall policies and more. It was important for me to test connectivity to our external facing services after the migration, particularly since I was taking this opportunity to apply hardened policies to our firewall. Once the migration was complete, I was able to use my cell phone to see check the status of our DNS zone propagation as well as to test how well my firewall policies were working (sometimes they work too well).

Tethering your cell phone to your computer can be a tricky subject. Many carriers charge an extra monthly fee for tethering for one reason or another. There are ways to tether your cell phone to your computer without the need for an additional tethering plan. I caution you in following any of these processes. There is the likelihood that if you abuse the “free” tethering your cell phone provider can find out and you may be stuck with a fat bill.

Moderation is the key here. In my opinion you should only use the free tethering process minimally, such a when you need to do testing. These processes should not be used if you plan to tether regularly, or use your phone as you would a data card.

Below are some links to tutorials on how to setup various cell phones for tethering. Remember; use them wisely and at your own risk.

• Using a Blackberry as a modem
• Windows Mobile phone as a modem
• Tetherbot (Proxy) for Android Phones
• Palm Pre Bluetooth Tethering

Network Tools