HiddenOne

I finally got around to uploading some of the pictures I took at my current place of employment. They are not great quality, since I took them with my Blackberry. Below are a few of my favorites and the stories behind them. You can view the rest of the pictures here.

First up is a piece of equipment that was a major piece of the network infrastructure at our largest building. We started getting complaints of poor network performance in the back 1/4 of the building, I was quickly able to confirm these reports with my own testing. I had only been in my position for a short while, so I wasn’t familiar with all of the nuances of the network. Luckily the building tech was with me and mentioned that there was a “switch” in the ceiling in that part of the building, however none of my network maps or scans showed this device. We go to the spot, I climb into the ceiling and there is a 10 Mb with all of its lights (activity and collision) lit! Apparently this thing had been there for years!

Next we have a switch that was mounted to the wall in a room. This is pretty common, but what really got me was instead of removing the switch to paint the wall they just painted around it! Even after I removed the screws that held it to the wall, I had to pry it off with a screwdriver since the multiple layers of purple paint where acting as an adhesive.

While you can’t help but laugh at some of this, it really has been a great opportunity as a network administrator. All seven of our buildings had networks in similar states depicted in the pictures above. This has given me numerous opportunities to put my network design skills to good use.

Recently while sifting through some logs at work, I discovered some odd traffic coming from our DHCP server. The server appeared to be sending DNS updates to external DNS servers. This traffic never made it out of our network since our egress policy was not expecting to see DNS traffic from the DHCP server, so the packets were dropped.

So why was our DHCP server trying to send these update outside of our network? We have two DNS servers locally that should be receiving all of the DNS updates.

My first step was to check out our primary DNS server and double-check the reverse lookup zones. Below is a list of some of the zones I found (IP addresses have been changed to protect the innocent):

10.1.5.x
10.1.8.x
10.1.20.x

Spot the problem? These are very specific reverse lookup zones. Say we have a subnet that has usable addresses spanning from 10.1.5.1 to 10.1.6.254. When the leases from the 10.1.5.1 – 10.1.5.255 range have been handed out and leases from the 10.1.6 group of addresses begin to be served out, the DNS updates for the 10.1.6 addresses have nowhere to go. Essentially we had reverse lookup zones for a very specific set of addresses.

The solution was to create the proper reverse zone that would cover all of the subnets we use. The zone, or zones, that need to be created depend on the address space your organization uses internally. Since we use addresses in the class A 10.0.0.0 range, I needed to create a reverse zone named “10.in-addr.arpa.”. Below is a quick list of the common private address blocks and the reverse lookup zones they require.

After creating the proper zone for our internal IP scheme, the DNS updates for all of the DHCP leases started to appear in DNS. Additionally, the log entries stating that our DHCP server was sending updates to external DNS servers stopped.

A common mistake is to think that you only need the reverse lookup zone for the subnet you are using. For example, if someone is using the 192.168.2.0/24 subnet, they may think they only need the “2.168.192.in-addr.arpa.” reverse lookup zone. In fact they still need to create the “168.192.in-addr.arpa.” zone and let their DNS server handle the rest.

This may seem like a basic rule to most people, but not everyone knows about reverse lookup zones. In my case the current design was inherited. Honestly I may not have noticed the problem with the reverse lookup zones if I wasn’t sifting through logs.

The lack of updates here recently has been compounded by a number of issues. The first of these issues is my “9-5″, which can be anything from an “8-4″, a “7-7″ or anything in between. Many times going home simply means I go home only to VPN in and complete some tasks I didn’t get to during the day.

Over the past few weeks we have been having some issues with our tape backups. Back in December of 2008 I inherited the duties of Backup Administrator since I had some experience at a previous employer. The first issue was that our tape library was appearing as offline in BackupExec. This was related to SCSI cable which was an easy fix. After we resolved that issue, we noticed that the “Job Rate” on our backups drastically dropped. Drastically means the job rates went from about 1,200 MB/Min to 190 MB/Min! This meant that the 100 GB Exchange database backup went from a little over two hours start to finish to over 26 hours! After some troubleshooting it was discovered that the tape library had been changed to use the HP drivers while I was troubleshooting the previous issue with our vendor. After changing the drivers back to the ones provided by BackupExec the job rates have returned to normal.

On the network front, I have been doing some work to bring the management of our network up to par. First I have deployed RADIUS authentication on our switches in order to better control and monitor access. Additionally I have deployed a banner or message of the day (MOTD) on each switch. This is the message you see before you are prompted to provide your login credentials. On HP Procurve switches HP provides a boring canned message asking you to register the switch. Deploying a banner is accomplished with one command:

banner motd #
This is my custom banner!

Don't mess with my network gear!
#

Once I get some additional copper modules installed in some of our switches, I will be enabling BPDU protection on the edge ports (Port that end-users connect into). Evil Routers has an article that covers configuring BPDU protection on HP Procurve switches.

BPDU protection prevents a rouge switch from changing your spanning tree topology. If the switch sees a BPDU on a port that has BPDU protection enabled, it can react in a number of different ways. My personal choice is to have the port disabled permanently. This requires the invention of a network administrator for the Port to be enabled me again. This gives me a chance to talk to the user, explain what happened, and why it happened.

My CCNA studies are still progressing slowly. I am hoping to begin the final preparation for the exam in the coming weeks.

You just landed that new networking position after months of interviews and waiting! If you’re lucky, the hard part is over and you will slip into a position where the network paracticly manages itself. Meanwhile, back in the real work…

Network administrators are like software developers. That is documentation is usually an afterthought, and sometimes it is not a thought at all! We have all faced this problem. This is the situation that I walked into this past summer. The key here is not to look at the situation as a problem, but as a challenge.  Our largest building was our biggest challenge. The cabling was not done in a structured manner, and in some cases I was baffled how some of the “redundant links” did not cause network loops since STP was not activated.

The biggest challenge in this building was troubleshooting link problems between the switch closets. Dropping the link from one cabinet to another would cause half the building to go down in some cases. Without proper network documentation, we were left in the dark. With the way the cabling between the switches was laid out, it would be a copious amount of work document this by hand. Enter Solarwinds with their LANsurveyor product.

LANsurveyor can be used to create network maps with a minimal amount of information. In my case, I was able to create a network map for each of our seven buildings using nothing more then the IP address range for that building and our SNMP community strings. Even if your community strings are not standardized, changing them on your devices would be considerably less work than creating a manual network map.

This convenience does come at a price. The version I use sells for about $2000 and is a standalone piece of software. There is a $500 version that integrates with Visio, but I have not used that so I cannot comment. You do have the option of downloading a trial before you purchase the software. If $2000 is not in your personal budget, you may be able to sell the software to your boss if you calculate all of the man hours it would take to map your network by hand!

Below are some screen shots of LANsurveyor, one of which shows the chaotic network map for the building I mentioned above.

LANsurveyor network discovery settings:

Sample Network Map (I zoomed out so you can see how difficult this would have been to document by other means):

Note: This is not an advertisement for Solarwinds, and thus I am not profiting from this post! If you have used other software to solve a similar problem, please share your experiences in the comments section.

The past five months have been crazy! I landed my first network related job, as a network administrator. As soon as I started there was a major project to put a new network in the one building over the summer (cabling and all). That was followed by some slightly smaller projects, such as an upgrade to our main router. On top of that I have been busy with personal projects, including working on my car.

Unfortunately all of this caused many things to fall by the wayside. This site and my CCNA study where two major things that I fell behind with. The past few weeks I have really stepped up my CCNA study. The down side of that is that I have had to review many old topics and my books are now a version behind the current CCNA tests.

I am going to try to update this site more often. One change I am going to make is the renaming of the “VMware Whitebox” section. I have not given up on that, but I have decided to play with some ofther virtualization options. That means I will be setting up and playing with Windows Sever 2008 with Hyper-V! I am hoping to get this setup soon and share my setup here.

Recently I came across a situation where I wanted to delete selected files from the Flash memory on my router. In order to delete the files, I issued the following command:

delete flash:/file.bin

After doing that I was surprised to see that the files were only marked for deletion in flash, which meant they were also still occupying the storage space I needed for other files. In order to permanently delete the files, the following command needs to be issued:

squeeze flash:

After trying to run the squeeze command, I received an error saying no squeeze logs could be found and the operation failed. After some research I discovered that you need to run squeeze on the flash memory while it is empty (I later found you can do this with just the IOS image in flash). Here is the process I followed.

1. Make sure you have a backup copy of the IOS image you want on the router.
2. Format flash (Do not reboot the router after this)

   erase flash

3. Reload the IOS image into flash.

   copy tftp://xxx.xxx.xxx.xxx/ios_image.bin flash:/

4. Run squeeze on flash

   squeeze flash:

After all of this is completed, you will be able to mark files for deletion using the delete command and then permanently delete them with the squeeze command.

Here is a song from someone at RIPE 55. I thought it was a great piece!

The Day the Routers Died

With the arrival of the 3524 switch, I ran into the first situation where I had to reset the password on a switch. I have performed this process in the past on routers, but even the CCNA book I have did not cover the process on switches. The process is quite different when compared to the password recovery procedure on a router.

Cisco calls this process password recovery, as opposed to a password reset. The goal is to maintain the current configuration on the device, in this case a switch, while allowing you to reset the PrivilegeEXEC and Telnet passwords to something you can remember. In my case, I skipped the last few steps in the Cisco process since I was not concerned with saving the configuration on the switch.

This is a good page to bookmark, since you will likely encounter a situation where you purchase a used switch that will have a password set by someone else.

Cisco Switch Password Recovery

After much thought, I decided to create this web site. My main motivation was to share some of the things I have been working on for the last couple of years, mostly related to networking.

Pursuing my CCNA certification, has been a goal of mine for over a year now. Only recently have I started to regularly train and study for the exam. I have had a small lab for about a year now, and it is just now seeing some proper use.

Eventually I hope this site turns into an open forum where people can share their thoughts, experiences, and opinions.