Attached you can find some notes I took while going through some CCIE Data Center material specific to the Nexus platform. Note: There is not expectation to accuracy for both content, spelling or grammar. These notes are meant to be a quick reference for myself when reviewing the material. CCIE Data Center Nexus Notes
The current environment that I support makes use of a Checkpoint solution for remote access. This environment is several revisions behind and is running on aging server hardware that we provided. Due to the current state of the remote access environment, we were looking at a forklift upgrade even if we stuck with the incumbent vendor. This
I like to keep the flash memory of my network equipment just like I keep my house; Clean and clutter-free. In the current environment I support, we make use of Cisco 2960s switches in a stacked configuration. When searching for 2960s stack upgrade procedures most of the Cisco documentation that Google pulls up details using
Today a co-worker brought to my attention new Cisco IOS command that make some related interface-related tasks a bit easier to implement. The command is “switchport host” and it is an interface level command. This means that it can be used on a single interface or on a range of interfaces, such as in the
In May of this year, I marked the one year anniversary of obtaining my CCNA certification by passing the composite exam. The following are my thoughts, opinions, and observations one year out. Use it or loose it! As far as knowledge is concerned, I am a firm believer that what you do not user you
Port/Traffic mirroring is a tool that you will quickly learn to love as a network administrator. The process of port mirroring copies the traffic from the source port to the destination/exit port. Usually the source port has the traffic of interest, and the exit port usually has some sort of network monitoring tool attached (such as a laptop running Wireshark).
Today I am going to detail the process of configuring local port mirroring on a switch. The act of local port mirroring copies the traffic from one port to another port on the same switch. It is possible to redirect the traffic from the source port to a port on another switch (This is called remote mirroring), but this will be covered in another article.
Local port mirroring can be configured with two commands. The first command defines the mirroring session (you can have up to four per switch), as well as the exit port (the port to which the traffic should be mirrored). The following is an example of this command:
mirror 1 port A2
The mirror session number can be any number from one to four. Obviously the exit port can be any port on the switch, as long as it’s not the same as the source port.
Next the interface command is used to configure the source interface. Below is an example command:
interface A1 monitor all both mirror 1
The interface that is specified is the source interface. Next, the “monitor all both” command specifies to monitor all traffic in both directions (in and outbound). You have the option to change “both” to just “in” or “out”. Finally, you specify the mirror session that the traffic should be copied to. The session number should be the same number that you used in the previous “mirror” command.
This covers basic configuration of local port mirroring. With these commands you will be able to quickly configure a local mirroring session for troubleshooting. As with anything, refer to the manual for your switch for further configuration options.
Future articles will cover applying ACLs to mirroring sessions, as well as configuring remote mirroring sessions.
Recent news about at least two major fiber cuts in California have been stirring thoughts about how fragile the our network infrastructure really can be. These cuts have some very obvious consequences, such as wide spread internet outages that can have a major impact on many businesses. Many other services usually transition to fiber optic cable for their long-haul transit. The result in this particular case is that some areas are sans cell phone, land line, and emergency (911) services! Being without voice, video and data services today is the equivalent of being cut off from the rest of the world!
What concerns me the most is that were no backup links to keep mission critical services up and running in this case. From a networking perspective we design our networks to have multiple diverse paths in an attempt to avoid such outages. We also implement disaster recovery (DR) plans in the event that our best laid engineering plans are thwarted.
One question to answer is where does the responsibility lie with respect to maintaining emergency services in such a situation? Is it the responsibility of the carrier that is moving the packets to ensure that in the event of a fiber cut, packets deemed mission critical have an alternate route? It is my opinion that the responsibility for ensuring emergency service are not effected by fiber cuts lies with the local government. Just as it is the responsibility of a company to maintain redundant paths in the event that their primary service provider fails to meet their SLA.
From a business perspective, the company has the right to file for penalties according to their network agreement when their service provider does not meet their SLA, however the onus lies with the company to ensure business functions can continue when the service provider has a failure. I feel this same principal can extend to local governments and the emergency services they provide.
The best way to prevent mistakes in the future is to learn from those in the past. With any luck, this incident will bring some of these concerns into the public light so that they may be addressesed.
The lack of updates here recently has been compounded by a number of issues. The first of these issues is my “9-5″, which can be anything from an “8-4″, a “7-7″ or anything in between. Many times going home simply means I go home only to VPN in and complete some tasks I didn’t get to during the day.
Over the past few weeks we have been having some issues with our tape backups. Back in December of 2008 I inherited the duties of Backup Administrator since I had some experience at a previous employer. The first issue was that our tape library was appearing as offline in BackupExec. This was related to SCSI cable which was an easy fix. After we resolved that issue, we noticed that the “Job Rate” on our backups drastically dropped. Drastically means the job rates went from about 1,200 MB/Min to 190 MB/Min! This meant that the 100 GB Exchange database backup went from a little over two hours start to finish to over 26 hours! After some troubleshooting it was discovered that the tape library had been changed to use the HP drivers while I was troubleshooting the previous issue with our vendor. After changing the drivers back to the ones provided by BackupExec the job rates have returned to normal.
On the network front, I have been doing some work to bring the management of our network up to par. First I have deployed RADIUS authentication on our switches in order to better control and monitor access. Additionally I have deployed a banner or message of the day (MOTD) on each switch. This is the message you see before you are prompted to provide your login credentials. On HP Procurve switches HP provides a boring canned message asking you to register the switch. Deploying a banner is accomplished with one command:
banner motd # This is my custom banner! Don't mess with my network gear! #
Once I get some additional copper modules installed in some of our switches, I will be enabling BPDU protection on the edge ports (Port that end-users connect into). Evil Routers has an article that covers configuring BPDU protection on HP Procurve switches.
BPDU protection prevents a rouge switch from changing your spanning tree topology. If the switch sees a BPDU on a port that has BPDU protection enabled, it can react in a number of different ways. My personal choice is to have the port disabled permanently. This requires the invention of a network administrator for the Port to be enabled me again. This gives me a chance to talk to the user, explain what happened, and why it happened.
My CCNA studies are still progressing slowly. I am hoping to begin the final preparation for the exam in the coming weeks.
Continuing with my Procurve articles, my current article will detail how to use a RADIUS server to handle authentication on a switch. This article will cover using a RADIUS server to handle authentication for the following methods of access on a Procurve switch:
- Web Interface
This configuration was tested on an HP Procurve 5412zl switch running K.13.25 software. Configuration on other switches with different software versions may vary. Consult the Access Security Guide for your switch for further details.
Configuring your RADIUS server is beyond the scope of this article, particularly since there are many options for RADIUS servers out there. Free Radius is one popular option. I can personally confirm that this configuration works with Microsoft’s IAS which was setup to authenticate against Active Directory.
Once your RADIUS server has been properly configured, you need to configure your switch so that it knows how to communicate with the server. This is done using the command:
radius-server host IP Address key 'MyKey01'
This command accomplishes a few tasks. First, it tells the switch at what IP address to reach the RADIUS server. You should substitute IP Address with the IP address of your server. The second part establishes the password or key that has been set for this switch. This key is used by the RADIUS server to verify the identity of the switch. On your RADIUS server, you can set a unique key for each switch (which are usually called RAIDUS clients).
With the connection to the RADIUS server established, you are now ready to configure authentication for each of the modes of access. This will be accomplished using the aaa authentication command. Below is an example of the format of the command.
aaa authentication method enable/login auth1 auth2
The first option is the access method for which you would like to apply the authentication. Your options here are: console, telnet, ssh, and web. The enable/login options specify when authentication should be attempted. Login specifies that authentication should be attempted when a user first attempts to login. Enable specifies that authentication should be attempted when a user enters the “enable” command. The enable option will not apply to web access.
The last two options specify the means of authentication. Auth1 is the primary means of authentication, which in this case you should set to radius. Auth2 is optional as it is a backup means of authentication. However, it is a good idea to set this to local, just in case the switch cannot communicate with the RADIUS server. Local specifies that the passwords set locally will be used for authentication.
Here are some of the various commands that could be used to enable RADIUS authentication.
aaa authentication console login radius local aaa authentication console enable radius local aaa authentication web login radius local aaa authentication telnet login radius local aaa authentication telnet enable radius local aaa authentication ssh login radius local aaa authentication ssh enable radius local
One final command to note is something that many network administrators may find useful. That command is:
aaa authentication login privilege-mode
This command tells the switch to expect additional parameters from the RADIUS server upon successful authentication. These parameters tell the switch which mode the user can gain access into. You can configure the RADIUS server to send the Administrative-User service type for users that are granted access to enable/configuration modes. On the other hand, you can configure the server to return the NAS-Prompt-User service type for users that you only want to grant access to operator mode.
This is very useful if you have a group of users that you only wish to have minimal access to your switches. If these limited access users attempt to enter manager mode with the enable command, they will be prompted to login again but they will then be presented with an “Access Denied” error message.
After everything has been configured, you can confirm that the proper forms of primary and secondary authentication have been applied with the following command:
Below is some output produced by this command:
J8698A-01# show authentication Status and Counters - Authentication Information Login Attempts : 3 Respect Privilege : Enabled | Login Login Enable Enable Access Task | Primary Secondary Primary Secondary ----------- + ---------- ---------- ---------- ---------- Console | Local None Local None Telnet | Radius Local Radius Local Port-Access | Local None Webui | Radius None Local None SSH | Local None Local None Web-Auth | ChapRadius None MAC-Auth | ChapRadius None
I made a change to the comments feature tonight, and in the process discovered a “bug”.
First, readers will now be able to leave a comment without being a registered user. All you need to do is provide your name and an email address. There are still some conditions that will hold a comment until it is approved, such as if it contains a certain amount of links.
In the process, I dicovered that after leaving a comment users were redircted to a 404 error page. After some quick investigation, I found the cause to be an extra space I had in the permalink configuration. The error has now been remediated so everything is running normally.
Update: I should also mention that when you post a comment, your email address is not displayed. You name will be a hyperlink to your web site.
A topic not often covered in the Procurve community is the issue of licensing certain features on a switch. HP does an excellent job of providing a robust feature set on most of their equipment out of the box. However, some applications require additional features that are not included with the switch.
An issue arose at work where we wanted to use a 5406zl to route to an external entity. There was a thought that we could peer with the entity using OSPF. After some quick digging around the command line, I found that our 5406zl only supported RIP with the current license. Upgrading to a premium license would be required to support OSPF. The premium license adds support for the following on 3500 and 5400 series switches.
- PIM Dense mode
- PIM Sparse mode
- QinQ (IEEE 802.1ad)
How can you tell what license is currently installed on your switch? If you are running OSPF, that is a good indicator that you switch currently has a premium license installed. A better way to view the installed license is with the show licenses command:
J8698A-01# show licenses
Feature Group License Status
premium Not Installed
Here you can clearly see that this switch does not have the premium license installed, as is indicated by the “Not Installed” license status.
Installing a premium license is a multi-step process that will require you to use the my.procurve.com web site. Obtaining a premium license is beyond the scope of this article, however I will cover how to obtain the hardware ID for you switch. The hardware ID is required when obtaining a premium license.
Obtaining the Hardware ID
In order to obtain the hardware ID for the switch, you need to enter configuration mode using the configure terminal command. Once you have entered configuration mode, issue the command licenses hardware-id premium. The output of this command is shown below:
J8698A-01(config)# licenses hardware-id premium
The hardware Id for package “premium” is
There are a few points I would like to note. First, the hardware ID has been sanitized. However, the format above is the same format that you can expect on other switches. Second, the exact command to be used will depend on the model switch in use. Some HP documentation makes use of the licenses hardware-id premium-edge command to retireve the hardware ID. The use of context sensitive help will reveal which command should be used on your switch.
Installing a Premium License
Installing a premium license is strightforward. Again, you need to enter configuration mode using the configure terminal command. Once in configuration mode, issue the following command:
licenses install premium <license-key>
You will replace <license-key> with the key that was provided to you on the my.procurve.com web site. Keep in mind that after the switch installs the license, it will reboot.
Removing a Premium License
At some point, you may need to remove a license from a switch for various reasons. Like the previous commands, you must be in configuration mode to uninstall a license. From configuration mode issue the command:
licenses uninstall premium
This command will issue the following warning:
J8698A-01(config)# licenses uninstall premium
**** CAUTION ****
Completion of this operation will remove the functionality that is enabled
in the premium feature group, and will delete the configuration for
those features from the active configuration.
Device will be rebooted, do you want to continue [y/n]?
After you answer yes, the switch will delete the key and issue an uninstall verification number. It is very important that you copy this uninstall verification number, as you will need to enter it into the my.procurve.com website to “release” the license for use on another switch. After this the switch will reboot.
Viewing Uninstalled Licesnes
Should you need to view a license that was uninstalled from a switch you can use the following command:
show licenses uninstalled
This will show the license that was uninstalled, as well as the uninstall verification key. This command comes in handy if you forgot to save the uninstall verification key after removing a license.
Virtualization has been a hot topic in the IT would for a while now. From the datacenter to our home labs, virtualization is changing the way we work and play.
I recently made an addition to my lab that I think will allow me to add more complexity to lab simulations, at the same time it has saved me some money! A few months back you will recall that I purchased some additional hardware and made a virtualization server (using Windows 2008 Server and Hyper-V). Since that setup was not getting much use, I went a different route that should prove to be more useful.
There are three physical parts to my current lab setup. They are: 16U rack with physical network equipment, server running VMware Server, and my laptop. The rack consists of the following:
- 1 x 2528 access server
- 3 x 2525 routers
- 1 x 3640 router (NM-2FE2W, NM-1E2W, NM-8A/S)
- 1 x 2924 switch
- 2 x 3550 switches (L3 support)
The server running VMware has a quad core Intel Xenon processor with 8GB of RAM and 5 x 500GB hard drives in a hot-swappable drive cage. This used to be a file server which is why it has so many hard drives and a hot-swap cage. Only one hard drive is being used to house both the operating system and all of the virtual machines. Finally, my laptop is my old reliable Compaq Presario V2000. With 1GB of RAM, a 1.8Ghz AMD Turion processor, and a 5400RPM hard drive this thing gets bogged down in a hurry when I start running anything more than a few routers in GNS3! It survived five years of college as many other laptops around me crumbled, so I can’t come to replace it just yet!
How do I connect all of this? It’s not overly complicated once you get everything straight in your head, however when putting something like this together for the first time I suggest some actual planning! The diagram below will help illustrate what I have done.
The two large boxes at the top (vmware-server and Laptop) are the two physical computers. The setup on my laptop is straightforward, since I only have GNS3 running there (no VMs). As you can see, I use the cloud feature in GNS3 to connect the wired network card on my laptop to my physical lab (The rack at the bottom of the diagram).
The VMware server is what complicates things. First you will notice that the server has two network cards. I have used this to my advantage to segment some things out. The first network card (eth0) is used for management of the VMware server through the web interface, as well as linking any VMs to the Internet (either directly via a bridged network or through some GNS3 routing). The second network card (eth1) is strictly used to tie the VM running GNS3 to my physical lab rack. There are two virtual network adaptors (vmnet0 and vmnet5) that are bridged directly to each of the physical network cards.
Inside of the VMware server there are four other virtual network adaptors (vmnet1 to vmnet4). These four devices are host-only network adaptors. This means that they are not tied in any way to either of the servers physical network cards. Notice that the GNS3 VM connects to each of these virtual network adaptors. I did this so that I can perform routing between each subnet. The GNS3 VM is also tied to each of the bridged network adaptors so that I can route traffic externally as well.
One thing not pictured in the diagram is the other VMs I use. These are simple setups, as each VM connects to one (and only one) of the host only network adaptors. These VMs act as endpoints/nodes to test connectivity between other nodes.
Hopefully this will provide other people with a jumping off point for adding virtualization to their lab. This can be accomplished for a small amount of money. A desktop with plenty of RAM will work just fine as a VMware server. Combine the price of memory now with the low, low price of free for Linux and VMware Server and you have an incredible deal!
It seems like not too long ago that I had to jump through hoops just to be able to get some lab time in at work, or when I was otherwise away from my rack. My original routine for remote labbing was to leave the equipment I needed powered on with my laptop connected via console cable to my access server. I then setup port forwarding to my laptop so that I could RDP into it while I was at work.
This setup was inefficient for several reasons. The most obvious was the fact that I had to leave all of that equipment powered on for a full eight hours just so I might be able to get less than an hour of lab time. Another problem I ran into was that the connections between the equipment could not be reconfigured when I was not physically present.
My recent purchase of a Western Digital Passport external hard drive sent me on a quest for more portable apps to install onto it for remote use. It was then that I discovered GNS3 could be installed onto a flash drive or external hard drive (just about any removable media for that matter). The process was simple, during the installation of GNS3, you just set the installation directory of a folder on your removable media. The installation only requires about 40MB, so a minimal amount of space is required. From there you just copy the IOS images you need onto the flash drive and you are good to go!
It should be noted that GNS3 does WinPcap if you would like to bind some of your labs to the physical network card on the laptop. If you do not require this feature, then there are no other dependencies.
Network jitter, or jitter, is the name given to a variation in the time delay between packet arrival. This concept is better explained using the illustration below.
In the above illustration, the gaps between each of the packets represent the time it takes for each packet to reach the destination. Jitter is shown by the uneven gaps between packets two and three, as well as three and four.
In a perfect world, every packet should arrive at a set amount of time after the preceding packet. In reality, there are many factors to take into consideration when jitter is experienced. Sometimes the cause of jitter is beyond your control, since issues may arise outside your network.
The effect Jitter has on network applications can vary. It is unlikely for a user surfing the internet to report a problem that is a result of jitter. Other real time services, such as VOIP, can experience serious problems related to jitter. Lucky for us, many vendors (including Cisco) build provisions into their routers that can compensate for jitter.
On Cisco routers, the playout delay buffer (PDB) is the mechanism that is used to compensate for jitter. The PDB stores the incoming packets and then sends them to the next destination as a steady stream. This buffering process is similar to that used with other real-time protocols such as those used for audio and video. The ultimate goal of the buffer is to negate any jitter by relaying the packets as a steady stream. An illustration of the packets before and after the playout delay buffer does its job is shown below.
The buffer can only compensate for packets that are delayed within a specified range. If packets start to arrive outside of the working range of the buffer, those packets are dropped. With VOIP a dropped packet can mean the loss of some of the audio, which can make part of a conversation seem choppy.
In a Cisco router, the playout delay buffer sends a steady stream of packets to the digital signal processors. The main job of the DSP is to convert the audio from digital to analog. A secondary function of the DSP is to compensate for missing packets. If a packet is missing, the DSP can make an educated guess as to the contents of the missing packet and insert that missing piece into the audio stream. The result is that the end user never hears a difference. The DSP can only compensate for a finite amount of dropped packets before the end users start to notice an effect on call quality.
Jitter can present some interesting problems, particularly when you are dealing with real-time services. Cisco has built some provisions into their routers to help counter some of the effects of jitter.
You just landed that new networking position after months of interviews and waiting! If you’re lucky, the hard part is over and you will slip into a position where the network paracticly manages itself. Meanwhile, back in the real work…
Network administrators are like software developers. That is documentation is usually an afterthought, and sometimes it is not a thought at all! We have all faced this problem. This is the situation that I walked into this past summer. The key here is not to look at the situation as a problem, but as a challenge. Our largest building was our biggest challenge. The cabling was not done in a structured manner, and in some cases I was baffled how some of the “redundant links” did not cause network loops since STP was not activated.
The biggest challenge in this building was troubleshooting link problems between the switch closets. Dropping the link from one cabinet to another would cause half the building to go down in some cases. Without proper network documentation, we were left in the dark. With the way the cabling between the switches was laid out, it would be a copious amount of work document this by hand. Enter Solarwinds with their LANsurveyor product.
LANsurveyor can be used to create network maps with a minimal amount of information. In my case, I was able to create a network map for each of our seven buildings using nothing more then the IP address range for that building and our SNMP community strings. Even if your community strings are not standardized, changing them on your devices would be considerably less work than creating a manual network map.
This convenience does come at a price. The version I use sells for about $2000 and is a standalone piece of software. There is a $500 version that integrates with Visio, but I have not used that so I cannot comment. You do have the option of downloading a trial before you purchase the software. If $2000 is not in your personal budget, you may be able to sell the software to your boss if you calculate all of the man hours it would take to map your network by hand!
Below are some screen shots of LANsurveyor, one of which shows the chaotic network map for the building I mentioned above.
LANsurveyor network discovery settings:
Sample Network Map (I zoomed out so you can see how difficult this would have been to document by other means):
Note: This is not an advertisement for Solarwinds, and thus I am not profiting from this post! If you have used other software to solve a similar problem, please share your experiences in the comments section.