Ten Gigabit Switching Thoughts

As mentioned in a previous post, I recently attended the Gestalt IT-organized Net Tech Field Day in San Jose, CA. This event brought me back in contact with a former colleague (Terry Slattery) and a number of my podcast/blogging friends (Ethan Banks, Greg Ferro, Brandon Carroll and Ivan Pepelnjak). In addition, I met in person for the first time many of the network-related blog authors and tweeters I follow (Jeremy Gaddis, , Josh Horton, Jennifer Huber, Steve Rossen and Jeremy Stretch). Oh, and Bob Plankers was there too, but he is just a server guy ;)

Thank you all for your company and your contributions to the many technical and non-technical discussions. And a special thank you to Stephen Foskett and Claire Chaplais for organizing this event. It was an amazing feat of logistics and vendor management. I am in awe of how smoothly the event went. I often run into more trouble during my daily commute, and I work from home!

During the planning of this event, Stephen Foskett asked the attendees what they were most interested in hearing about. The plurality of the responses, including my own, mentioned Data Center technologies. The vendors did not disappoint, as no fewer than five of the seven participants focused on this area. We received briefings on data center switching technologies from Hewlett-Packard, Force10, Juniper and Arista Networks.

The goal of this post is to compare/contrast the 10gb switch offerings of these vendors. Also, because the presentations/discussions made it clear that these vendors measure themselves against Cisco Systems in both market share and feature parity, I’ll include Cisco as well. I regret not including Foundry, Extreme, Oracle/Sun and anyone else, but I do not have any firsthand knowledge of their offerings. Any mention I would make of them would be strictly web-based research. You can do that yourself. :)

(Note… I decided to delay my review of the vendors’ chassis aggregation technologies for another blog post. This one was getting too long even without it.)

Fixed Configuration 10gb Switches

In short, they all have them. Arista sort of “ups the ante” in terms of advertised performance with their 7148SX switch. It is advertised as a low-latency device suitable for High Performance Computing (HPC) and High Frequency Trading (HFT) needs. Force10 also competes in this space with their S2410 device, which is promoted as a component of the New York Stock Exchange’s network. Our Arista contact made the point of saying that their switch is not eligible to be deployed at the NYSE because their company is not listed on that exchange. Abner Germanow (@abnerg) of Juniper mentioned that their devices were used in stock exchanges as well. Most of the supplied documentation mentions Juniper's M-Series routing platforms. At least one link (http://bloga.tw/a648bU) mentions the use of Juniper's EX-series LAN switches. I also noticed the Juniper routers included in the 60 Minutes feature on HFT. Cisco and HP do not appear to be competitive in the HPC/HFT arena, although I may have missed something in my research.

Another important item to note is that each of the presenting switch vendors at Net Tech Field Day has a 10gb, fixed configuration Layer-3 switch in their portfolio. Cisco (who did not present at the event) does not yet have this available. In a meeting with Jim Capobianco of Cisco last week, I learned that the upcoming Nexus 5548 & 5596 switches will eventually have this capability. It will require the installation of a Layer-3 Forwarding Engine, and will not be available until Q1 CY011. I am surprised they’re taking so long to deliver this, as it must be the cause of lost sales opportunities. The 4900M has something to offer in this space, but it is clearly not an integrated part of the new Nexus DC approach.

Lack of Innovation in the Space

I was struck by the similarities of all the switching vendors. Perhaps it was the tight timing constraints of the Net Tech Field Day sessions (most were 2 – 4 hours, with hard stops at the end), but with the exception of Arista Network’s offering (which I’ll discuss below), all of the vendors had very similar stories. This has been noted by several of my fellow attendees (Most notably Ivan). I don’t mean to pile on the criticism, as I’m sure developing these products is very difficult in itself, but I would love to see a significant differentiator from each vendor.

Juniper’s One OS

Aside from Arista, Juniper probably did the best job of differentiation with their “One OS” discussion. Their claim is that having a unified OS across multiple switching and routing platforms reduces the OPEX. Support for this claim include:

1. Network engineers only need to learn a single CLI
2. Commands are common to all platforms, allowing for better configuration standardization
3. Feature parity across all devices

Counterpoints include:

1. Not everything is standardized in JUNOS, such as hardware-based QoS configuration
2. Feature-set differences negate part of the feature parity claim (for example, no MPLS on LAN switches)
3. Cisco’s assertion that purpose-built OSs are better suited for unique environments

I’m not yet sure how I feel about this one. In all honesty, I’ve not had significant trouble learning new CLIs. The feature parity argument carries a bit more weight with me, especially considering my challenges with implementing features across IOS and NX-OS. I am also sympathetic to the OS-sprawl argument, best described by Mike Morris on his Network World blog. I suppose I’ll let the industry sort this one out without my input. I’m sure they’ll manage :)

Arista… Finally Something New

I was clued into Arista Networks about a year ago, when I read that Jayshree Ullal (and later Doug Gourlay) jumped ship from Cisco to join the 10gb switching startup. An industry friend of mine also highlighted their offering to me a few months later. At some point, I got added to a sales list for the company, and for the last few months I’ve received occasional marketing materials via email (thanks Alicia!). For me, the buildup to Arista Network’s presentation was quite extensive. I did not want this influence my fellow attendees’ perceptions, so I kept my thoughts to myself during the event.

Doug Gourlay of Arista scored a lot of points with the audience when he quickly explained that Arista Networks builds Data Center switches, and nothing else. It is clear that Arista is not attempting to be all things to all people. Their switches’ TCAM has room for 16K routes, 16K MAC addresses and 16K ARP entries. They are not going to be able to hold the Internet routing table, nor is Arista attempting to sell products that could do that. Doug was quite blunt when he said that “Arista is selling to companies where IT makes money.” Such businesses include Wall Street firms, HPC opportunities (Bio-Tech and other sciences) and social media websites.

So what’s new? For one thing, the switch runs a nearly-standard version of Linux, Fedora Core 12, kernel 2.6.31 (thank you to Doug for the correction) . According to the company, only about 750 lines were changed in the kernel to support the movement of device interrupts from system space to user space. This facilitates the starting and restarting of device drivers, and protects those processes from affecting the stability of the overall system. End users can build FreeBSD-compatible programs and run them in user space within the OS. EOS, Arista’s switch operating system, normally only requires 10% of one CPU core. On their dual-core switches, this leaves 95% of the processor power to custom-written applications. We were assured that EOS receives priority, so it is unlikely that a user application would affect the stability of the switching function. This capability is a standard feature of the Arista platform, unlike the additional cost of Cisco’s NM-based machine.

A second compelling EOS feature is VM Tracer. This allows a network admin to determine what device is attached to a particular port. If it is an ESX/ESXi server, it can query the server using VMWare’s API to determine which VMs are running on it. If a VM is VMotion’d to another ESX host, the switch can detect this and move the port-profile to the new location. It would be interesting to see exactly how this feature stacks up against Cisco’s NX-OS capabilities.

Summary

There are plenty of options available for 10gb data center network builds. Cisco is likely the safe option, although I do not see that they have any compelling features that would preclude me from choosing another vendor. Long ago I learned that one of the best negotiation tactics is to find two (or more) solutions you would be happy to deploy, then let both vendors know it. This will often get you the lowest price for your project. For the last few years, this has been difficult to do, since Cisco has done a relatively good job of innovating in the data center space. My recent Net Field Tech Day experience has shown me that there are other options that meet or exceed Cisco’s performance specs, so maybe it is time to search for competitive bids.

The Arista Networks presentation also demonstrates that other vendors are not standing still. Arista appears to be in a unique position to be able to price their devices at a premium. As a market strategy, Cisco and the other switch vendors need to begin innovating to put themselves in a similar position. (Hint… FCOE is not the answer). For Cisco, UCS is a potential driver of network equipment sales, but what about the other vendors?

(Disclaimer – Arista Networks, Force10, Hewlett Packard, and Juniper were sponsoring organizations of this event. There is no obligations for me to write anything about these companies or the other participants in this event. So while these musings came out of a sponsored trip, they are assuredly my own thoughts.)

The Ascendency of Hewlett Packard?

On Thursday I sat through several hours of Hewlett Packard presentations as part of the Net Tech Field Day program. Is there finally a viable competitor to Cisco across multiple product lines?

(Disclaimer – Hewlett Packard is one of the sponsoring organizations of this event. There is no obligations for me to write anything about HP, or the other participants in this event. So while these musings came out of a sponsored trip, they are assuredly my own thoughts)

Through its recent acquisition of 3Com, Hewlett Packard has acquired some fairly impressive networking technology. The A-Series switch line stacks up well to Cisco’s Nexus product line. If you read the following marketing description, can you determine if this is a description of the Nexus 7K platform or of the HP A-12500? I don’t think I could:

Next-generation, large core/data center switching platforms with innovative Intelligent Resilient Framework (IRF) technology. 18- and 6-slot switches with up to 6.6 Tbps performance and up to 128 (1:1) or 512 (4:1) 10GbE ports, and 864 GbE ports. Non-blocking, zero service interruption design, and architecture support for 40 GbE and 100 GbE. Wire speed L2/IPv4/IPv6/MPLS.

Except for the mentioning of IRF and the slot count, it sounds like the Nexus 7K to me. (Reference http://h10144.www1.hp.com/products/switches/index.aspx#A12500).

Unfortunately HP didn’t provide any tangible information about how this switch looks and feels in a predominantly Cisco network. The Tech Field Day participants pleaded for some white papers and/or deployment guides, but none are available at this time. In a somewhat ominous sign, the A-Series presenter mentioned that a design guide had been written, but HP has not yet determined if it would be treated as proprietary information for HP’s services organization.

Takeaways

I have two main takeaways from the HP portion of Tech Field Day:

1) Cisco may have a viable competitor across their networking product line.

If the industry rejects new Cisco-proprietary protocols in the Data Center (L2MP, VNTag), HP and other DC switching vendors (Arista, Force10, Juniper) will be able to compete directly on “speeds and feeds.” This would not be good for Cisco. Once a company goes multi-vendor, Cisco will lose its pricing power across the majority of its product lines. This is a good thing for the industry. Buying 3Com makes that equipment line a viable solution for businesses. I did not have enough faith in 3Com as a standalone company to recommend their gear for my network. (As an aside, I do not have faith in Foundry, even after being acquired by Brocade)

What also must be said is that HP is taking a huge risk here. My organization is building a greenfield data center soon. Historically we’ve gone with HP for the majority of our compute power (C-class blade chassis) and Cisco for most of the network components. If HP wants an opportunity to bid for the network side of this project, we’ll also need to open the compute side up to Cisco. All’s fair, right? The issue with this is that the network opportunity is approximately one-third the size of the compute opportunity. HP is effectively risking X for a shot at X * 1.33, while Cisco is risking Y for a shot at 4 * Y.

2) HP’s large services organization could significantly hinder their ability to gain traction in the marketplace.

There are basically two types of resellers in the network marketplace. There are those that compete on price and attempt to shave a few percentage points of margin off every sale, and there are those who make their margins based on consulting or management services that are sold along with networking equipment. My organization generally prefers the former, as we are comfortable with our ability deploy complex networking solutions. Price is usually our deciding factor for equipment purchases, although we have made exceptions for new technology, such as WAN acceleration and Cisco’s ACNS video platform. The problem with utilizing the cheapest provider is that the purchaser and networking equipment vendor has to take responsibility for the network architecture, product selection and installation.

For organizations that have less network know-how, purchasing from resellers that can deliver the installation and management services required is a more attractive option. They can provide network designs and product selection advice. If the reseller has agreements with multiple equipment vendors, they can help determine which vendors' equipment best fits the customer’s needs. As a network engineer, these are the resellers I want to see in the marketplace, because they offer interesting jobs.

So what does this have to do with HP? As mentioned above, HP isn’t sure whether their deployment guides will be released to the general public, or for that matter, even to their resellers. This is ridiculous. If HP wants the world to adopt their product line, they need to do everything possible to educate the engineers responsible for deployments. If HP’s plan is to have their own services organization take the bulk of HP network deployments, they’ll lose all the value-added resellers. Those are the resellers they need the most, since they are the ‘trusted advisors’ for businesses. Without their support, HP will be forced to identify and win every networking deal on their own.

HSRP, vPC and the vPC peer-gateway Command

My recent post concerning my Migration from Catalyst to Nexus received a number of interesting and helpful comments.  One comment from routerworld caused me to do a bit of research into the “vpc peer-gateway” command.  This blog post is a summation of that research.

How HSRP Works

Hot Standby Routing Protocol is a well-known feature of Cisco IOS.  The goal of HSRP is to provide a resilient default-gateway to hosts on a LAN.  This is accomplished by configuring two or more routers to share the same IP address and MAC address.  Hosts on the LAN are configured with a single default-gateway (either statically or via DHCP).
Upon sending its first packet to another subnet, the host ARPs for the MAC address of the default gateway.  It receives an ARP reply with the virtual MAC of the HSRP group.  The IP packet is encapsulated in an Ethernet frame with a destination MAC address of the default gateway.  If the primary router fails, HSRP keepalives are lost, and the standby HSRP router takes over the virtual IP address and MAC address.  The host does not need to know that anything has changed.

In the diagram above, the user (10.1.1.100) is configured with a default-gateway of 10.1.1.1.  When the user sends its first packet to 10.5.5.5, it ARPs for 10.1.1.1.  In my example, Router A is the HSRP primary router, so it sends an ARP reply with the virtual MAC address of 0000.0c07.AC05.  The User PC then encapsulates the IP packet (destination IP=10.5.5.5) in an Ethernet frame with a destination MAC address of 0000.0c07.AC05.  Router A accepts the frame and routes the packet.
The above paragraphs tell the story of packets coming from the HSRP-enabled LAN.  But what happens to reply packets coming from 10.5.5.5 to 10.1.1.100?  The answer is simple, and intuitive if you follow step-by-step.  First, the Server creates an IP packet with a destination of 10.1.1.100.  It encapsulates it in an Ethernet frame and forwards it to its default gateway (for this example, let’s say it is Router A).  Router A strips the Ethernet framing and determines the next hop is on the local subnet 10.1.1.0/24.  It encapsulates the packet in an Ethernet frame with a MAC address of 0021.6a98.1952.  The source MAC address is the physical MAC address of Router A (0024.F71E.3343).  Router A does not use the virtual MAC address for packets it routes onto the local subnet.

So, what is vPC?

Now that we’ve covered HSRP, let’s talk about Virtual Port Channeling (vPC).  vPC allows two NX-OS devices to share a port-channel.  Attached devices believe that they are connected to a single device via an etherchannel bundle.  This is great because it eliminates spanning-tree blocking along parallel paths.
To allow this to work, the paired NX-OS devices use two vpc-specific communication channels.  The first is a vpc peer-keepalive message.  This heartbeat lets one switch detect when the other has gone off-line, to prevent traffic from being dropped during a failure.  These are lightweight hello packets.
The second communication channel is the vpc peer-link.  This is a high-speed connection between the two NX-OS switches that is used to stitch together the two sides of the port-channel.  If a frame arrives on switch A, but is destined for a host on switch B, it is forwarded across the peer-link for delivery.  All things being equal, it is undesirable to forward frames across a vpc peer-link.  It is much better for the frame to be sent to the correct switch in the first place.  Of course, there’s no way for the attached device to know which path is more appropriate.

In the above example, the User PC is sending an Ethernet frame to the Server.  It creates a frame with a destination MAC address of 0033.9328.12A1 and sends it to the L2 Switch.  The L2 switch has an entry in his forwarding table indicating that the destination MAC is accessible via the Port-Channel 100 interface.  It uses its etherchannel load balancing hash algorithm to determine which physical interface to forward the frame onto.  It is equally likely that it will choose the link to Nexus B, even though the more efficient path is to Nexus A (someday TRILL will help us, but for now there is no solution).  If the frame is sent to Nexus B, it will forward the frame over the vPC peer-link to Nexus A.

Cisco’s current recommendation is to build the vPC peer-link with multiple dedicated 10GE connections for performance reasons.  Cisco also recommends that all devices in a vPC-enabled VLAN be connected to both Nexus switches.  In the diagram above, the Server is considered to be a vpc orphan port.  This is undesirable, since it requires usage of the vpc peer-link.  It also has implications with multicast traffic forwarding.

vPC and HSRP Together

Now we’ve arrived at the point where we can pull all this information together.  In the following diagram, the User PC has been moved to a new VLAN.  The user is again trying to communicate with the server.

The User PC ARPs for his default gateway.  Nexus A (the HSRP primary) replies with the virtual MAC address of 000.0C07.AC05.  The user creates an Ethernet frame with a destination address of the virtual MAC.  It then forwards the frame to the L2 Switch.  The L2 Switch uses its etherchannel load balancing algorithm to determine the physical link to use.  The difference is now that it doesn’t matter which link it uses.  The NX-OS switch on the other end will accept and route the packet.  In effect, both Nexus switches are HSRP active at the same time.  This is eliminates the need to forward Ethernet frames across the vPC peer-link for packets that are destined for other subnets.

What Does “vpc peer-gateway” Do?

If we left everything alone, the story would be complete.  Unfortunately, storage vendors thought it would be a good idea to optimize their handling of Ethernet frames.  Some NetApp and EMC equipment ignores the ARP reply given by the HSRP primary and instead forwards Ethernet frames to whichever MAC address it receives frames from.  This is nonstandard behavior.
Using the diagram above, let’s assume say that the User PC is now a EMC Celera storage device.  The Server sends its packets (IP destination 10.1.1.100) to Nexus B, which routes them to the Ethernet LAN.  All IP packets with source IP 10.5.5.5 will be encapsulated in Ethernet frames with a source MAC address of 0022.5579.F643.  The EMC Celera will cache the source MAC address of these frames, and when it has IP packets to send to 10.5.5.5, it will encapsulate them in Ethernet frames with a destination MAC of 0022.5579.F643.  It is choosing to ignore its default gateway for these outbound packets.
I suppose the theory behind this feature was to eliminate the extra hop within the LAN.  When HSRP is enabled, it is necessary to disable ICMP redirects.  This means that the routers will not inform hosts on the LAN that a better default-gateway is available for a particular destination IP address.  This storage feature saves a LAN hop.
Unfortunately, this optimization does not work well with vPC.  vPC relies on virtual MAC address sharing to reduce utilization across the vPC peer-link.  If hosts insist on addressing their frames to a specific router, suboptimal packet forwarding can occur.  According to Cisco, “Packets reaching a vPC device for the non-local router MAC address are sent across the peer-link and could be dropped by the built in vPC loop avoidance mechanism if the final destination is behind another vPC.”  At the application level we saw very poor performance due to these dropped packets.  Enough of the packets got through to allow access to the storage device, but file load times were measured in the tens of seconds, rather than milliseconds.
The “vpc peer-gateway” allows HSRP routers to accept frames destined for their vPC peers.  This feature extends the virtual MAC address functionality to the paired router’s MAC address.  By enabling this feature, NX-OS effectively disables the storage vendors’ optimization.

Conclusion

If you are running vPC and HSRP, and you have EMC or NetApp storage equipment, you probably need to add the “peer-gateway” command under your vpc configuration.  The only caveat to peer-gateway is the following (from NX-OS 5.0 - Configuring vPC):

Packets arriving at the peer-gateway vPC device will have their TTL decremented, so packets carrying TTL = 1 may be dropped in transit due to TTL expire. This needs to be taken into account when the peer-gateway feature is enabled and particular network protocols sourcing packets with TTL = 1 operate on a vPC VLAN.

I have yet to face this issue, so my recommendation is to add this to your vpc configuration as a default.

Primo Hoagie Nutritional Information

(My apologies to anyone not interested in this.. it is completely off-topic from my usual network-related postings)

I recently decided to pay more attention to what I’m eating. My favorite hoagie shop is Primo Hoagie. I looked online for their nutritional information but I could not find it anywhere. The local shop was able to supply me with a printed sheet. There must be others who are interested in this information, so I decided to post it to my blog.

Primo-Sized Sandwiches (click for full-sized page):

Wraps (click for full-sized page):

A Brief Introduction to LISP

At Cisco Live 2009 I was introduced to the Locater/ID Separation Protocol (LISP).  I thought the idea seemed interesting, but I didn’t quite follow the practical purpose for it.  While planning my Cisco Live 2010 schedule I made sure to revisit this topic to get a better understanding of it.  After the breakout session and a few hours of experimenting, I believe I have a good feel for the issue LISP is attempting to solve, and the manner in which it intends to solve it.

By the way, I’m going to skip the requisite joke about the recursive programming language.  I will say that in college I spent a semester and a half programming in Scheme.  This first semester was an Introduction to Computer Science course, and wasn’t too bad.  The half semester was an advanced course called Artificial Intelligence.  It was a struggle for me to wrap my head around all the recursion.  When I reached the halfway point of the course, I asked the instructor about my outlook for a good grade.  She politely suggested I consider dropping the course :)

 

The IP Routing Problem

Before discussing LISP, it is useful to compare/contrast how DNS works versus how Internet IP routing works.  Both DNS and IP Routing deal with a large databases.  With DNS, the database is truly distributed.  End user DNS servers (for example, your corporate Internet-attached DNS server) are configured with specific names for their authoritative zones, plus enough information to allow them to look-up any other information they might need.  Random DNS entries (for example, www.cisco.com) are not pre-loaded into your DNS server.  If you need to resolve this name, the DNS server requests the information from an upstream server.  Caching adds some efficiency, but does not change the overall structure.  This system has allowed the number of DNS zones to scale well into the millions.

The database for IP routing is handled quite differently.  In the Internet’s Default Free Zone (DFZ), all routers must have the entire Internet routing database.  Summarization can help alleviate this requirement to a degree, but summarization also comes at the price of less accurate routing information.  The IPv4 routing database is currently 325,000 routes, give or take a few thousand.  Ultimately the IPv6 table will be as large, or more likely considerably larger, and the increased size of the address space will result in larger memory requirements.  And remember, this is high-speed router memory, not generic DRAM.  Wouldn’t it be great if we could transition IP routing from it’s current ‘replicated database’ model to a distributed database, like DNS?  As a matter of fact, that’s the goal of LISP.

 

Following a Connection in a LISP-Enabled World

Let’s go through a simple example in a fully LISP-enabled environment.  A user PC would determine the destination IP address of a web server via DNS and create a standard IP packet, with its own IP as the source and the IP of the web server as the destination.  The packet would be routed through the user’s LAN until it reached an Internet gateway.

 

At this point, the LISP-aware ISP router (in LISP-speak, the Ingress Tunnel Router, or ITR) would perform a lookup for the destination IP address.  An answer would come back with the IP address of the Egress Tunnel Router (ETR) for that destination.  The ITR would then encapsulate the user’s packet in a LISP packet, with a source IP of the ITR’s ISP interface and a destination of the ETR’s ISP interface.  This packet would then be sent through the Internet to the ETR.

The ETR receives the LISP-encapsulated packet, removes the header and routes the native IP packet (with the user’s PC as a source IP and the web server’s IP as a destination) into the local LAN, to the web server.  Return traffic from the web server to the user follows the same procedure in reverse (the web server’s ISP router acts as the ITR and the user’s ISP router is the ETR).

As a side note, the “Ingress” and “Egress” designations for Tunnel Routers are relative to the LISP-encapsulated tunnel.  The router that encapsulates a packet into LISP is the ingress router.

 

Tell Me Again… Why Is This Better?

On the surface this seems like extra work for little to no benefit.  Let’s dig a bit deeper to see how this helps each component in the path.

User PC – Nothing is different

ITR – This router only needs a default route to the Internet.  It is not clear from this example, but even if there is redundancy, in a fully LISP-enabled environment, only a default route is required.

Internet PE / P routers – This is where the magic happens.  The Internet PE and P routers only need routes for the ISP interfaces of the customer routers.  All packets between customers will be encapsulated, with the source and destination IP addresses coming from the WAN circuits.  Memory requirements are greatly reduced in these devices.

ETR – Only requires a default route to the Internet.

Web Server – Nothing is different

 

Drawbacks

So what are the drawbacks to LISP?  I can think of several:

MTU Issues – At its heart, LISP is a tunneling technology.  All tunneling technologies suffer from potential MTU issues.

Complexity – This paradigm is clearly different than what most of us are comfortable with.  But we didn’t enter this field thinking nothing would ever change, right?

Delay – The initial packet towards a new destination will likely get dropped, because the destination lookup takes time.  After the first packet the destination information is cached, so subsequent packets should flow without delay.  According to the Cisco LISP team, testing has shown this isn’t as big of an issue as it appears in theory.

A few optimizations have been suggested to deal with this delay issue.  First, a set of common destinations could be pre-programmed into ITRs (subnets associated with Google, for example).  Colin McNamara had a particularly interesting suggestion of ITRs performing DNS reply snooping, as it is highly likely that a DNS lookup will be followed quickly by an initial packet to that destination.  I’m not sure if this is being worked on, but it seems like a great idea.

 

What is Missing?

I glossed over the entire destination IP address lookup portion of LISP.  I will post a follow-up article describing this step.  For now, trust me when I say that it is not much different than DNS.

I also skipped over the other advantages of LISP.  For one, the ETR completely controls how inbound traffic is delivered to it.  If a destination IP address has multiple ISP gateways, those gateways can instruct ITRs to load-balance between destinations.  Experienced network engineers should immediately see the power of this feature.  This could signal the end of our rudimentary BGP-based load-balancing mechanisms (AS prepending, subnet splitting/disaggregation, etc).

A second benefit that LISP provides is its ability to send non-native traffic over a routed backbone.  In the example above I did not specify any of the IP addressing involved.  It is possible for the user PC and the web server to use IPv6, while the ISP network uses IPv4.  The ITR would receive an IPv6 packet from the user and perform a lookup which resolves to the IPv4 ISP address of the ETR.  The ITR would then encapsulate the IPv6 packet into an IPv4 LISP packet, which would then be sent over the ISP infrastructure.  When the ETR receives the LISP-encapsulated packet, it strips the header off and routes the IPv6 packet towards the web server.  This can also happen in reverse, where a pair of IPv4 speakers communicate over an IPv6 backbone.

Lastly, I completely bypassed the transitioning technologies to a fully LISP-enabled Internet.  They do exist, and it is very possible to deploy LISP incrementally.  We are long past the point of Flag Days.

 

What’s Next?

The LISP presentations I’ve seen have spent a lot of time describing the benefits of LISP for IPv4.  I’m not sure that is the correct place to focus.  As stated above, we are at about 325,000 IPv4 Internet routes.  At most I foresee a doubling in size of the routing table, to somewhere around 600,000 routes.  This factors in the usage of the 10% or so of remaining IPv4 space, as well as increased use of subnetting for load-balancing purposes.  I see the LISP protocol filling two needs.  First, it can be a great transitioning technology to IPv6, as well as a way to keep IPv4 alive over an IPv6-only infrastructure.  In an upcoming LISP blog post I will demonstrate how I am using LISP to reach the IPv6 Internet over an IPv4-only ISP.

Even more importantly, we are at the very beginning of the IPv6 route table explosion.  If the LISP team can get significant traction with LISP6, we can avoid the routing table bloat issue we’ve run into with IPv4.  Remember, the IPv4 routing table isn’t going anywhere, so we will only be adding to our problems with the deployment of IPv6.

If you are keeping track, I’ve promised two upcoming blog posts on LISP (my implementation experience and how the mapping database system works).  In the meantime, there are two useful resources online – lisp4.cisco.com and www.lisp4.net.  For a more scholarly take on LISP, see Petr Lapukhov’s article at iNE.com.

Migrating from Catalyst to Nexus

There are already a couple great resources on the Internet for network engineers who are migrating from a Catalyst-based data center to a Nexus-based data center. Two of my favorites are Cisco docwiki.cisco.com site, which hosts a bunch of IOS –> NX-OS comparison pages, and a page put together by Carole Warner Reece of Chesapeake Netcraftsmen. Both of these resources helped me quite a bit in working out the syntax of NX-OS commands. This blog post is an attempt to supplement that base of knowledge with my own experiences in converting a production data center over to the NX-OS platform.

My Catalyst network was a fairly standard data center design, with a pair of 6509s in the core and multiple Top-of-Rack switches cascaded below. We used RAPID-PVST, with blocking occurring in the middle of a TOR stack.

The new Nexus environment looks pretty much the same. We have a pair of Nexus 7010s in the core with a layer of Nexus 5020 switches at the edge. Each 5020 supports 4 – 6 FEXs. The FEXs only uplink to a single 5020 switch. This new network was built alongside the existing Catalyst one. The plan was to interconnect the two environments at the cores (at layer 2) and migrate server ports over whenever possible. Once we reached a critical mass of devices on the Nexus side of the network we planned to move the Layer 3 functionality from the IOS environment to the NX-OS side.

A few months ago we finished building out the Nexus LAN and interconnected it to the Catalyst LAN. We used vPC on the Nexus side to reduce the amount of SPT blocking. All was well, and we began migrating servers to the Nexus infrastructure without any issues. Eventually we reached our pre-determined “critical mass” and scheduled a change window to migrate the SVIs to the Nexus side and reconfigure the core Catalyst 6509s as Layer-2 only devices. The configuration work for this migration was around 1500 lines, so it was not by any means trivial, but it was also quite repetitive due to the number of SVIs and size of the third party static routing tables. Here’s where the fun began.

The first issue we ran into was with an extranet BGP peering connection through a firewall. In our design, we connect various third parties to an aggregation router in a DMZ. The routes for these third parties are advertised to our internal network via BGP, through a statically-routed firewall. Most of our third party connections also utilize BGP, so we receive a variety of BGP AS numbers. In two cases, the BGP AS number chosen by the third party overlaps with one of our internal AS numbers. To rectify this, we enabled the “allowas-in” knob on the internal BGP peering routers. Unfortunately this knob will not be available on the Nexus platform until NX-OS 5.1. I should have caught this in my pre-implementation planning. This was fixed with a small set of static routes. Our medium-term plan is to work with the two third parties to change their AS numbers, and eventually we will implement “allowas-in”, once we upgrade to NX-OS 5.1. Another interesting thing to note about BGP on NX-OS is that the routers check the AS-path for loops for both eBGP and iBGP neighbors. IOS does not do any loop-checking on iBGP advertisements.

With that behind us, we moved on to the SVIs and migrating our spanning-tree root to the Nexus switches. The SVI migration was trivial, but the SPT root migration caused issues. We were bitten by the behavior of Bridge Assurance, a default feature in NX-OS that was unavailable in our version of IOS (SXF train). Surprisingly, the lack of Bridge Assurance support didn’t prevent the Catalyst<->Nexus interconnect from working while the SPT root was on the Catalyst side of the network, but once we moved the root to the Nexus side, Bridge Assurance shut down the interconnects. The only acceptable solution to this issue (that I could find) was to disable Bridge Assurance globally on the Nexus switches. My error here is that I took for granted that my interconnect was properly configured, because it had been working for several months.

After encountering this issue I took another look at Terry Slattery’s blog post on Bridge Assurance, and at Cisco’s Understanding Bridge Assurance IOS Configuration Guide. The problem I experienced is Bridge Assurance requires switches to send BPDUs upstream (towards the root), while normal RSTP behavior is to suppress the sending of BPDUs towards the root. When the Catalyst side of the network contained the root, the Catalyst switches sent BPDUs downstream to the Nexus switches (normal RSTP behavior) and the Nexus switches sent BPDUs upstream to the Catalysts, which is abnormal for RSTP, but harmless. The Catalyst switches simply discarded the BPDUs. Once the SPT root was migrated, the Nexus switches sent BPDUs to the Catalyst (normal), and the Catalysts suppressed all BPDUs towards the Nexus switches (normal for RSTP, but not correct for Bridge Assurance). For the first few seconds, this was not a problem and forwarding worked fine, but eventually the Bridge Assurance timeout was reached and the Nexus switches put the ports into BA-Inconsistent state. The “right” way to solve this issue is to upgrade the Catalyst switches to SXI IOS and re-enabled Bridge Assurance. My preference is to simply retire the 6509s, so I’ll have to keep tabs on the migration effort. If it looks like it will drag on for a while, I’ll schedule the upgrades.

(Edit on 8/9/2010 - commenter "wojciechowskipiotr" noted that configuring the port-channels towards the 6500s with "spanning-tree port type normal" would also disable Bridge Assurance, but only for those specific ports. If I get an opportunity to try this configuration, I will report on whether it is successful.)

The remaining issues I faced were minor, but in some cases are still lingering or just annoyed me:

1) Static routes with “name” tags are unavailable. I had gotten into the habit of adding a named static routes to the network, especially for third-party routing. It appears that NX-OS does not support this.

2) VTP is unavailable. Based on conversations with other networkers, I’m probably the last living fan of VTP. I am sad to see it go. Fortunately in the Nexus environment there are fewer places to add VLANs (only the 5ks and 7ks).

3) Some of the LAN port defaults are different (when compared to IOS). For example, QoS trust is enabled by default. Also, “shutdown” is the default state for all ports. If a port is active, it’ll have a “no shutdown” on the config.

4) OSPF default reference bandwidth is now 40gb, rather than the 100mb value in IOS. This is a good thing, since 100mb is woefully low in today’s networks.

5) Proxy-arp is disabled by default. Our migration uncovered a few misconfigured hosts. Not a big deal, but it is noteworthy. Proxy-arp can be enabled per SVI, but do you really want to do it?

6) We ran into a WCCP bug in NX-OS 4.2(3). For some reason NX-OS is not load-balancing among our two WAN accelerators. Whichever WAE is activated last becomes the sole WAE that receives packets. I have an active TAC case to find a solution to this issue. For now, we are running through a single WAN accelerator, which reduces the effectiveness of our WAN acceleration solution.

I hope this helps someone with their own migration. This is going to be a common occurrence in our industry for the next few years, especially if Cisco has their way. If anyone has questions, please send me an email or post a comment.

Cisco Live 2010 - Tuesday and Wednesday

I thought I'd be able to keep up with this on a daily basis, but I was wrong. So here's a recap of the bulk of the conference. I'm sitting in a Virtualization Case Study on Thursday while writing this.

Picking up where I left off in my previous post, I attended the LISP session. It was a good refresher, as I had not followed up on the technology since my first introduction at Cisco Live 2009. I'll save my thoughts on the technology for a later blog post.

On Tuesday I attended an Overlay Transport Virtualization Breakout Session. It was a good compliment to my OTV Lab on Sunday. I felt that the presenter glossed over the significant architectural change that introducing Data Center Interconnect (DCI) brings to a network. He (rightly) stated that configuring OTV is fairly trivial, assuming you have the appropriate NX-OS and a pair of Nexus 7000 switches. Once you enable L2 adjacency between disparate DCs, there's no going back. On the plus side, if you need DCI, OTV looks much better than the alternatives (dot1q dedicated links, carrier VPLS, internal VPLS, EoMPLSoGRE, L2TPv3... I'm sure I am missing a couple).

The John Chambers keynote was good as usual. The introduction of the Cisco Cius was probably the highlight. I haven't investigate it yet, and probably won't get around to looking into it in detail. I have to confess that I'm not terribly interested in technology. I don't personally own any Apple products and I've only carried a Blackberry smart phone for three years. Probably the 'coolest' technology that I own is an original Kindle, which meets my nontechnical reading needs. For that matter, I only have a single Cisco router (and no switches) at my home. Somehow I manage to keep up, so don't let anyone convince you that you need racks full of equipment to participate in our industry.

My one complaint is that I always feel like my organization is 5 - 10 years behind when I leave the keynote. For example, the Cisco Live 2008 keynote was centered around complete virtualization. Two years later, we're nowhere near there. I know the point of the keynote is to show the future direction of Cisco, and I wouldn't want anything to change, but I don't like the feeling of being behind. I always leave the session with a sense of urgency to get us back on track. This year I brought my manager, so hopefully he'll feel the same way and we can get things moving a bit faster.

After lunch on Tuesday I attended a second Panel on Experiences with Deploying IPv6. I am finally convinced that my employer needs to begin the transition. This was a great session. Presenters from Comcast, Google, Microsoft and Tata described their IPv6 deployments and gave recommendations. For once, Apple took a bit of a beating (and not from the Microsoft presenter) because no Apple OS supports DHCPv6. This requires the use of SLAAC for acquiring IPv6 addresses in mixed PC environments. It's not a show stopper, but flexibility would be nice. I prefer SLAAC anyway for the simplicity, but I am firmly in the minority, based on a show of hands during the session.

On Tuesday evening I spent a few hours with John Chambers and my ~75 closest CCIE/CCDE friends. This was my first year attending the CCIE/DE NetVet reception. It was quite an experience. John Chambers took questions for about 1.5 hours, and gave very direct responses to some difficult questions. I received some great information that will help me position my organization for upcoming shifts in Cisco strategy. While I'm fairly confident this was information I could acquire elsewhere, when it comes from the CEO, it is reliable.

After the NetVet reception I spent an hour or so attending the CCIE party at the Voodoo Lounge. I didn't stay long, but I did get a picture of the cool (both literally and figuratively) CCIE logo ice sculpture. I would have preferred the original laurel leaf router logo, but I was still impressed.


On Wednesday morning I attended "IS-IS Network Design and Deployment". I was looking forward to this session more than any other, as I have had great difficulty getting this content from other sources. There are a few books that cover IS-IS, but Cisco Live Virtual has never had the audio for this session. The slide deck I found in the past (and the one used during this presentation) lacks sufficient detail to reconstruct the information. The presenters did not disappoint, and I took about four pages of notes (the only notes I've taken at Cisco Live 2010). I intend to write a separate blog post on why IS-IS can no longer be ignored by Enterprise-focused networkers. If this describes you, you'll have a year or two to prepare.

After lunch I attended a session called "Designing Multipoint WAN QoS". This session didn't stand out in the session list when I was registering, but I was tipped off by a friend (the presenter) that it would be a good session. He didn't disappoint. I am almost embarrassed to say that I had never given much thought to the problem this session attempts to solve. It certainly deserves an individual post, but I'll describe it in a few sentences here. The best example of the issue is the Teleworker problem, where the remote user has a small Cisco router with a VPN tunnel to a Corporate headend router. The user also uses the same ISP connection (DSL, cable, etc) for home Internet usage. The problem is how do you ensure that the ISP connection prioritizes real-time Corporate traffic over general Internet web traffic downstream (towards the remote user)? To put it more generally, how do you get outbound QoS from a Service Provider network that is not QoS-aware?

My last session on Wednesday was Enterprise IPv6 Deployment. I've seen this content before, and originally I had scheduled a DCI session (DCI w/ Advanced-VPLS), but I thought it would be best to reinforce the IPv6 content. The presenter (Shannon McFarland) is very good, so I was not bored by seeing some of the information for a second time. His recommendation to begin in the core and work out towards the edge is somewhat unconventional, but he defends it well, to the point that I was convinced. Maybe more contemplation will change my mind. More information to come in the future. The biggest point he wanted to get across was "Dual Stack Where Possible, Tunnel When You Must." Tunnels introduce MTU issues and create difficult-to-troubleshoot network architectures. Maybe he should talk to the TRILL, LISP and OTV groups! :)