Monday, August 31, 2009

Recap of Cisco Certified Architect Update

On Thursday I attended a Cisco meeting on the CCDE and Architect certification programs.  There was not a lot of new information offered, so some of this has already been discussed on the blog.

 

Timeline

October 2009:  Blueprint available on Cisco Learning Network

November 2009: Application process open on CLN

January 2010: First board review availability

Candidate Application Process

To apply, you need to be an active CCDE.  The next step is to submit a resume, which should demonstrate 10 years of networking experience and some architect/design responsibilities (not necessarily 10 years of these responsibilities).  The candidate also needs to submit a Project Summary for an architect or design project that he/she led.  During Q & A, I asked if there would be a specific format for this.  That hasn’t been determined yet, but I don’t see how the application review team could handle this without a specific format, since we all use unique project documentation.  The resume and project summary will be reviewed, and questions will be developed to be asked during a short (15 minutes?) phone interview.  This will allow the application review team to be certain the candidate is qualified to take the exam.

 

Test Pre-Work

Assuming the candidate has passed these checkpoints, he/she will submit payment and receive the project documentation / test materials.  The candidate will have a few weeks to develop an RFP-style response, which will consist of a Functional Specification, High-Level Architecture Diagram, an outline of the Board Meeting presentation and a few other documents that I failed to note.   The presentation used during the meeting is not yet available for download, so I can’t look up the missing documents at this time.  This documentation will be submitted in advance of the board review, to allow the board to develop questions based on its contents.

 

Board Review

The board review is scheduled for six hours.  No mention was made of where it will be, or if Telepresence will be used for the reviews.  There seemed to be a hint or two of using technology during this process, and I believe the initial board is physically located in separate offices, so I’m expecting a Telepresence presence, so to speak.  The timeline for the review is:

1 hour – Present design solution to Executive Team

1 hour – Respond to questions from Design Team

15 minutes – Presented with “What If” change to original architecture challenge

2 hours, 45 minutes – Create solution based on new “What If” requirements

1 hour – Present and defend new solution

Out of the six hours, three hours are spent presenting or defending solutions.

 

Other Notes

- I don’t believe anyone mentioned when the candidate will receive a pass/fail result, or what sort of feedback is expected.  As a potential candidate, I’d like feedback with either result.  Even successful candidates have room for improvement.  That’s one of the negatives to receiving only a “Pass” result to the CCIE/CCDE exams.

- Someone from Cisco on the call suggested that there would be around 100 successful Cisco Certified Architects in year one.  I’m sure most of the call attendees (and a few of the presenters!) did a double-take at that news.  One of the Q & A questions bluntly asked “If there are 7 current CCDEs, how can you get 100 Architects?”  The response was that the CCDE program is expected to pick up steam over the next year, and many CCDEs are expected to immediately pursue the Architect certification.  I have my doubts.. my guess is 15 in 2010.  I don’t think there have been 100 unique CCDE candidates to date.

- There will be a recertification process, and it will likely consist of contributing to the program via content development or serving on future review boards.  It’s a good strategy, as developing this content will be highly time intensive.  It will not be possible to ramp up this certification program to hundreds of candidates without significantly expanding the certification team.  This should be a cost effective method of accomplishing that.

- I was a bit surprised at the lack of questions from the meeting participants.  Perhaps everything is too new at this point.  I was going to also ask about the expected pass rate, but I doubt they have a target, so it didn’t seem worth the time to me.  The only notable question was about the cost, which I don’t believe was addressed in the presentation.  The $15,000 price tag was confirmed, with some mention of it being comparable to Microsoft’s top-level certification.  I have no knowledge of their programs, so I’ll take the Cisco team at their word on it.

- The presentation is supposed to go up on CLN soon.  It went into nice detail on the Candidate Objectives, so it will be nice to see it on my own time.  That part of the presentation went a bit too fast for me to absorb all the info.  If I had known in advance, I would have taken some screenshots!

- The presentation team went into a bit of detail on the differences between Network Designers and Architects.  Most people I know (including me!) use the terms interchangeably.  The basic point being made was that Network Architects take business requirements and turn them into functional network specifications.  Network Designers take the specifications and create a network.  I don’t know of anyone who functions as an architect without also creating the network design.  I certainly know of engineers who only perform the second step.  So who creates the functional network specifications for them?  In most cases, it isn’t really created.  I have been guilty of this.  I’m told of a business problem that needs to be solved, and almost immediately I fire up dynamips and configure a solution.  I would like to think that intuitively I’ve thought through the architecture and concluded that my solution is the best one, but without explicitly going through the process, I can’t truly know that I am correct.  Over the last year, I have made a point of incorporating architecture/design reviews into my project planning, and asking for more input from project team members.

Wednesday, August 19, 2009

CCDE Written and Practical Study Plans

In a recent blog post I made mention of creating a structured study plan and following it. To demonstrate this, I dug up my CCDE written study plan.

Background

I learned about the Cisco Certified Design Expert program on the shuttle ride from the airport to Cisco Live 2007. By chance, I was on the same shuttle as a Lee, a former co-worker that I hadn’t seen in seven years. He mentioned that there was an invitation-only announcement of a new expert level certification taking place at Networkers. As I had not received an invitation, I didn’t know anything about it. While wandering around the show area on Monday, I ran into David Bump, who was involved in the CCDE launch. He kindly extended an invitation to the kick-off meeting so I wouldn’t feel left out.

The attendee list in that meeting was impressive, to say the least. I saw a number of well-regarded network architects, and more than a couple friends and former colleagues. It was clear that Cisco invited the right group! The presentation and handout made me realize that I had a number of holes in my knowledge that would need to be addressed if I wanted a chance at success on the CCDE written beta exam.

Creating My CCDE Written Study Plan

My first step in creating a study plan was to make a duplicate of the handout, as I didn’t know if I would be able to get another copy. It’s now on-line at http://www.cisco.com/web/learning/le3/ccde/ccde_exam_information.html, so this step is no longer necessary. On my copy, I highlighted the areas that I felt were weaknesses. Unfortunately, it seemed like most of the paper had been highlighted. Who is Paul Baran* anyway? ;)

After analyzing the areas, I grouped the technologies into seven major topics, and I assigned a level of confidence in my abilities. I also planned to address some of the other blueprint topics, like Network Management and Security, but these were the topics I felt I should focus on:

OSPF Medium
EIGRP Medium-High
IS-IS Low
BGP Medium-High
Multicast Medium-High
MPLS Low
Quality of Service High

Next, I figured out what resources were available to me. I purchased several books from the CCDE Written Exam Reading List (Optimal Routing Design, Network Management Fundamentals, BGP Design and Implementation) and pulled a few classics off the shelf (Routing RCP/IP Volume 1, Developing IP Multicast Networks). I also had access to Cisco Live Virtual, and of course the RFCs were readily available. I mapped these resources to my topics, and determined how much time I had available to devote to each one. This is the result:

CCDE Study Plan:

66 hours total

OSPF (10.5 hours)

Routing TCP/IP OSPF Chapter - 2 Hours
OSPF Tech Notes - 4 hours
OSPF Handbook - 30 minutes
RFCs - 2 hours
Review - 2 hours

EIGRP (5 Hours)

Routing TCP/IP EIGRP Chapter - 1 hour
EIGRP Tech Notes - 2 hours
EIGRP Handbook – 30 minutes
Review - 1.5 hours

IS-IS (10.5 hours)

Routing TCP/IP IS-IS Chapter - 2 Hours
IS-IS Tech Notes - 4 hours
IS-IS Handbook - 30 minutes
RFC - 2 hours
Review - 2 hours

BGP (10 hours)

BGP Tech Notes - 4 hours
BGP Handbook - 1 hour
BGP Book - 3 hours
Review - 2 hours

Multicast (5 hours)

PIM RFC - 2 hours
Multicast Book – 1 hour
MBGP on CCO - 2 hours

MPLS (20 hours)

Intro to MPLS presentation - 2 hours
Other presentation? - 2 hours
Books - 10 hours
BGP/MPLS - 2 hours
Cisco website - 4 hours

QoS (5 hours)

RFCs - 3 hours
Translation to Transport Layer - 2 hours

I can’t say that I stuck completely to this schedule, but it served as a guide for my efforts. It allowed me to focus time on my perception of my weaknesses. As is usually the case, I did better on my ‘Low’ topics than I did on my ‘Medium-High’ ones. I try to remind myself to at least brush up a bit on my strengths, but life often gets in the way of studying, and I naturally concentrated on the less familiar topics. It’s the topics that I work on every day that give me the most trouble, because I consider myself an expert, but when I really think about it, I’m only an expert on the portions of the topic that I use regularly. For example, I was certain I knew Quality of Service cold.. but did I know how ToS gets mapped into MPLS EXP? No, because I didn’t run MPLS in my network. It’s a trap I fall into regularly when studying.

CCDE Practical Thoughts

To the group of engineers attempting the CCDE Practical on August 26th, good luck! I’d like to see a good number of successful candidates. I hope to hear that someone cracked the 50% mark, which I believe has not yet been breached in this exam. Like the CCIE exam, successful candidates do not get a graded score, only a PASS result, so this is a rumor, not a fact. My opinion is that passing this exam has been too difficult, and I fear that if the exam gets labeled as ‘impossible’, the certification program will lose its candidates, and ultimately its market value. I know a few of the unsuccessful candidates (personally and by reputation), and I can say that they are undoubtedly good network design engineers. If the goal of this certification program is to identify and recognize engineers of their caliber, it must be missing the mark, at least by a small degree. Perhaps they’re not good test takers, but it concerns me when obviously qualified candidates are unsuccessful. That said, we’re still in the very beginning of the life cycle of this certification. I am confident that it will be properly tuned to stop great candidates from failing.

CCDE Practical Advice

Unfortunately, I didn’t create a good study plan for the CCDE Practical, or I would share that as well. The two books that I think were most helpful in preparing for it were Optimal Routing Design and Definitive MPLS Network Designs. If I felt that Quality of Service was a weakness, I would have concentrated heavily on RFC 4594, Configuration Guidelines for DiffServ Service Classes and perhaps End-to-End QoS Network Design (but skip all the configuration; it’s not important for this exam).

The key to the CCDE Practical is to know the technology! I don’t believe it is possible to pass this exam if you have holes in your tech stack, so to speak. The exam is scenario-based, and if you run into a scenario based on IS-IS, and you don’t know the protocol, it would be practically impossible to make up the points in another scenario. It’s not that the exam is intended to test your technical knowledge; that’s not the primary goal. The issue is that is uses the technical blueprint as a basis for asking network design questions. You need to be able to speak the same technical language as the exam creator. For those with a programming background, it would be like testing application design skills using Java, when you only know C++ and Pascal. Before you ask, yes, my programming knowledge is that out of date.. feel free to substitute relevant programming languages if you retell this analogy :)

If you have the technology covered, the rest is pretty much intuition. There’s a certain feel to a good network design, as opposed to inefficient ones. The open-ended questions are very difficult, as the right answer isn’t staring at you from the options list. Also, because you can’t go back and change answers, you will certainly answer a question, click ‘Next’, and immediately learn you got the question wrong, based on the wording of next question. It happened to me at least once, and probably several times. Don’t let it get you down. As I mentioned above, I don’t think anyone has scored 50% on this exam yet. Answering incorrectly is part of the process. If (when) this happens to you, rethink the previous answer, and change your course of action from here out if you feel you were incorrect. Don’t be stubborn. There are no points available for being consistently wrong!

Again, good luck to all who are attempting this exam, whether it is in August, December or a future date.

Jeremy

* Paul Baran is the father of packet-switched networking.. something I’m almost embarrassed to say I didn’t know prior to studying for the CCDE.

Monday, August 17, 2009

Corporate Versus Consulting Jobs

One question that repeatedly comes up in conversation is whether corporate or consulting jobs are better.  Once I get past the obvious ‘It Depends’ response, there are some clear differences that aren’t necessarily apparent at first glance.  Here is my take on the question.

 

Career Versus Job

I’m definitely not breaking new ground when I say there is a clear distinction between a career and a job.  My chosen career is computer networking (or less specifically, Information Technology), and my current job is a network manager / architect for a Fortune 500 insurance company.  I’ve had several jobs in my career, but only one actual career.  Your career is generally the answer to “What do you do?”, while your job is the answer to “Where do you work?”  When I’m asked, my answers are “I build and maintain computer networks” and “I work for an insurance company” respectively.  It is extremely important to put the emphasis on your career.  Often times, focusing on your career and focusing on your job go hand-in-hand, but occasionally they will diverge.  For example, after I earned my CCIE certification, I sat down for my first performance review.  My manager, who I respected very much, was thrilled with my performance, but could only offer a modest increase in compensation on a low base salary.  He mentioned that the plastics industry (my employer’s field) was in bad shape, and he would have offered more if it was possible.  I understood the predicament, but later I realized that I didn’t work in the plastics industry.  I work in the computer networking industry, and things were going quite well there.

Don’t take this advice to an unwise extreme.  Demonstrating a history of indifference to your employer will undoubtedly decrease your value over the long term.  It would be very difficult to find a suitable job if your resume is long on experience but short on loyalty.  As a manager, I can easily understand early career applicants showing a few employers, especially if the increase in responsibility and experience demonstrates a logical pattern.  If the pattern continues into the applicant’s later career, I’d be inclined to wonder why things weren’t working out with so many different organizations.

 

Categories of Networking Jobs

I broadly separate my employment history into two categories:  Corporate positions and Consulting positions.  Within the networking field, corporate positions entail working in a specific environment to solve problems for your organization.  You don’t necessarily need to be a full-time employee of the organization; I’ve had corporate jobs where I was a contractor.  Consulting positions usually have all the responsibility of corporate positions, plus an extra layer of organizational overhead.  To be truly successful in a consulting position, you’ll need to focus on the profitability and success of your consulting organization first, while also fulfilling the needs of your customer (the corporate entity).  I’m leaving out reseller positions, primarily because I’ve never had one, so I haven’t given much thought to the environment.  I’ll theorize that it is similar to consulting, with the added pressure of pushing products (physical, rather than labor) on the customer, but that’s just a guess.

I spent three years (1998 – 2001) in consulting, and nine years in corporate environments.  My consulting roles were during the heyday (and later the crash) of IT consulting.  Maybe things are different now, so keep that in mind while considering my thoughts below.

 

Points of Comparison

Suitability for Early Career

I’ve frequently questioned the suitability of consulting for early career engineers.  At my local Netigy Corporation office I saw the drawbacks of hiring junior engineers.  They’re often the first to go during tough times, and without suitable assignments, it is very difficult to learn new skills.  Most engineers need meaningful work to build their skills, and at times, it can be a challenge to find that in a consulting environment.  Consulting organizations are hesitant to place an engineer in a role where they haven’t already proven their abilities.  Corporate environments generally provide better opportunities to acquire and utilize new skills.  There is also a great advantage to implementing a solution to a problem and then seeing how it works over the long term.  This is more likely to happen in a corporate environment.

Compensation

Perhaps the best way to understand compensation is to think about what makes an engineer valuable to an organization.  In consulting, I’d say about ninety percent of your value is tied to your revenue (bill rate X utilization).  I was once directly told that my compensation had hit a peak because my bill rate didn’t support an increase.  This was despite my efforts to grow my account from two to seven consultants, often at significantly higher rates than I was billing.  More sophisticated consulting companies will factor this in, plus mentoring / leadership, pre-sales work and other factors, but ultimately profitability comes down to individual revenue versus expense.  Compensation is not as easy to understand in a corporate environment.  You can look at replacement cost (how much would it cost to higher a new engineer for the same role), corporate salary structures, value delivered, etc, but it’s less quantifiable than in consulting.

It has become apparent to me that corporate salary structures are getting much closer to the consulting environment.  In the late 90s, I found it necessary to enter consulting to achieve the full value of my experience.  Even in 2001, when the consulting market was beginning to crumble, I couldn’t justify moving back to the corporate world on the basis of compensation.  I chose to stick with Netigy until they closed their doors in September 2001.  Only then did it make sense for me to move back to the corporate world, this time as a contractor.  In talking with other engineers who have recently changed jobs, I can see that the pay disparity has shrunk considerably, and when you add in the value of non-cash benefits, we may be nearing parity.

Job Security

In consulting, your job security is dependent on your ability to bring in revenue.  There’s always a place for cash-flow-positive consultants.  In the corporate world, again, things are not so clear.  Because there is no direct correlation between job performance and revenue, job security comes down to relationships, replacement value, the organization’s view on the value of IT, and countless other factors.  You also need to look at the viability of the organization.  If your employer (whether consulting or corporate) isn’t able to keep the doors open, it doesn’t really matter how good you are at your job.

I prefer to look at job security more broadly, in terms of employability.  It never much mattered to me if my current job was secure, as long as I could be confident that I would find another job in a reasonable amount of time.  This viewpoint allowed me to find a wonderful opportunity when Netigy went out of business.  If I had been focused on my specific job’s security, I would certainly have left Netigy for another organization before they went bankrupt.  It also served to calm my nerves a bit in March 2009 when the stock market seemed to indicate that my current employer would not make it through the financial crisis.  If this viewpoint appeals to you, be sure to save some money.  I’m sure you can find better financial advice from others, but I suggest having a considerable amount of available cash to handle the inevitable periods of unemployment.

Relationship Building

Without a doubt, I met more people from the networking industry in my three years of consulting than I did in nine years in the corporate world.  Some of the relationships I’ve built have lasted for a decade or more, and several have resulted in jobs for both me and my counterparts.  In fact, I’ve dragged one guy between three different companies, and I wouldn’t hesitate to call him again if the need arose.  In the corporate world, it is easier to create lasting relationships, but I still suggest acquiring the ability to quickly build rapport.  Use social networking (Twitter, LinkedIn, Cisco Learning Network, etc) and get to know people in training classes and seminars.  Don’t think of this as creating a network to use later, but as a way to add a social component to your career.  No one wants to be used, and if that’s your intention, it will be obvious.

Work/Life Balance

Bear in mind, I’m coming at this category as a married father of three boys.  What I want out of this category may be wildly different than your needs.  In my experience, a corporate position provides the better work/life balance.  I am sure there are corporate positions that require travel, and there are consulting positions that allow you to be home every evening.  I’ve done the constant travel thing, and it has a few benefits:  Frequent flyer / hotel points, plenty of study time, seeing new places, etc.  If I weren’t a family man, I would probably find even more great things about the traveling lifestyle.  Even in my corporate positions, I’ve found opportunities to do some travel, and it has been a nice change of pace.  My worst work/life balance issue in the corporate world was the three years of long (90+ mile each way) commuting I signed on for.  And of course I still get to ‘carry the pager’ for on-call work and escalations, as well as work the strange hours that our change windows dictate.  In return, I get to work from home and put my kids on the school bus in the mornings.

These are rather gross generalizations of the two environments.  I know of one consultant who has worked almost exclusively with the same government client for a dozen years.  He lives near his client and seems to have worked out a great work/life balance for himself.  I have also worked for a consulting company that strived to keep everyone local.  Every situation is unique, so don’t rely on these generalizations.  Do your homework if you are interested in changing jobs.

Need for Certification

Consultants need credentials to find work.  Consulting organizations use certifications in their marketing and sales pitches.  It’s a required part of the game, and it benefits the consultants, as their value in the industry increases because of it.  I once pursued my CCIE in WAN Switching to help out my consulting employer.  I got as far as the CCNP and CCIE written, and was scheduled to take the lab when I got sidetracked.  It assisted us in landing a significant engagement and allowed me to broaden my horizons a bit.  I only rarely used the knowledge, and by now, it’s almost completely gone, but it was a great experience.

As I mentioned in a previous post, corporate employees should also acquire certifications.  It makes life significantly easier when you find the need to market yourself via a resume.  Don’t shortchange yourself, especially if you can a supportive employer who is willing to assist you in achieving your goals.  If you are having trouble selling the idea of getting certified to your employer, remind them there are benefits for them too.  My pursuit of the CCDE cost my employer approximately $3000, including travel, books and registration fees.  In return, they got dozens of hours of evening/weekend study time, and in my opinion, a more qualified Network Architect.  All for less than the cost of a one-week training course.  If that doesn’t convince them, remind your employer that CCIEs get priority escalation for TAC cases, which will cut down on MTTR.

 

Additional Thoughts

I certainly don’t want to come across as recommending that you should always chase the money.  A friend commented on my last post that there are other factors to job/career satisfaction other than money, and he’s 100% correct.  I would not take or stay at a job that didn’t meet my work/life balance requirements, even if the money was excellent.  As a matter of fact, I worked with David at a previous employer, where I left for this very reason.  He doesn’t remember, but in fairness, I was only there for six weeks before I realized the fit wasn’t right.

David made a second point that bears repeating.  Non-technical skills are very important, and need to be developed alongside technical ones.  It’s a bit more difficult to build that into a study plan.  My recommendation is to emulate successful people at your job.  If you don’t see any, well, you’re probably not with a very good company and you should be considering whether you’re at the right place.  Most people are happy to provide advice and mentoring.  It’s one of the best parts of my current job.

 

Summary

Again, remember that my consulting experience is from about a decade ago.  Things may have changed significantly since then.  If they have, please add your thoughts in the comment section so I can be more accurate in future posts.  As for me, I’m reasonably happy with my current job.  It has given me the flexibility to pursue my career interests, including management.  I have also been able to do a bit of consulting to maintain skills in technical areas that don’t exist in my employer’s environment.

Hopefully these descriptions will help you make the right career decisions.  Keep in mind, if you ever do make an unwise career move, you can always make another move to get back on track.  I don’t know many people in this field who haven’t made a bad career move at least once...  I know I have!

Monday, August 10, 2009

Career Advice From a Networking Veteran

It’s hard to believe I’m a veteran of computer networking, but the facts are indisputable:

X

Performed an IGRP to EIGRP Conversion on a Production Network

X

Significant Work on a Token-Ring Network (Not an Ethernet Conversion!)

X

Passed CCIE Lab When it Was Two Days Long

X

Holding Shares of CSCO With $60+ Per Share Cost Basis

(Sadly, the Cisco shares are in my son’s account. I guess I’ll owe him a few dollars out of my bank account when he’s ready to spend the money to make up for that decision!)

While I don’t feel too old, I must have enough life experience to ‘give back’ to the next generation of Networkers. So here’s my attempt to help out.


Early Career

If I could make only one recommendation to early career network engineers, it would be to get certified. CCNA, CCNP, whatever.. The job market is too crowded to not have some sort of credentials attached to your resume. I’ve been on both sides of the hiring game. Without some sort of certification, you need an extremely strong social network to get your foot into most doors. There are too many HR-provided ‘resume filters’ in place now.

One of my most recent hires came to us without any certifications. In almost all circumstances I would have dismissed the resume quickly. Fortunately for us, the candidate and I had worked at the same consulting firm a decade ago. Although I didn’t know him personally, I was able to reach out to a former colleague, who contacted the candidate’s former manager. After an excellent recommendation, I was willing to put my trust in the candidate’s resume and after a series of positive interviews, we made a great hire. If it were not for the common former employer, I doubt we would have had the confidence to make the job offer. Our organization would have been worse off for it, but the costs associated with making a bad hire are such that I needed to be extremely confident to make an offer. My assumption is that if you are in your early career, you haven’t had time to build the necessary social network to overcome the lack of a certification.

A second recommendation is to immerse yourself in the subject matter. Create a structured study plan and execute it. When I began working in this industry in 1997, I knew very little about networking. I was put into a ‘sink or swim’ situation, as the two engineers who previously managed the company’s network had both left with little notice. I started in June of 1997, and went to the ICRC and ACRC at Chesapeake Computer Consultants within the first two months. With those courses as a base, I decided on a study plan of reading the (then current) IOS 11.2 Configuration Guides from beginning to end. There weren’t a lot of study materials available at the time, so I made the best of the situation. As a college student finishing up my Senior year, I would print thick sections of the documentation and bring it to my classes to read. I don’t know that this is the best study method today, given the abundance of targeted study materials available from Cisco Press and others, but it worked for me at the time. I ultimately passed the CCIE lab using this method. For a current example of a structured study plan, I suggest you follow Aragoen Celtdra’s Route My World blog. I used a similar method when I pursued the CCDE written and practical exams.

Mid Career and Later

The studying and reading only intensifies once you achieve your CCIE certification. If only someone had mentioned this to me in 1998! Like all successful CCIE candidates, I was on top of the world after passing the lab. I was sure I was an expert my field; after all, that’s what the ‘E’ stood for! I decided to interview for a position with Chesapeake Computer Consultants. That’s when I found out how much I didn’t know! While I did ultimately received a job offer, I was astounded by the depth of the technical interviews. It was at that time that I realized my chosen profession required constant study to maintain relevance. The CCIE recertification process reinforces this, and that’s why I don’t have any issues with sitting for a written exam every two years.

My recommendation is to constantly seek positions which challenge you, and hesitate before trading opportunity for money. I spent over three years in a lucrative contracting position, but during this time I didn’t learn very much. I sort of missed the MPLS revolution, and I’ve been catching up ever since. I even took the easy way out on my CCIE recertification by repeatedly taking the CCIE WAN Switching exam, which didn’t appear to change at all from 2000 until 2003. Money is an important thing in life, so while I would never advise someone to turn it down, I would highly recommend weighing a slight increase in current dollars to the impact a position will have over your career. I didn’t do irreparable harm to my career, but I certainly set myself back a bit by narrowly focusing on older technology. On the plus side, I worked with an incredible set of people and built wonderful relationships that eventually delivered me to my current position.

A more controversial suggestion is to strongly consider changing employers as you work your way up the salary ladder. I have seen first hand that it is extremely difficult for an employer to keep up with high performing employees. Most corporate salary structures are not designed to cope with the rapid increase in value delivered by such employees. I found it necessary to leave my corporate employer after achieving my CCIE certification for this reason. I ultimately decided that although it would be difficult to leave, it was unfair to my wife and child to work for less than market value. This was perhaps the most difficult professional decision I’ve made thus far, but in retrospect, it was clearly correct. This is less true in the consulting world, where it is easier to judge an employee’s value. Under most circumstances, bill rates are a strong proxy for value. Consultants can easily demonstrate their value by increasing the revenue delivered to their employer. For a fun example of this, take a look at this Game Theory view of salary negotiation: http://mindyourdecisions.com/blog/2009/08/04/how-to-negotiate-a-pay-raise-with-game-theory/.

Summary

So, to recap, get certified, study methodically and regularly, chase money and opportunity in your early career, but go for balance.. don’t overvalue either. Also, make sure you build lasting relationships along the way. It ultimately makes everything you do more meaningful, and adds a fun social component to trade shows and seminars. My favorite part of Cisco Live is catching up with friends and former colleagues. If you see me in Las Vegas at Cisco Live 2010, please introduce yourself!

Monday, August 3, 2009

When Administrative Distance Doesn’t Count

In troubleshooting a network issue, I came across an interesting situation where EIGRP and OSPF metrics are directly compared. Obviously, this is a flawed design, and it has been updated to fix the issue. I thought this might be a good opportunity to share the experience.

Problem
Traffic is not taking the optimal path from RouterE to 10.34.0.0/16. Here is the topology:


Traffic from RouterE to 10.34.0.0/16 should go to RouterD, then to RouterY, and then to 10.34.0.0/16. What we're actually seeing is this traffic take the less-optimal path of RouterE->RouterC->RouterD->RouterY.

Cause
Here is the output of 'show ip route 10.34.0.0' on RouterE. We should see the next hop as "10.0.3.1, from 10.0.1.11", but here is what we see instead:
RouterE#show ip route 10.34.0.0
Routing entry for 10.34.0.0/16
Known via "bgp 65000", distance 200, metric 4294967294
Tag 65001, type internal
Last update from 10.0.4.1 2w1d ago
Routing Descriptor Blocks:
* 10.0.4.1, from 10.0.1.20, 2w1d ago Route metric is 4294967294, traffic share count is 1
AS Hops 1
Route tag 65001

So the question is, why would RouterE prefer to go all the way to RouterA to get to the 10.34.0.0/16, when there's an equally good (and closer!) path to the network off of RouterD. To find the answer, we need to look at the BGP table for 10.34.0.0/16. Here's that output:
RouterE#show ip bgp 10.34.0.0/16 BGP routing table entry for 10.34.0.0/16, version 107467
Paths: (2 available, best #1, table Default-IP-Routing-Table)
Advertised to update-groups:
2
65001, (aggregated by 65001 10.1.34.1), (received & used)
10.0.4.1 (metric 13) from 10.0.1.20 (10.0.1.20)
Origin IGP, metric 4294967295, localpref 100, valid, internal, atomic-aggregate, best
65001, (aggregated by 65001 10.1.34.2), (received & used)
10.0.3.1 (metric 26112) from 10.0.1.11 (10.0.1.11)
Origin IGP, metric 4294967295, localpref 100, valid, internal, atomic-aggregate

I highlighted the key information, which is the IGP metric to reach the BGP next hop. This is the eighth point of comparison for BGP routes, but the seven more important comparison points are all equal (Click here for BGP Path Selection Process). Notice that 10.0.4.1 is only metric=13 away, while 10.0.3.1 is metric=26112 away. Where do these values come from? Here is that output:
RouterE#show ip route 10.0.4.1
Routing entry for 10.0.4.0/31
Known via "ospf 80", distance 110, metric 13, type intra area
Redistributing via eigrp 80
Last update from 10.0.0.37 on GigabitEthernet1/1, 1w3d ago
Routing Descriptor Blocks:
* 10.0.0.37, from 10.0.1.20, 1w3d ago, via GigabitEthernet1/1
Route metric is 13, traffic share count is 1

RouterE#show ip route 10.0.3.1 Routing entry for 10.0.3.0/31
Known via "eigrp 80", distance 90, metric 26112, type internal
Redistributing via eigrp 80
Last update from 10.0.0.40 on Vlan50, 1w5d ago
Routing Descriptor Blocks:
* 10.0.0.40, from 10.0.0.40, 1w5d ago, via Vlan50
Route metric is 26112, traffic share count is 1
Total delay is 20 microseconds, minimum bandwidth is 100000 Kbit
Reliability 255/255, minimum MTU 1500 bytes
Loading 1/255, Hops 1

The route to 10.0.4.1 is learned via OSPF, while the route to 10.0.3.1 is learned via EIGRP. In this environment, OSPF uses fairly small metrics (LAN links have a cost of 1, WAN links have a cost of 10). We use the default EIGRP metrics, which are much larger numbers. Under nearly all other circumstances, IGP metrics are not compared between different routing protocols. This is one of those 'special cases'. Normally different routing protocols are compared based on Administrative Distance, and the routing protocol with the lower Admin Distance is preferred, with no regard to metric values.

Solution
To solve this issue, I removed the RouterD<->RouterE link from EIGRP routing, but more importantly, I also removed all AS65000 network links from the EIGRP topology table. This prevents RouterE from learning the 10.0.3.0/31 route via EIGRP. With both 10.0.3.0/31 and 10.0.4.0/31 learned via OSPF, the metric comparison makes sense, so the path is properly chosen. Here is a diagram of the new environment (notice the subtle movement of the EIGRP 80 domain to exclude the RouterD<->RouterE link.. it's the only change):

And here is the show command output confirming the solution worked:
RouterE#show ip route 10.34.0.0 255.255.0.0 Routing entry for 10.34.0.0/16
  Known via "bgp 65000", distance 200, metric 4294967294
  Tag 65001, type internal
  Last update from 10.0.3.1 00:02:17 ago
  Routing Descriptor Blocks:
  * 10.0.3.1, from 10.0.1.11, 00:02:17 ago       Route metric is 4294967294, traffic share count is 1
      AS Hops 1
      Route tag 65001

RouterE#show ip bgp 10.34.0.0/16
BGP routing table entry for 10.34.0.0/16, version 118687
Paths: (2 available, best #2, table Default-IP-Routing-Table)
  Advertised to update-groups:
     2        
  65001, (aggregated by 65001 10.1.34.1), (received & used)
    10.0.4.1 (metric 13) from 10.0.1.20 (10.0.1.20)
      Origin IGP, metric 4294967295, localpref 100, valid, internal, atomic-aggregate
  65001, (aggregated by 65001 10.1.34.2), (received & used)
    10.0.3.1 (metric 2) from 10.0.1.11 (10.0.1.11)
      Origin IGP, metric 4294967295, localpref 100, valid, internal, atomic-aggregate, best

RouterE#show ip route 10.0.3.1 Routing entry for 10.0.3.0/31
  Known via "ospf 80", distance 110, metric 2, type intra area
  Redistributing via eigrp 80
  Last update from 10.0.0.40 on Vlan50, 00:02:34 ago
  Routing Descriptor Blocks:
  * 10.0.0.40, from 10.0.1.11, 00:02:34 ago, via Vlan50
      Route metric is 2, traffic share count is 1

RouterE#show ip route 10.0.4.1
Routing entry for 10.0.4.0/31
  Known via "ospf 80", distance 110, metric 13, type intra area
  Redistributing via eigrp 80
  Last update from 10.0.0.37 on GigabitEthernet1/1, 2w0d ago
  Routing Descriptor Blocks:
  * 10.0.0.37, from 10.0.1.20, 2w0d ago, via GigabitEthernet1/1
      Route metric is 13, traffic share count is 1



Summary

In the grand scheme of things, a single additional gigabit Ethernet hop is inconsequential. So why does this matter enough to open a change request and wake up at 6am on a Sunday to fix? For me, it’s a sense of completeness. There was something ‘not right’ about the way things were working, and fixing it pushes the needle a little closer to ‘done’. I equate this to a love of symmetry. For some people, it’s important for things to be balanced. When I take two eggs out of the carton, I take one from each row. It isn’t more correct, but it makes me feel better. Same goes for the network. If it is working 100% optimally, then any small issue is readily apparent. Terry Slattery said something similar in his blog entry on network hygiene a few weeks ago:
If you don't keep things clean, interactions between otherwise minor problems can create a larger problem
He’s absolutely right.. eliminating the small issues makes the big ones far less complicated.