Triple CCIE so what? – Passed CCIE SPv3
Well, I think the time has come for my blog post to hit the cloud re the journey to the world of triple CCIE!. I have consolidated my third CCIE journey in this blog post. The purpose of this post is to inspire other candidates who are still on the road to achieving their CCIE. Most people think once they get their CCIE, they are the guru. I don’t think so, you don’t have to have a CCIE to be a master of a particular technology. I have seen many of my friends who do not have a CCIE but they are capable of writing an RFC and doing embedded design work. However, what CCIE really teaches you is discipline and thinking outside the box and tackling any challenges that you might see in the field and quickly come up with the right approach. Being a triple CCIE, I really think there is so much to learn and always there is a gap to learn new things.
Way back in 2004, when MPLS was quite new, I was involved in deploying a few small MPLS networks. The largest one I deployed was a 7-nodes MPLS network that covered the whole country. Wheee?. 7-nodes covering the whole country, a bit of a surprise here? That’s true, yeah, it was a small island country in the south pacific, where Telco normally do not have many customers. I am talking about way back in 2005-2007. I love working on new things and in the past I had an opportunity to work on N7K/ASR(XR). XR is totally a new beast in the CCIE SP lab but not new for me. Don’t be afraid of XR as it has slightly different command syntax than IOS. Afterwards I had an opportunity to work on different technologies (voice, datacenter and big data). Recently I have been involved in a project that would not need more of my extra brain cells and I had plenty of (outside office hours) time to do and try something “new”. I was not thinking for another CCIE at all but instead I started thinking about porting a customised version of Openflow on a netFPGA board.
I bought 2 boards (bloody expensive though) and started building a list of *nix lib and packages that I’d want to make my first netFPGA board working. The idea was to build a small embedded OS that can port OpenFlow. I started working with buildroot to build the embeded OS and spent nights and nights in coding and preparing the Makefiles to be able to do what I wanted to do to port OpenFlow. My OS poting was completed and the netFPGA board was running on my own customised embedded OS. I had a few issues with the Openflow libs, the interfaces wouldn’t come up. I was then running the latest Linux kernel v3.8. I went back to the buildroot again and re-compiled the whole libs and right kernel 2.6.32 and ported it on the netFPGA board. This time bingo..I was so happy as I could see my whole thing in ACTION, interfaces were up and working as I was expecting. 🙂 Whooooooooooooo!
To give you a few ideas of what a netFPGA application can be, have a sneak peak at the image below:
I was enjoying the journey of building my own switch based on netFPGA board. BTW, I hated rasberry pi boards as there was too much work and vendor wouldn’t provide a lot of libs/firmware that is required to do something new!. Anyway… oh no, I thought I was writing a blog post for my CCIE journey, sorry lets go back to it.!
I changed my job in 2012 and was playing tennis. It was so much fun as I switched to tennis from cricket. As I was new to tennis, I forgot everything about my backyard netFPGA project. The fact was it was too time consuming. To get something working, I had to play with script/codes for ages. I don’t write codes on a regular basis so even trapping a mistake in a few codes logic, it was taking a lot of time. It is not fun when something is not working as what I had initially expected. There is not much support that one can get from the Internet. One day I was doing some reading on BGP and it eventually got interesting when MPLS-TE came along with the above diagram that was using the netFPGA for Optical networks. I thought this is it, I wanted to learn and master MPLS BGP. Then I thought should I go for SP CCIE. I just wanted to challenge myself so decided to go for SP CCIE track. I did some research on the Internet on SP lab and found out most people on blogs were talking about doing R&S before the CCIE SP. I just ignored them and decided to take the hard bullet first.
1.2 INITIAL PREPARATION:
- I downloaded CCIE SP lab techtorial from CiscoLive site and started reading them. I went through all the slides and then I bought some MPLS TE books written by Azhar Sayeed, Monique Morrow.
- A friend who failed SP lab 3 times and unfortunately he finally gave up! (what a shame!). He was happy to hand out free IPexpert/INE CCIE SP V2 video tutorials. I went through them as well. They were awesome in terms of understanding the technology and the goachas!. The INE workbooks were outdated but had very important tips and tricks. It didn’t cover the XR part though.
- Awesome custom labs from CCIE SP program manager (a must have) – Vincent Zhou
- XR fundamentals and deep dive training material by Cisco (unfortunately this is for internal use only)
- Obviously the famous DocCD.
1.3 PREPARATION TIME:
It took me almost 6 months to pass the lab- This is from start to finish. The Detailed breakdown is below:
- CiscoLive SP Techtorials – 1 week
- MPLS TE book – 1 month (was reading ’em on and off)
- INE/IPExpet Video – 2 months
- INE/IPExpert workbook practice – 1 month (only done the last labs but went through all the labs to see what they were trying to simulate)
- Vicent Zhou’s custom labs – 3 weeks (this is a must have, I repeat, this is a must have material)
- Tech notes and docCD – almost most of the time when I needed any clarification on something that was new to me.
- Improving lab start-finish speed – 2 weeks full time labbing.
I am not a plug for INE/IPExpert but they do have some awesome info that I couldn’t get any where else.
1.4 CCIE SP HOME LAB:
I spent almost a month in buying a proper server for my SP home lab. It happened someone in Cisco (internal) wrote a script for INE workbook practice in a simulator. It was such a time saver for me, as I didn’t have to write IOU/XR topology files to practice the INE labs. Thanks Brad R (A great guy in Cisco Advance services:). It took me a while to make the simulator (what is it, how can i download it?) work in a way that I wanted it. But after initial hurdles it worked like a charm. I was able to do most of the lab tasks with this magic simulator. For any few features that the Sim couldn’t do, I used our internal company’s lab based on real hardware i.e. beefy XR routers, ME 34xx switches and other IOS routers. Please do not ask me about the simulator as this is internal and Cisco partners only at this stage.
Here is Vincent’s sample lab topology:
1.5 WORK/FAMILY LIFE and TIME MANAGEMENT:
I was lucky enough to work in a good team environment and as a result I didn’t need to work after hours to resolve customers’ issue(es). In our team, all members are CCIE (double/triple) and are pretty switched on. A few guys in the team are like a walking encyclopaedia, you just ask and save time in googling/labbing. That means you spend less personal time to solve customers’ issues and more time for study (umm). I know a few guys who have got such an understanding manager that allowed them to study while at work 🙂 . Is there anyone else who has this kind of understanding manager? I wish I was one of them but unfortunately my work commitments didn’t allow me to have such a luxury. However, I developed the following strategy that worked well for me and I think that might also work for some of you dudes lurking out there :):
- Organise and tidy up your work – for example, if you have 10 tasks to deliver this week. You can sit one day at home and work on all these 10 tasks together. at least you may complete them 60-80%. You can almost do a week’s worth work in less than 16-18 hours (am I crazy in saying this?). You can keep these work related tasks handy. Keep them ready in your pocket just like candies. When your Manager asks to deliver one of those pending tasks, you just smile and an hour later hand it out to your manager just like candy. This is just an example re how you can save your brain cells and balance out your work. This will also give you a true sense of 9 to 5 job and you’ll have some room in your brain cells for study. The idea is, the less office work related tension you have in your brain, the more productive you’ll be in observing the study juice. If you’re job is like replying to hundreds of emails on a daily basis then you might need to come up with email strategy. This is just another skill to learn but it is very useful to save time. Many of you out there would know what I mean. CCIE and crazy work schedule do not go together.
- Email management: Most of us do this but may be useful for those who are not doing it yet. I know this has become an old trick now. Use office Email on your mobile and answer them while you are commuting. This saves a lot of time. Many work related tasks you can do while you are not at work. for example, if you are waiting for an input from customer you can email them 11pm and they will usually reply to you before you get to the office in the next morning. It’s good for everyone. saves you a lot of time waiting on a customer.
- Weekend fun/activities ? – Give it up for a while if you can, no more drinking with friends, I gave up playing tennis and going out. No more bush walking either.
- Social life – Forget about this, not going to happen. CCIE and social life do not go together either. This is completely legit to tell your friend that you’ll be busy and do not appreciate their offer to go to a pub and enjoy one of those late night outings :(. A good friend will understand that. I guess, it would be hard to negotiate with your wife!.
- Exercise: Everyone knows, sitting in front of the computer is not good. You must take breaks now and then. Between the lab break, go out and do some exercise. If you’ve a swimming pool, that is the best. A 30 minutes swim or 25 minutes run is more than enough. (BTW, I can swim now) Typical thing I’d do during the break was clean up the house. Wash the dishes in the kitchen, do dusting, vacuum , anything that is moving your body. I think these easy activities chew up a lot of calories. Obviously, eat healthy food during study as the last thing you want to do is to get <sick>.
1.5 LAB PRACTICE STRATEGY:
I have used my own home lab using special simulator in VMware environment :). It was very handy as I could just do a part of the lab and (if I want to ) have a power nap and then resume the lab without worrying about time running out. Unlike online rack rental we have to finish the lab within the time constraint. Unfortunately CCIE SP lab can not be “FULLY” simulated (XR Part) with the material that is openly available on the Internet. On stolen IOU and freely available dynamips/GNS3 you can do some practice. An alternative could be you can build up SP lab based on stolen IOU and dynamips/GNS3 and when lab date is near then hire online rack rentals.
Last 2 months of my journey I used to practice everyday, if not alternate days with different workbook. You must go through Vincent’s labs. Without them it is not possible to pass the lab, unless you design and deploy MPLS a day-in and day-out basis. A guy who is reading this post and been to the lab before must be smirking re what I am on about. He knows what I mean!. Anyhow, you’ll know what I mean once you go to the lab first time and then see Vincent’s labs later on. Without breaking the NDA, I’ll give you an idea what gear an actual CCIE SP lab has:
- 4+ x XR routers
- 20+ IOS routers
- 2x ME3400
My typical study and labbing time was:
- Monday to Wednesday – study, watch videos, browse through tech notes.
- Thursday – resting.
- Friday Saturday and Sunday – extensively labbing.
- I studied part time with my FULL time job. For some people it could be challenging to get time for study. But hey, if you really want to have a drink, u will find a pub no matter how far u got to drive. The same thing applies to the study and “MAKING” time for it. If you are serious make time for it :)= my plug.
I’d spend most of the time in study and less time in practice. Later stage (near the lab date) it was reversed -Most of these hours I’d spend in practising, checking notes, summarising the task list and updating my to-do list. I documented everything that I encountered during the practice labs. I focussed on straining my brain cells on why this way the solution is working and not the other way.
I initially thought to use alias in IOS/XR to save some time but once I did the lab 10 times, all commands that were required to verify the tasks were straight in my head and I could type them quicker than remembering the alias and typing them out: for example: top 10 commands:
show bgp ipv4 blah..,
show ip ospf int bri,
show ip igmp ,
show mpls int,
show mpls ldp discovery
show mpls ldp nei
show mpls traffic-eng tunnel
show ip pim rp
show ip pim nei
show ip pim int
show clns protocol
show clns nei
show isis nei|int (on xr)
1.6 FIRST ATTEMPT EXPERIENCE (JAN’13 SECOND WEEK):
Being based in Sydney, I was able to get SP lab date easily. In fact there are not many people taking SP lab here. Probably it’s too boring for them while seeing the other nicer cakes a.k.a Bigdata, Wireless and DC lab just out there. I booked the lab for the second week in January, 2013. I was feeling 90% ready as I have been through the MPLS MBGP, MCAST, BGP and L2/L3 VPN technologies. I repeated the videos and re-viewed/practised the vendor and vincent’s workbooks. Nothing was left that I could think of but I was excited to go ,sit and experience another lab game for the third pie!
2 weeks before my lab date, I took a week’s break from work and focussed on speed and summarising the tasks and the right order to implement them. I was doing labbing 16 hours a day and a few hours of revision. Day before the lab, I went to bed at 11pm. I had a few glasses of red wine. I woke up in the morning 15 minutes before my alarm bell rang. Got ready and in 20 minutes time I was on the way to the lab. It only takes me 35 minutes to get to CCIE lab in Sydney by train. I know a few folks have to travel from overseas and interstate but for me it was just like another commute to work.
In the CCIE exam center, we were 4 guys (unfortunately, no ladies), 1xSP candidate, 1xR&S, 1xVoiec and 1xWireless candidate. Nobody was doing storage/DC. After a quick briefing, the proctor showed us the POD#. I was very impressed to see dual monitors (I wish it would stay this way) on my POD. The SP lab exam has no papers like we did for Voice/Security exams. I opened the question paper and diagram on the right hand side monitor and then opened up the sessions to all devices on the left hand side monitor. None of the other CCIE pods had dual monitors so I was already feeling like a very SPECIAL frog (see frog in the left, jumping frog from the Rack). I think somebody had well thought out about putting dual monitors for SP lab candidates. It makes complete sense to have dual monitors for SP lab. The matter of the fact is that there are so many devices in the SP lab that need a separate monitor. Jumping between the routers is a waste of time. I read the whole lab in 45 minutes, noticed there were a lot of questions that I didn’t know the answers. I just made a quick summary and diagram in mind. I didn’t draw it because lab diagram is already there on the screen why bother duplicating the same thing?. I made a list of important parameter for IGP, iBGP, eBGP peers, interfaces that need to be enabled for LDP/PIM and then started doing tasks in the order they were presented in the question paper. I skipped a few tasks because I didn’t know the answer as I said before. Then just before lunch, my IGP started playing up. The RouteReflector router would show up in the routing table but somehow the whole BGP topology was broken. I thought, right on, here we go again the cat and mouse game is on for the third time. I said to myself I have seen this before so I was calm and was going through the show and debug commands to trap the fault. I managed to fix it and it turned out that this issue was with the hardware. Proctor was so nice and understanding, I explained to him what the problem was and he said, continue doing other tasks while he fixes this. Finally the proctor fixed the problem and gave me 30 minutes extra. How kind? well I really spent about 40 minutes to trap this fault so why shouldn’t I get it in the first place?. As you know every minute in the lab matters.
Just before lunch I had IGP, e|iBGP, MPLS, PIM up and running. I had some issues with the ISIS, and one of the Mcast/MSDP questions I misread and implemented the wrong solution. I went back to the docCD and found the answer and commands that were required to finish up the task. Anyway..during lunch break everyone had quick lunch with proctor. Surprisingly, after 20 minutes everyone was back to their POD# again. I think the proctor wanted to finish early and go home and also the student wanted to get back to the lab ASAP. That quick lunch strategy works out well. After lunch see the the voice guy was sweating and bleeding, his phone was not dialing out anything as I could hear fast busy beeps.. That reminded me of my first few lab attempts when I was doing my Voice lab. This guy must have been frustrated!. Anyway, after lunch, I finished CSC, MVPN and l2vpn section. My CSC was broken and I couldn’t fix it. Eventually I ran out of time and noticed that proctor calling ‘times up’. It was 5pm when everyone left the lab. I got the train back home and was hoping I might pass. 2 hours later the bad news finally come, …….FAILED!. I went through the lab score report and found that I did a bad job in VPN and IGP sections was not 100%. I nailed L2vpn and other sections. I just took it as a part of the learning curve and to come up with better strategy and be better prepared for the next time. I thought to come back after a month or so and re-attack the beast with a sharper sword. The lesson to learn here is (to the reader as well) after failing the lab, never put your journey off track. Get back on the horse ASAP and stick to it. Never ever think about giving it up (just like image in the left, frog can do so can we)! Immediately after I started looking for the lab date. I found another date and booked the exam for Feb last week. This time I thought I desperately needed a break from everything. I have been studying and working full time for more than 5 months. It happened we had a new team member who just joined us and was pretty switched on and I asked if he could fill in my role. He said no issue. After this, I got approval from my manager and I was on the mission of doing nothing but wanting to give my brain a thrill, challenging myself and a good work out and a true break from work and CCIE lab crap!. The bottom line, never go the lab until you feel 200% ready. I thought I was ready but I wasn’t. I ran out of time, had no time for verification and tackling the lab jitters. Once you are 200% confident and ready, your CCIE# is just around the corner (Quoting Narbik here as these are his words).
1.7 BREAK FROM EVERYTHING:
I decided to do some adventurous sports so that I could give my brain cell a real thrill and challenge. I flew from Sydney to Queens Town New Zealand and did the following sports activity. While on-board in the plane this unexpected briefing of Air New Zealand made me feel that I was really going on a holiday to the middle of the earth! whoooooo. It was simply the most awesome briefing that I have ever seen on any aircraft. I wonder if that’s why Kiwis are different than rest of the world and do things differently.
Here is the link for the full airNZ video:
Those who have not been to Queenstown yet, you don’t know what you are missing. There are so many things to do. You’d never be a bro-dam! You can see below activities I was not sitting idle during my stay there.
- Day1 – Flying foxes. Flying from tree to tree. This was just like putting my feet into the tub.
- Day2 – 134 meters Nevis bungy. This really blew up my mind. For the first time I have done 134 meters.
Here is the video watch first half:
- Day3 – 46 meters Kunwara bridge bungy (it was just a kiddie one but I wanted to do just because this is the place where bungy sport was born)
- Day4 – Farm trip, famous sheep and cattle farm.
- Day5 – Four wheel drive (4WD) tour.
- Day6 – Sky dive from 15,000 feet (first time dived from such an altitude did before 12,000 feet)
- Day7 – Nevis Swing – Biggest swing in Asia. Was fun but not as good as 134 meter bungy
Here is the video watch second half:
- Day8 – Milford Sound – loved the trip in the outback to natural beauty.
- Day9 – rest and fly back home to Sydney.
After a good break, I felt good and very confident. I challenged myself and got the success that I wanted. I thought if I could do 134 meters bungy, I could also do the lab as well. Should be a piece of cake for me now. All sorts of constructive thoughts were going through my mind. Doing all these sports gave me a new direction to stick to the lab game again.
1.8 BACK ON LAB PRACTICE AGAIN:
I know lab is coming very soon after a few weeks. Went back to work and then stuck to my original timing. After 9-5 job, practice the lab and re-define my lab notes. After a week of practice, again everything all IGP/BGP/MPLS-TE and CSC commands were in my head. I had no issue in recalling them again. I did two weeks practice. This time when I was doing practice, I could do the whole lab in less than 5 hours. I thought if I could do the same in the real exam I would have enough time to verify the task. I really felt I could do it . The last week before the exam, I didn’t do anything but just reviewed all the material, researching on why we are doing it this way and what happens when we do in other ways, where the lab breaks and how to verify a task. I came up with a list of commands to verify the tasks and their order. I’ll give you an idea what kind of verification recipe I developed:- Lets say if we are verifying MPLS and MPLS-TE. I would type these verification commands in a notepad and then paste on each device to save time. It really works. There is a slight difference between IOS and XR command syntax.
show mpls int
show mpls ldp dis
sh run | s mpls
show mpls traffic-eng tunnels brief
show mpls traffic-eng database fast-reroute
show run | s ip explit
show run | tunnel-te|tunnel
show ip ospf int bri | in 192.10
show ip ospf nei
show ipv ospf int bri
show ipv ospf nei
show run | s ospf
sh run route ospf
show clns protocol
show clns neigh
show ip route isis
show run | s isis
1.9 SECOND LAB ATTEMPT EXPERIENCE (FEB LAST WEEK):
A day before the exam, I did something silly and had 4 stitches on my forehead. So never do any dangerous sports just before the lab exam. Keep your energy and sleeping patterns intact for the lab day. Anyway as usual, I went to bed late 1 AM and had to take Panadol pain killer to get some sleep. I woke up at 7AM, arrived at the exam center at 8 AM again. This time the proctor knew my face but couldn’t remember the name!. The matter of the fact is that a lot of candidates attempt exam and it’s almost impossible for a proctor to identify them by their names. There were 4 other candidates – 2xR&S, 1xVoice and 1xSP (myself). After initial proctor briefing, we went into the exam room. Proctor showed my pod. I was given the same POD# as before but this time there was only one monitor. During the lunch time I asked the proctor what happened to the Monitor, he said a candidate complained about dual monitors being a distraction and Cisco removed it. I was like WTF?. Just turn it off if u don’t want it. But important thing here is, Cisco does listen to the genuine complaint and act very quickly. This is just an example. I see a lot of complaints from candidate(s) winging about proctor changing config, what a rot!
I logged into my POD and started reading the questions and topology. This time I was given a different exam. I finished reading and observing the topology in my head within 25 minutes. Sigh.. last time it took me 45 minutes!. Felt some improvement in my strategy. I made notes about IP address, tunnel requirement, iGP/BGP topology and CSC mind-map. I Just gave another 10 minutes and finalised the whole labs (recipe) requirement and noted them on the blank paper. I adopted the same approach that any celebrity Chef would do in a cooking show. Chef prepares all the recipe/ingredients and keeps them handy. When the show starts Chef just throws the things in the pan without even looking at the recipe book (the DocCD) :).
Well, this time everything went well till lunch time. I was done with 80% of the tasks before lunch time. I left CSC and mVPN sections for after lunch. At this stage, I was just configuring tasks and doing simple verification. IGP, BGP , multicast, MSDP. Fortunately they all were up and running nicely. I already felt that I was almost there but was very careful with my strategy. Finally all candidates went out lunching with the Proctor. After 15 minutes break, we were all back on the POD. A guy who was doing R&S (from NEC Melbourne) started asking proctor questions and the guy on the side doing Voice lab was having the same issue with phone as I saw in the previous attempt with a voice candidate. As far as my lab went, I was too busy copying & pasting CSC configs. It was too much typing in CSC section. You’ll have to touch about 14 routers to complete the CSC section. My fingers were already getting sore…(look at the left side image, the frog’s finger:) The way I was bashing the keyboard, I thought I must been annoying the other candidates in the room but then thought hey, If I don’t’ do this keyboard bashing, I am in deep trouble so I just ignored the other candidate’s hostility. I finished CSC and mVPN sections with 2 hours spare time in hand. Then I took 10 minutes break, went outside the lab and started chatting with receptionist. I just wanted to get out of the lab environment!. I drank a bottle of water , went to the toilet and got back in with a fresh mind set onto the frogPOD . I thought after this time, all I had pending was verification. I thought, wow, it has been smooth sailing. 10 minutes later when I was pasting my verification commands. During verification I noticed I had about 15 mistakes. This is why verification is very important in the lab as well as when doing a real world implementation. The good thing this time was I did not have to touch the docCD (except for a few questions that I didn’t know the answer) as all the required commands were in my head. I left those 2 questions and verified every other task three times. I also checked how many points I had collected so far and it was 92 points. I thought 2 questions will not affect my result. All of us finished the lab at 5PM, afterwards I had to go to do a cutover for a customer. I was really exhausted by this time and then I had to work on a cutover till 11PM. What a sucker!..
50 minutes later I got an email on my mobile from email@example.com, with the subject “Your exam result available blah..”. I was in the middle of doing a cutover and couldn’t resist waiting until I get home. I jumped onto the web and closed my eyes and then took a deep breath counted 5…4…3…2….1 (just like when I did my bungee jump), and bingo, opened my eyes and saw”PASSED”. I refreshed the screen a few times and clicked through the exam critique to make sure it was showing a correct result. And … it still was showing PASS. I was so happy that my unexpected triple CCIE mission was over!. I worked till 11pm and then went to a local pub ate some pub food nice schooner and next morning I was back at work.
- Draw a good mind-map study plan and stick to it. Summarise tasks.
- After failing the exam, make sure to get back on to it ASAP, do some research on why solution is the solution and why the other way of doing the same task will not be acceptable.
- Come up with a better strategy. Review the previously drawn mind-map and update/alter it to meet the current situation.
- Reduce other activities and stay focused on study.
- Have regular breaks, eat good food, stay healthy and stick to mandatory 30 minutes of regular exercise.
- In the lab, perform tasks quickly and accurately. We should have at least 2 hours of spare time in hand for verification, troubleshooting and any lab jitters that might need a fix.
- Cut down on alcohol consumption. I don’t smoke, never done drugs so won’t have a clue what it does to study. I would suggest stay out of it as it distracts the concentration power of the brain cells. Alcohol does anyway. Now you realise why drinking and driving do not go together. The cop will catch ya.
- Adopt the right learning method:
repetitive method works well for me. I suggest others to do repetitive method of learning as well. i.e. first skim through the topic, then re-read a few times. Highlight and re-read it again, do some research if you see anything new. Repeat the same topic every week then fortnight and then once month. This is the way my brain adapts/observes new things.
1.10 WHAT’S NEXT:
A few friends have asked me this question -What I am gonna do next. Now that I have achieved triple CCIE status. My answer is, nothing else but play Tennis and I am eyeing on 233 meters bungy from Macau Tower in China! Whoohoo, that’s gonna be real fun. Has anyone done MacAu tower bungy?, I would be interested in knowing their experience.