I wish I had learned how to read a packet capture much earlier in my career. I was taught how TCP establishes the three way handshake early in my career but but I never really understood data flows until I opened up Wireshark and started digging through line by line.

In my previous life working for a service provider being able to look at detailed packet captures was invaluable.

Here's the tip I give every new techie....

Somewhere, there will be a log file, or an option to enable the log file.

Linux, Unix, Windows, Network gear, etc. There is a log that will tell you why things are going wrong. You may not understand it, but it will be generating an error or some info.

Copy the error log and paste it into google.

9 times out of 10, someone else has seen it. Learn to use the search tools like that, paste the errors in somewhere, and dig a little bit.

If you don't learn to search these issues on day one, and learn to use the entire internet, these days... It's hard to help.

Troubleshooting is A number 1 in the tech biz. If you can't troubleshoot using every resource around, you may as well look for another job.

This is clearly a 'trick', rather than something really substantive but on a Linux system, if you want to see if a link has been established (L2, not L3), ifconfig should show the interface to be 'RUNNING'. There are countless occasions on which I'm not sure a cable has been attached in the right port and this is how I go about it. Far too many people use ping or ssh, which requires IP networking to also work on both ends and no firewall or routing complexities.

Slightly higher level view is to debug one layer at a time when things don't work. If you're not sure you haven't got a system networked right, don't go ping google.com to test. Start with link layer, move up to IP, then bring in routing and DNS.

This is my first post here, so if this is far too obvious, my apologies.

I wish whenever to start understanding or study new technologies, study the Header format first. When I was troubleshooting MTU issues by ping with df bit set early in my career, I could not understand how it really works. But later when I went through the headers of IPv4 it makes things very clear and be on memory eternally. Else things will be forgotten and misunderstood. Also Visualize things how it works.

Also how to effectively use network forums be it cisco netpro, juniper forum, gosammer mailing lists, networkforum etc. Sometimes even OEM TAC engineer takes time to reply, posting or searching the issue in the forums provide quicker results.

Following good technical blogs also is a very good way. Latest technologies not only get updated but people around the world share their issues and that issue might be faced by others too.

I wish I had known that you can't see anything hardware switched in the debug ip packet output. As a workaround for "capturing" packets I sometimes use ACLs like this:

access-list 100 permit <TRAFFIC-I-WANT-TO-SEE> log
access-list 100 permit ip any any
!
ip access-list log-update threshold 1

bind this ACL to some interface and you'll see each matching packet in the log. (beware to not overwrite existing ACLs)

That the problem isn't always the network. Early in network careers you get suckered into believing every problem is the network when sometimes it's the application, the user, or the end host.

A scripting language. A lot of things were less tedious after I learned TCL. Linux. We had different flavors of Unix when I was in college but I let my knowledge of it slide. I still use a Windows PC but find myself having to use Linux based tools.

In order of how I wish I'd learned, not importance (everything on this list is important, in my opinion).

The first thing, however, really is The First Thing and I cannot overemphasise it:

Learn to touch type

As a network engineer (or any kind of IT professional), you're going to spend a disproportionate amount of time typing things into various command lines and editors. Therefore:

Don't let your fingers be the bottleneck to getting stuff done.

There are lots of tutorials online to get you started; once you know the basics, the rest is just a matter of practice. I use Typeracer myself.

Using Linux

• Get comfortable in your shell.
  • Customise your environment.
  • Learn about shell pipelines and text processing.
• Editors
  • Learn one of vi(m) or emacs really well.
  • Learn enough of the other editor to get by.
• Learn to use a terminal multiplexer.
  • screen
  • tmux
• Learn at least one scripting language and how to do useful things with it.
  • Python
  • Perl
  • Ruby

You're only as good as your tools and environment allow you to be.

Protocol fundamentals

• Bridging/switching
• ARP
• IP routing (and how to check IP routing tables on various hosts as well as on network devices)
• DNS

Note the above needs modification when dealing with serial/optical media, as well as when dealing with IPv6, but the principles remain the same.

Where available, read the RFCs describing these protocols.

Also, in particular, note DNS. I've dealt with far too many network engineers who view DNS as an application protocol not worthy of their attention. Here's a clue: DNS is how your users find network services; as far as they're concerned, it's part of the network.

How to troubleshoot a.k.a the scientific method

• Come up with a theory.
• Test the theory.
• Modify your approach based on the results.

Cargo cult troubleshooting is worse than useless. If you're seeing 100% utilisation on a port, swapping the cable isn't going to make the problem go away.

Packet captures

• Massively useful, but should be held off until basic troubleshooting skills have been developed, lest people become overly reliant on it.
• Learn BPF. Learn it well.
• Learn filters for interesting traffic. e.g.
  • all BGP
  • all ARP
  • all traffic from a particular MAC/IP address
  • IPv6 filters; these are particularly fun, since oftentimes you'll need to calculate byte offsets yourself.
• Learn how to string said filters together.
• Learn which filters are useful in what situations.
• Use packet captures to reinforce your lower level protocol knowledge. Look at packet captures for common protocols (ARP, DNS, etc).

That's a reasonable start, at least. I have a half written blog post on "fundamental stuff to learn" which I may eventually finish; if so I'll post a link here.

Two things to add:

1. For as much as we probably all rolled our eyes at the 7-layer OSI model when we were green and first learning it, it remains one of the most fundamental concepts and is essential for designing, maintaining and troubleshooting networks on a daily basis.

2. Handy IOS CLI shortcuts:

show [command] | include [string]

(config) do show [show command]

(config) interface range [interfaces]

(config) default interface [interface]

The ability to manipulate the CLI quickly during an outage is invaluable.

Esc, B Moves the cursor back one word. Esc, F Moves the cursor forward one word. Ctrl-A Moves the cursor to the beginning of the line. Ctrl-E Moves the cursor to the end of the command line.

Ctrl-D Deletes one characters right of the cursor. Ctrl-K Deletes all characters right of the cursor. Ctrl-U or Ctrl-X Deletes all characters left of the cursor. Ctrl-W Deletes one word to the left of the cursor. Ctrl-Y Recalls the most recent entry in the buffer.

Ctrl-T Transposes the character to the left of the cursor. Ctrl-R Redisplays the current command line. Esc, L Changes the word at the cursor to lowercase. Esc, U Capitalizes letters from the cursor to the end of the word

Differentiating between layers 1,2, and 3. So many times I see people looking into what I like to call string theory solutions, and it ends up being physical(won't lie, I've done it too).

I also wish that somebody had taken to time to teach me early on the best way too make changes(start at the furthest logical location and work your way in). It would have saved me some headaches as a junior tech.

Switches don't give a damn about IP. Not so much for myself but I've seen a few people start freaking out because they can't ping a switch(We control over 600 switch stacks...shit happens and sometimes they lose management. Unfortunately we used to have a mostly Nortel network that liked to randomly dump management). They never think to look and see if they can reach devices behind that switch and spend lots of time troubleshooting something very simple.

DNS. My shop gets thrown alot of issues that claim to be network connectivity but it's really DNS. If you don't know how to troubleshoot that then you're going to be wasting alot of time.

VRF-lite. It's a fair bit of ground work to setup but it's such a nice solution to a lot of problems.

The one thing that I have learned is that documentation sucks but no matter how much it sucks it is a necessary evil. You may hate writing out how things are connected for this or that; or you may hate documenting why this is configured this way; but in the end, it will help you figure out why things are wonky. As Will Dennis said earlier, keep that notebook handy but engineers need to remember that that notebook does only one person any good if they are the only ones that can read it.

The other thing is learning shortcuts. Shortcuts are great for fixing something but when it comes time to fully document something and all that is known are shortcuts, how are those helpful to new people? Learn the full command and then learn to shortcut. It will help the newbie to know the difference between subtle commands and why they work in one instance and not in others.

Finally, I wish someone had told me how to keep my mouth shut and listen. I am still learning this one. That one lesson would have saved me a lot of time and heartache if I would have just shut up and colored in the lines. You have to know when to fight the battles and when to let some go so that you can mount a better offensive for something else that is more important. The ability to recognize the difference leads to better working relationships with people and can lead to a better working environment.

I started my career as a programmer and linux/unix sys admin. Then I worked a lot with ADCs (application delivery controllers) like F5 (iRules), Radware, Netscaler etc. So a lot of the skills other people mention here I've had for a long time.

as a cisco network engineer the most important SIDE skill has been packet capturing and linux. here is an example of both

one time I used Linux server as a sniffer with a port mirror on a switch and I saved 320GB worth of pcap data with tshark (impossible with wireshark it crashes), then wrote a script that made tshark chew thru 320GB of data to find instances where LDAP protocol response was greater than 200ms. Some app was showing timeouts due to sporadic LDAP response timeouts. the script helped divide the pcap files and create various instances of tshark to take advantage of multi-core cpus (using one stance would've taken forever).

The tshark instances looked for LDAP responses that were greater than 200ms and the output was configured to just show the information I needed in ASCII stdout.

the end result showed slow LDAP responses every 5 hours... this helped the server people identify that some cache was emptying every 5 hours. causing slow responses for about 10 mins... until the cache filled again or something... they disabled this and problem fixed thanks to my packet capture and scripting skills lol

the reason why I did all of this was because the LDAP server people were blaming the network (that I administered) saying it had delays thats why the responses were slow so I used my capturing skills to tell them to SHOVE IT!! lol

I'll post the tshark command but I'll leave the scripts up to the imagination of the readers lol

tshark -n -r $CAPTURES -R "ldap.time > 200" -T fields -e frame.number -e frame.time -e ip.src -e tcp.srcport -e ip.dst -e tcp.dstport -e ldap.searchrequest -e ldap.searchresdone -e ldap.searchresentry

1. Step back from you issue, even just for 5-10 minutes. It's The last thing you are thinking about doing when your entrenched in problems. But trust me, you need take a break and get your composure. I can guarantee that you will come back to the issue with a new perspective on fixing the issues.

2. The world is not going to end, so don't stress out too much. Also, don't not worry about it too much. I mean, the thing is broken after all.

3. Build yourself a good tool kit. Software and "real" tools. It's like a Network engineer's safety blanket. It's amazing how much grief I've saved with a simple $1 loopback plug

4. Read News groups, blogs, and tweets to keep yourself aware of whats going on in the industry. Dare I say "Cloud" and "BYOD"

IOS Tips:

type '/' and the when going through a "show run" and it will find that keyword. performs the same function as "sh run | beg

Use the 'archive' feature to locally log commands entered into the device also you can use it to back up the configuration to a remote server every time you write it.

Enable bpduguard on all access ports. It's like an insurance policy against stupid users. Who the hell brings hubs to work anyway? Is it 2012 or am I mistaken? I don't care how progressive your company is but, bring your own spanning tree loops is not going to be OK!

'find problem' and 'fix problem' commands to find and fix problems. Requires Level 16 access.

When all else fails....'Debug all' and walk away. **'Debug all' is typically a resume generating event

Get to know your gear, the users and the usage of your network.
Never stop looking for new features :), the knowledge and the bugs just follow.

Learn to read graphs, packet captures, keep offline copy of any useful info from your network (images, configs, counters, etc.) and learn to automate tasks.

Respect your network and you probably won't bite the dust :))).
Cheers!

P.S. I managed to learn one useful trick in JunOS
-> show config | display set | match
It saved me lots of time. :)

People skills. You can defuse most situations before even looking at the equipment if you can deal with the people. The best engineers I know never advanced very far because they couldn't relate to people.

Always remember we're in the business of COMMUNICATION.

I wish I had learned that you can test port connectivity (TCP) via telnet quite easily.

telnet (host IP) (port)

Typically if a host is reachable via ICMP, it doesn't always guarantee the Service/Application is available.

i.e.
telnet www.google.com 80
Trying 74.125.224.208...
Connected to www.l.google.com.
Escape character is '^]'.
^]
telnet> quit
Connection closed.

Now we not only know that google.com is reachable but it is in fact listening on port 80 (HTTP). I find this helpful for other TCP applications that are behind firewalls and listening on non-standard ports. It's a very quick test on hosts that have telnet capabilities to test client-server connections.

HTH

1.) I hate to think of all the times I waited for a mistyped ping or traceroute to finish.

CLI Escape sequence (for telnet, traceroute, ping, etc)

CTRL+SHIFT+6

1. CLI String Search - Pipe options and regex

Example:

sh run | section ospf
sh run | include vlan|interface
sh run | include bpduguard|interface
sh run | include address|interface

http://www.cisco.com/en/US/docs/ios/12_0t/12_0t1/feature/guide/cliparse.html

quick way to undo all changes without restart :

from enable mode :

configure replace nvram:startup-config force

Many good comments posted already. Wireshark skills is a big one, output filters another one, finding packetlife.net earlier on, and getting familiar with appliances sooner - i.e. Riverbeds, F5, Bluecoat, among others.

One thing I wish that was covered more thoroughly in the Cisco Academy and other studies is hardware. We never took apart a switch or router and went over the internal components. This also leads to upgrading the RAM, IOS, flash, etc.

To be honest, I wished I understood the OSI model earlier, before I used to work with networking devices.

It is essential to understand this model for everyone who works in the IT and has got to do with troubleshooting on a daily basis :)

To prove it's not the network and that it could possibly be the server...

netstat -an |find /i "listening"

Honestly, the most important thing I ever realized about being a network admin is that people are the most expensive resource when it comes to figuring out the cost of things.

Hardware is expensive, but compared to creating workarounds for capabilities your hardware lacks is more expensive. Licenses cost more money then you'd believe, but if they license a product that saves people time on the maintenance side of things, it's miniscule.

people are the most expensive resource around, so any skill you teach yourself that makes you more efficient at doing your job is worth it.

Use scripting (in several scripting languages, somehow the one you know is never available for the system you're working on), regular expressions. Learn to use any management tool efficiently, and what most people overlook, get acquainted with the applications you use for documenting. I cut my documentation time in half when I learned to use onenote and visio well. You're valuable, use yourself well.

That and ofcourse I can't believe you need to get to the CCNP Switch official certification guide before someone writes down the different kinds of fiber and fiber connectors there are. You'd think that knowing what sfp+ module to order would be something cisco would value.

To program using Perl or Python.

Operate Linux.

that all users lie.......

seriously though, that walking someone through troubleshooting steps from A-Z isn't to insult their intelligence, it is to make sure they are completing each step and completing it properly.

and never assume the basics have been covered. Always start from step 1 when troubleshooting, even if another tech has handed the issue off to you.

I don't know much about Backtrack but I wouldn't dual boot if I could get the performance I needed using a VM. It's just too easy to blow it away and recreate if if you screw something up instead of potentially crippling the whole machine.

P D N T S P A

Layer 3, with all the trappings of subnetting, masking, routing tables, ACLs and so on, gets all the emphasis. And it is important to know and understand it. But if you REALLY want to know what's going on within your network, learn layer 2. MAC addresses and ARP tables are not glamorous but layer 2 bridges the gap between the physical and logical network. When you have connection issues, the ability to query a switch can save you a trip. Not recommended, but if necessary, you could map your entire network and identify all the end devices without leaving your chair.

Someone said learn to touch type – I agree.

Someone also said learn a text editor. Yes, but also master a word processor and a spreadsheet. There's a lot of power in the ability to sort and manipulate text and sometimes a word processor is the simpler tool. Importing formatted text into a spreadsheet allows you to to parse numeric information such as addresses, timestamps, and counters and treat them as numbers for sorting and calculations. (Yes, you can do this with scripting but you don't always get to choose your tools. Every user gets the "productivity" tools.)

All layers except the physical layer are human abstractions, not bound by physical laws or written on stone tablets by God. The people who thought this stuff up like patterns and mathematics. So as you start learning this stuff, look for patterns and relationships. Notice that there is a method and organization, which fosters predictability and control. After a while, when presented with something new you can probably recognize the pattern. As a corollary, in your own work, keep things simple and organized by applying patterns that make your network predicable and controllable. (Ex: Don't populate the static DHCP table from a random address the host pulled that morning - set the static DHCP table with easily masked boundaries and control the assignments. Those static addresses will be showing up in ACLs, routing tables, logs, etc. The ability to easily recognize and control (mask or script) static addresses will pay off.)

Don't use the production network as your personal lab. (A previous net admin had done this and every device had a different setup.) Keep the configurations uniform and only as complex as 70% capability of the least competent person on your team. No one performs better than 70% when they're called on Friday night after a full week (and maybe a drink or two) and every team member should be capable of taking the call.

The network does not belong to you. No matter how much you love it, no matter how fun, or how much time you devote to it. It belongs to the business and its only purpose is to facilitate the business' interests with the least amount of expense and effort. Unless you work for an ISP, those "Morons" out there (your fellow employees doing the business of the business) are not required to understand how the network operates. Educate them when you can, but always treat them with patience and be polite. Remember, YOU get to play with the network, THEY have to work. Sweet!