essential (book) references for SysAdmins

Last week was LOPSA‘s 5th birthday and this is LISA week in San Jose.  Both are focused on the system administration community.  One perennial topic in the LOPSA mailing lists and IRC channel, and at the LISA conference is “How do you become (or become a better) system administrator?”

Of course, one of the best ways to become a better sysadmin is by trying and doing more and different sysadmins tasks.  Another good way is to hang out (or be mentored) by someone with more (or different) skills.

Another way is to hit the books…

System Administration is a very broad field, spanning all manner of technologies.  Whether it is a specific operating system, or networking or storage or databases….  there are core skills that apply.  Time management, technical planning, troubleshooting and interpersonal skills, for example.

No matter what your specialization (if any) in system administration, there are a few books that I consider necessary.  If you haven’t read these, you’re missing out, and may end up spending a lot of time re-inventing some aspect of system administration.

• The Practice of System and Network Administration, affectionately know as “TPOSNA” is the current essential desk reference.  Limoncelli, Hogan and Challup are all recognized leaders in the field, and this is in part the sum of all their combined years of experience.
• UNIX and Linux System Administration Handbook is the latest (July 2010!) incarnation of the Linux Administration Handbook (2006), which is itself a major re-write of the classic UNIX System Administration Handbook, which set the standard for system administration reference books back in 2000.  The team of Nemeth, Snyder, Hein and Whaley have delivered a fitting update to the series.  Keep this one by your keyboard.
• Essential System Administration by Frisch is a favorite of mine because it ventures into some of the “more different” UNIX flavors, including AIX and HP-UX.

Moving up from the deeply technical, I recommend these books to sharpen your softer skills:

• Time Management for System Administrators (another book by Tom Limoncelli) is more useful than the more generic time management books (Covey, etc.) because it offers specific solutions based on the way that system administrators actually work.  Achieving the necessary balance between firefighting and projects is an essential skill, and this book can really help here.

I’ll be back to talk about references for system administration managers.  If you’ve been thrust into management, you’ll want to check back for that installment…

Apple abandons Xserve – why should anyone care?

This post started as a comment on an article at Ars Technica announcing the demise of the Apple Xserve.  Too many of the comments focused (as expected) on the unfairness of all the evil corporations and IT departments that “never gave the Xserve a chance”.

Not exactly.  Here’s an updated version of my response….

As one of those “IT people that everyone hates because we didn’t have enough faith in Apple”, let me tell the story from a different side.

We tried to buy Apple for the business. Really. For the past six years. We have an Apple Enterprise support team. Apple “Enterprise” has been a bad joke, from the strategy and pre-sales standpoint. The Enterprise people themselves did a very good job, given the way their hands were tied.  It was always obvious that they wanted to be helpful, but Apple’s policies made that difficult.

We bought Xserver where it fit, like Xsan, some render farms, Open Directory servers, etc.

I buy servers. Lots of servers. Sometimes its been almost a thousand servers a year. Sometimes only a few hundred. Mostly to run Linux, but there’s still a fair amount of Windows. Almost all the sysadmins on my team use MacBook Pro laptops, like the one I’m using now. Some of us (like me) have Mac Pro desktops. Many of us are Mac at home, too. My wife has a Mac Mini at home and will never have another Windows product. For the sysadmins, MacOS is the right combination of *stability*, nice GUI, tools and ability to also run *lots* of PERL and Python to get stuff done. Frankly, its a better desktop interface than Linux, yet the BSD underpinnings and the shell make it a great platform to use to support Linux.

Go to a system administration conference, like LISA or Cascadia IT and see the prevalence of Mac laptops.  Talk to the people on the LOPSA mailing lists or in their #lopsa IRC channel and see how many use MacOS laptops (or wish they could).  Then ask how many Xserves they have.

At the large scale (say over 50 or 100 servers), it’s all about planning, power consumption, manageability and support. Let’s see how the Apple “Enterprise” products stack up:

1. Planning. I have to have a 1-2 year roadmap for server products. Longer is better. I need to know what’s coming down the pipe so that our team can plan how to support our customers.  We can also help those users plan for what they will need to solve their future problems, as we plan datacenter space, power, support, and most importantly, budgets. My team (or one of our sister IT groups) gets these roadmaps from IBM, Dell, Rackable (now SGI), EMC, Network Appliance, BlueArc, Cisco, Juniper, and all our major software vendors. Hell, I can practically get a roadmap from Linux, just be looking at what’s in the current development kernels. Mozilla publishes better roadmaps than Apple has ever dreamed of. We can plan around these roadmaps, and in general these vendors act as partners, letting us know what’s coming so that we can even influence some final design and pricing decisions.

There’s no way I’m going to ask management for hundreds of thousands of dollars (or even a $million or more) without a long-range plan.

We have mutual NDAs with Apple (as well as all those other companies). What does that get us?

“Apple does not comment on unannounced products”.

2. Power consumption. Datacenter costs are directly related to and dominated by power consumption. More power means higher costs, period. A current generation server in the right configuration can be well under 200W, if you aren’t stuffing it full of local disk. Using Low-voltage CPUs and chipsets like the LV Xeons and its descendants can save thousands of dollars per month if you have lots of servers. Smart power management like HP’s and IBM’s, DC powered products like SGI, disk spin down (MAID), and all the other things that “Enterprise” vendors do to control power save us money. Commercial collocation (in most cases) gets you three ways on power. You have a max power input per square foot, a max heat BTUs out per square foot (calculated based on input power in most cases), and of course they also charge you for the power. The more power you draw, the more square feet you need to lease, the more it costs.

What did the Xserve do in the area of power management?

3. Manageability. At “Enterprise scale”, it’s all about managing servers as groups, not individual servers. And remotely. With 100, 500, 1000, or 5000 servers, if you are managing them individually “by hand” you are losing. (You actually lost at about 5-10.)  You have to get the “server:sysadmin” ratio up to something reasonable. At a smaller shop you might only need 10:1. At the mid-scale, 500:1 is pretty cool. At one point our org hit 750:1 for a while, but that exposed some limits in our tools.  I would imagine at the Amazon/Google scale, they are up over 1000:1. This means server management software. Whether it’s cfengine, puppet, bcfg2 or some other open source product, or “server orchestration” products from IBM, HP, CA or similar, it has to be there. MacOS Server never got much traction with the commercial products (that I am aware of) and Apple never offered any options of their own.

And, what about servers all around the world? While it would be fun to fly to Amsterdam to put a CD or thumbdrive in a sever for an upgrade or an install, or to do that rare power off hard reset, that’s unfortunately not practical. If you are in collocation, they will be happy to go “touch” your hardware, usually at $100-$200/hour and the minimum charge is for one hour.  No lights out management, no Enterprise market. The Xserve iLOM supposedly wasn’t bad, but with no iLOM on the new product, well…

4. Support. AppleCare is not Enterprise support. Enterprise support is more than just having a phone number to bypass the consumer support hotline. How about coming out to do critical firmware updates on a mission critical host? Stocking spare drives in my datacenter so that we can swap a drive in 10 minutes instead of waiting for “4 hour support” to return a phone call, and then shipping a drive to us overnight? I could buy spare drives myself, but an Enterprise vendor will offer the option of “depoting” parts in my datacenter at their cost.  HP, IBM, Sun, etc have all done this for decades.

Finally, look at what Apple runs themselves. As an earlier poster pointed out, Apple isn’t even pretending to run their business on their kit. If they won’t even try, why should you? Having visited Sun, IBM, Dell and HP datacenters over many years, I can assure you that those companies are serious about the Enterprise. They are running (and betting) their own companies on their own products and ability to deliver.

That’s a commitment that Apple never even tried to make.