Archive for category the business of system administration
I feel like we’ve been here before. The Administration is planning to sponsor legislation to make it easier to (legally) “wiretap the Internet“. Based on what little has been written, it appears that Justice is arguing that CALEA (and more!) should apply to the Internet. If that’s the case, then every manufacturer of Internet routing and switching gear would be required to build in the capability for law enforcement to activate a “tap” remotely and with no way for the provider to be aware of it. Oh, and LE gets decryption assistance, too.
This will not end well. I don’t have lots of answers, but I’ve got a lot of questions. Feel free to answer them in the comments 🙂
1. Why bother with the legislation? The Bush Administration already illegally authorized wiretapping. Oh, you want the evidence admissible?
2. Which equipment will this apply to? Large core routers and switches, certainly. What about my home router? What about equipment manufactured in China, Russia, Taiwan? So, all networking gear has to have government approval before installation? What about a VM appliance, or a home-grown BSD-based firewall? Will it become illegal to create your own firewall, or use an open source based router/firewall?
3. How will the requirements to support decryption work? Will US citizens (and companies) be forced to use NERF’ed encryption? Will the end-to-end SSL/TLS model be deliberately broken to force enabling of a man-in-the-middle attack? How will this play against PCI requirements to use best practices. We’re already seeing massive data spills of credit card and personal data, and the common denominator is often poor or nonexistent encryption.
I don’t claim that there is no need for increased ability for law enforcement to collect and process digital evidence, including network traffic. That need is real, and in our collective best interests. But this legislation, as currently described, is impractical and over-reaching, prone to abuse and unenforceable, and completely changes the balance of power between individuals and the government.
- Report: FBI Talks To Google, Facebook About Wiretapping (searchengineland.com)
- Wider Web Wiretap Law Is Sought (nytimes.com)
In the sysadmin business, a large part of our job is often to purchase hardware, software and services. Unfortunately, as technologists we often tend to focus almost exclusively on the products themselves, and not as much on the supplier./
Of course, you have to find the products that meet your technical needs, but there can be additional, non-technical requirements that should be considered. One of the most important non-technical considerations is the supplier.
In many cases, the supplier can be as or even more important than the product itself. This is why it is so important to decide if you need a vendor or a partner.
A vendor has a product that they want to sell to you. It might (or might not) meet your needs, but it’s really up to you to know. If you buy it and it works as advertised, but doesn’t solve your real problem, that’s still your problem. You might get great support from the vendor, but support isn’t the real difference. The product is what it is, take it or leave it. The transaction with the vendor is just that, an arm’s length transaction. With a vendor, you get what you pay for (if you are lucky) but rarely more. A vendor may fall back on the terms of the contract if there’s a problem. They’ll provide what you’ve paid for, and what they’ve committed to provide, and rarely more. A vendor is more likely to be selling you a commodity product or service, where there are few or no differentiators other than perhaps price.
Microsoft is the epitome of a vendor. They have products, and if you need them, you buy them. You get what they have to sell, no extensive customization, and you’ll get exactly the support that you pay for. Their success is not tied to yours. Even if you fail, they have enough other customers that they can still succeed.
A partner is truly different. You’ll start to know if you have a potential partner from the beginning of the relationship. A potential partner will be asking question about your company, your culture, your goals, and the problems that you want to solve. They’ll make sure that you both understand the real problem, not just the problem that their product will solve. A partner will consider making reasonable changes to their product to better meet your needs. A partner is more interested in a long-term business relationship that may not pay off for some time, not just making the immediate sale. A true partner will tell you if they do or don’t have a product to meet your needs. They may even recommend a competitor’s product, or something from one of their partners. A true partner may forgo immediate profit if it’s in your best interests.
Don’t expect this to be a one-way street, though. A partner is making this long-term investment in the relationship in the hope that it will eventually pay off. Of course, so are you. After all, they’ve got to make a profit at some point, and you need the additional value that you get from a true partner. They will be expecting that when they do have the right solution, they will have at least the first shot at making the sale. They may also expect that they’ll become your preferred vendor. In extremely strong partnerships, developed over a long period of mutual success, they may expect to at least get extra points on the evaluation scorecard based on their past performance and the strength of the relationship. A partner can’t be fully successful unless you are. They’re willing to put some of their skin in the game.
Obviously, what I’ve described are opposite ends of a spectrum. Few companies will be at either of these extremes. At each procurement, you’ll have to decide how far you’ll need to look towards these two endpoints. At the vendor end, you’ll be expecting price to be the biggest differentiator among very similar commodity products and services. At the partner end, you’ll be buying more custom solutions and expect to pay for that flexibility.
I work for a Japanese company. Part of the corporate culture is that we seek partners where it’s appropriate, not just vendors. One of the Japanese values is the expectation that we will form strong relationships with some of our most important suppliers. For the highest value products we need, we’re expecting that the partner will take on some of the risk as well. We’re more likely to look for a partner when the project has more risk, when we think we might need significant high-end support, we might need some customization, or when we expect to need to make a large investment over time.
Over the past years, we’ve had the best luck with small to medium companies. We’ve had the strongest relationships with companies that have done their homework before the first meeting. We’re part of a large multi-national, with many component companies with similar names. Potential partners who know who we are, as opposed to our sister companies have taken the first step towards a strong relationship. Some of these relationships are now eleven years old, some are six or seven years old, and other are still being formed, some have ended. These partner relationships are constantly being reevaluated. Some former partners have been replaced with new partners.
The best suppliers have spent more time asking about what we’re trying to accomplish than in telling us about their product line. In many cases, we’ve been one of their larger (but never their largest) customers. Their success has been intimately intertwined with ours. They’ve been willing to make changes to their base products (or create new products) to meet our needs. We’ve been willing to pay a little more for something that isn’t a run of the mill commodity product. When we’ve had problems with the products, these companies haven’t stood on the support contract, they’ve gone above and beyond to just make it right.
As you make your purchase decisions, think about what you need in addition to just meeting the technical specifications. Do you need a vendor or a partner? Are you willing to make the effort to build the relationships that lead to great partnerships? Will your management see the value of partnerships for some of your most critical purchases? That’s part of your job, making the case when it’s the best thing for your company.
This post started as a comment on an article at Ars Technica announcing the demise of the Apple Xserve. Too many of the comments focused (as expected) on the unfairness of all the evil corporations and IT departments that “never gave the Xserve a chance”.
Not exactly. Here’s an updated version of my response….
As one of those “IT people that everyone hates because we didn’t have enough faith in Apple”, let me tell the story from a different side.
We tried to buy Apple for the business. Really. For the past six years. We have an Apple Enterprise support team. Apple “Enterprise” has been a bad joke, from the strategy and pre-sales standpoint. The Enterprise people themselves did a very good job, given the way their hands were tied. It was always obvious that they wanted to be helpful, but Apple’s policies made that difficult.
We bought Xserver where it fit, like Xsan, some render farms, Open Directory servers, etc.
I buy servers. Lots of servers. Sometimes its been almost a thousand servers a year. Sometimes only a few hundred. Mostly to run Linux, but there’s still a fair amount of Windows.
Almost all the sysadmins on my team use MacBook Pro laptops, like the one I’m using now. Some of us (like me) have Mac Pro desktops. Many of us are Mac at home, too. My wife has a Mac Mini at home and will never have another Windows product. For the sysadmins, MacOS is the right combination of *stability*, nice GUI, tools and ability to also run *lots* of PERL and Python to get stuff done. Frankly, its a better desktop interface than Linux, yet the BSD underpinnings and the shell make it a great platform to use to support Linux.
Go to a system administration conference, like LISA or Cascadia IT and see the prevalence of Mac laptops. Talk to the people on the LOPSA mailing lists or in their #lopsa IRC channel and see how many use MacOS laptops (or wish they could). Then ask how many Xserves they have.
At the large scale (say over 50 or 100 servers), it’s all about planning, power consumption, manageability and support. Let’s see how the Apple “Enterprise” products stack up:
1. Planning. I have to have a 1-2 year roadmap for server products. Longer is better. I need to know what’s coming down the pipe so that our team can plan how to support our customers. We can also help those users plan for what they will need to solve their future problems, as we plan datacenter space, power, support, and most importantly, budgets. My team (or one of our sister IT groups) gets these roadmaps from IBM, Dell, Rackable (now SGI), EMC, Network Appliance, BlueArc, Cisco, Juniper, and all our major software vendors. Hell, I can practically get a roadmap from Linux, just be looking at what’s in the current development kernels. Mozilla publishes better roadmaps than Apple has ever dreamed of. We can plan around these roadmaps, and in general these vendors act as partners, letting us know what’s coming so that we can even influence some final design and pricing decisions.
There’s no way I’m going to ask management for hundreds of thousands of dollars (or even a $million or more) without a long-range plan.
We have mutual NDAs with Apple (as well as all those other companies). What does that get us?
“Apple does not comment on unannounced products”.
2. Power consumption. Datacenter costs are directly related to and dominated by power consumption. More power means higher costs, period. A current generation server in the right configuration can be well under 200W, if you aren’t stuffing it full of local disk. Using Low-voltage CPUs and chipsets like the LV Xeons and its descendants can save thousands of dollars per month if you have lots of servers. Smart power management like HP’s and IBM’s, DC powered products like SGI, disk spin down (MAID), and all the other things that “Enterprise” vendors do to control power save us money. Commercial collocation (in most cases) gets you three ways on power. You have a max power input per square foot, a max heat BTUs out per square foot (calculated based on input power in most cases), and of course they also charge you for the power. The more power you draw, the more square feet you need to lease, the more it costs.
What did the Xserve do in the area of power management?
3. Manageability. At “Enterprise scale”, it’s all about managing servers as groups, not individual servers. And remotely. With 100, 500, 1000, or 5000 servers, if you are managing them individually “by hand” you are losing. (You actually lost at about 5-10.) You have to get the “server:sysadmin” ratio up to something reasonable. At a smaller shop you might only need 10:1. At the mid-scale, 500:1 is pretty cool. At one point our org hit 750:1 for a while, but that exposed some limits in our tools. I would imagine at the Amazon/Google scale, they are up over 1000:1. This means server management software. Whether it’s cfengine, puppet, bcfg2 or some other open source product, or “server orchestration” products from IBM, HP, CA or similar, it has to be there. MacOS Server never got much traction with the commercial products (that I am aware of) and Apple never offered any options of their own.
And, what about servers all around the world? While it would be fun to fly to Amsterdam to put a CD or thumbdrive in a sever for an upgrade or an install, or to do that rare power off hard reset, that’s unfortunately not practical. If you are in collocation, they will be happy to go “touch” your hardware, usually at $100-$200/hour and the minimum charge is for one hour. No lights out management, no Enterprise market. The Xserve iLOM supposedly wasn’t bad, but with no iLOM on the new product, well…
4. Support. AppleCare is not Enterprise support. Enterprise support is more than just having a phone number to bypass the consumer support hotline. How about coming out to do critical firmware updates on a mission critical host? Stocking spare drives in my datacenter so that we can swap a drive in 10 minutes instead of waiting for “4 hour support” to return a phone call, and then shipping a drive to us overnight? I could buy spare drives myself, but an Enterprise vendor will offer the option of “depoting” parts in my datacenter at their cost. HP, IBM, Sun, etc have all done this for decades.
Finally, look at what Apple runs themselves. As an earlier poster pointed out, Apple isn’t even pretending to run their business on their kit. If they won’t even try, why should you? Having visited Sun, IBM, Dell and HP datacenters over many years, I can assure you that those companies are serious about the Enterprise. They are running (and betting) their own companies on their own products and ability to deliver.
That’s a commitment that Apple never even tried to make.