Archive for January, 2011

time for IPv6 at home?

IPv6 has been around since 1998, but has had almost no adoption in the United States.  I’ve been aware of IPv6, but  haven’t paid much attention to it.  Until the last year or so, running v6 wasn’t a trivial task, with few OSes and few home networking products easily supporting it.  Successful IPv6 at home required Linux (no problem) and custom home router firmware (still a minor inconvenience).

Then a friend sent me this link about the DoD pressuring network suppliers to demonstrate a commitment to IPv6 by (at the very least) providing v6 connectivity to their corporate web sites.  Since the article mentioned an old friend, I called him to get some more info.  As we started talking, he told me that his home has been v6 (via a tunnel) for over 3 years.  He’s running all the usual OSes at home, and the initial hurdle had been home networking kit.  Building his home v6 network would be easy today, as most home network vendors, including LinkSys and Apple have IPv6 capable products.

I started checking the blogs of LOPSA members and found a few that have made the leap. Here are a few posts:

With World IPv6 Day coming June 18, don’t forget to check your (and your ISP’s) readiness:

Leave a comment

World IPv6 day is coming – June 8 2011

Haven’t seen much about World IPv6 Day, but the information is out there if you look for it.

Basically, some major Internet services including Google, Facebook, Yahoo!, Akamai and Limelight Networks will offer their content over IPv6 for 24-hours.  The goal is to raise awareness about IPv6 and give companies and organizations  information and experience that will help them prepare for IPv6 to ensure a successful transition as IPv4 addresses run out.

While all currently shipping operating systems (*NIX, Windows and MacOS) have IPv6 stacks, very few end users (at least in the United States) have actual IPv6 connectivity.

Since incorrectly configured “dual stack” systems can see DNS and connection timeouts, you should visit to see if you’ll see any problems on “IPv6 day”.

I’ve asked all the providers in my area, and none (Cox Cable, SpeakEasy DSL) can offer any dates by which they will offer native IPv6.  I would have to build an IPv6 tunnel to get IPv6.  Hopefully the “last mile” providers will sort this out over the next year.

Leave a comment

The more things change (wiretapping the internet)

I feel like we’ve been here before.  The Administration is planning to sponsor legislation to make it easier to (legally) “wiretap the Internet“.  Based on what little has been written, it appears that Justice is arguing that CALEA (and more!) should apply to the Internet.  If that’s the case, then every manufacturer of Internet routing and switching gear would be required to build in the capability for law enforcement to activate a “tap” remotely and with no way for the provider to be aware of it.  Oh, and LE gets decryption assistance, too.

This will not end well.  I don’t have lots of answers, but I’ve got a lot of questions.  Feel free to answer them in the comments 🙂

1. Why bother with the legislation?  The Bush Administration already illegally authorized wiretapping.  Oh, you want the evidence admissible?

2. Which equipment will this apply to?  Large core routers and switches, certainly.  What about my home router?  What about equipment manufactured in China, Russia, Taiwan?  So, all networking gear has to have government approval before installation?  What about a VM appliance, or a home-grown BSD-based firewall?  Will it become illegal to create your own firewall, or use an open source based router/firewall?

3. How will the requirements to support decryption work?  Will US citizens (and companies) be forced to use NERF’ed encryption?  Will the end-to-end SSL/TLS model be deliberately broken to force enabling of  a man-in-the-middle attack?  How will this play against PCI requirements to use best practices.  We’re already seeing massive data spills of credit card and personal data, and the common denominator is often poor or nonexistent encryption.

I don’t claim that there is no need for increased ability for law enforcement to collect and process digital evidence, including network traffic.  That need is real, and in our collective best interests.  But this legislation, as currently described, is impractical and over-reaching, prone to abuse and unenforceable, and completely changes the balance of power between individuals and the government.

Leave a comment


Over the past few years, I’ve learned a great set of new words from my UK counterparts.  Many unfamiliar terms used in the UK have a great history, and sometimes they’re just… so perfect for what we do as system administrators.

Bespoke is such a term. It has many meanings, essentially being “fully custom” or “hand built”, or “hand made”. But is also has deeper meanings, alluding to “exactly fitting your personal needs”, “crafted”, or “personal touch”.  It comes of course, from the tailoring world, where it evokes a sense of old world craftsmanship, a very personal garment, hand measured and hand sewn, just for you.

Bespoke is at one end of the scalability spectrum.  A very few garments, made exactly for a single person.  Expensive and not very many will be made from any single pattern.  Like hand-built systems.

As you move to larger scale production you see much lower costs, but many fewer options and features.  Exactly like the IT world.

Bespoke is fully custom, exactly what you desire, no matter the cost.  Options and extras.  “High touch” support.  At the other end of the spectrum you have high scale, mass production: what you need, no more, no less, but at a more reasonable price.  It is a compromise solution to meet your most important requirements, but one you’ve decided to accept, usually for faster delivery or lower cost.

There is a time and place for bespoke, but increasingly we need to achieve high scalability, as we are increasingly pushed to do more with less.

There are some problems that just can’t be solved at the low end of the scale solution, or aren’t cost effective, or aren’t widely available.  For example, High Performance Computing (HPC) clusters of commodity computers are a high-scale alternative.  Commodity clusters brought supercomputing the masses, or at least to most research group, or smaller companies.  These high-scale solutions are compromises, but good enough and more widely available and affordable.

These cluster are great alternatives to the bespoke supercomputers of the past, such as the early Cray machines.  Our need to solve larger and larger problems, such as the HPCC Grand Challenge problems, eventually required more horsepower than a single, hand-built, bespoke machine could deliver at any affordable cost.

Moving away from bespoke supercomputers allowed us to scale in two ways:  we are able to make very capable systems widely available (at a reasonable cost) and we can grow systems at the high end where cost is not as important, but we need ever-larger capabilities.

As we explore solutions we have to ask ourselves where we need to be on the “bespoke” scale.  Most can’t really afford bespoke, and truly most won’t need it.  Automation allows us to build high-scale systems that provide most (or at least enough) of the features of a bespoke solution, but at an affordable price.


, , ,

Leave a comment

Vendors and Partners

In the sysadmin business, a large part of our job is often to purchase hardware, software and services.  Unfortunately, as technologists we often tend to focus almost exclusively on the products themselves, and not as much on the supplier./

Of course, you have to find the products that meet your technical needs, but there can be additional, non-technical requirements that should be considered.  One of the most important non-technical considerations is the supplier.

In many cases, the supplier can be as or even more important than the product itself.  This is why it is so important to decide if you need a vendor or a partner.

A vendor has a product that they want to sell to you.  It might (or might not) meet your needs, but it’s really up to you to know.  If you buy it and it works as advertised, but doesn’t solve your real problem, that’s still your problem.  You might get great support from the vendor, but support isn’t the real difference.   The product is what it is, take it or leave it.  The transaction with the vendor is just that, an arm’s length transaction.  With a vendor, you get what you pay for (if you are lucky) but rarely more.  A vendor may fall back on the terms of the contract if there’s a problem.  They’ll provide what you’ve paid for, and what they’ve committed to provide, and rarely more.  A vendor is more likely to be selling you a commodity product or service, where there are few or no differentiators other than perhaps price.

Microsoft is the epitome of a vendor.  They have products, and if you need them, you buy them.  You get what they have to sell, no extensive customization, and you’ll get exactly the support that you pay for.  Their success is not tied to yours.  Even if you fail, they have enough other customers that they can still succeed.

A partner is truly different.  You’ll start to know if you have a potential partner from the beginning of the relationship.  A potential partner will be asking question about your company, your culture, your goals, and the problems that you want to solve.   They’ll make sure that you both understand the real problem, not just the problem that their product will solve.  A partner will consider making reasonable changes to their product to better meet your needs.  A partner is more interested in a long-term business relationship that may not pay off for some time, not just making the immediate sale.  A true partner will tell you if they do or don’t have a product to meet your needs.  They may even recommend a competitor’s product, or something from one of their partners.  A true partner may forgo immediate profit if it’s in your best interests.

Don’t expect this to be a one-way street, though.  A partner is making this long-term investment in the relationship in the hope that it will eventually pay off.  Of course, so are you.  After all, they’ve got to make a profit at some point, and you need the additional value that you get from a true partner.  They will be expecting that when they do have the right solution, they will have at least the first shot at making the sale.  They may also expect that they’ll become your preferred vendor.  In extremely strong partnerships, developed over a long period of mutual success, they may expect to at least get extra points on the evaluation scorecard based on their past performance and the strength of the relationship.  A partner can’t be fully successful unless you are.  They’re willing to put some of their skin in the game.

Obviously, what I’ve described are opposite ends of a spectrum.  Few companies will be at either of these extremes.  At each procurement, you’ll have to decide how far you’ll need to look towards these two endpoints.  At the vendor end, you’ll be expecting price to be the biggest differentiator among very similar commodity products and services.  At the partner end, you’ll be buying more custom solutions and expect to pay for that flexibility.

I work for a Japanese company.  Part of the corporate culture is that we seek partners where it’s appropriate, not just vendors.  One of the Japanese values is the expectation that we will form strong relationships with some of our most important suppliers.  For the highest value products we need, we’re expecting that the partner will take on some of the risk as well.  We’re more likely to look for a partner when the project has more risk, when we think we might need significant high-end support, we might need some customization, or when we expect to need to make a large investment over time.

Over the past years, we’ve had the best luck with small to medium companies.  We’ve had the strongest relationships with companies that have done their homework before the first meeting.  We’re part of a large multi-national, with many component companies with similar names.   Potential partners who know who we are, as opposed to our sister companies have taken the first step towards a strong relationship.  Some of these relationships are now eleven years old, some are six or seven years old, and other are still being formed, some have ended.  These partner relationships are constantly being reevaluated.  Some former partners have been replaced with new partners.

The best suppliers have spent more time asking about what we’re trying to accomplish than in telling us about their product line.  In many cases, we’ve been one of their larger (but never their largest) customers.  Their success has been intimately intertwined with ours.   They’ve been willing to make changes to their base products (or create new products) to meet our needs.  We’ve been willing to pay a little more for something that isn’t a run of the mill commodity product.  When we’ve had problems with the products, these companies haven’t stood on the support contract, they’ve gone above and beyond to just make it right.

As you make your purchase decisions, think about what you need in addition to just meeting the technical specifications.  Do you need a vendor or a partner?   Are you willing to make the effort to build the relationships that lead to great partnerships?  Will your management see the value of partnerships for some of your most critical purchases?  That’s part of your job, making the case when it’s the best thing for your company.

Leave a comment

An interview with Maciej Ceglowski of Pinboard

A few weeks ago the “anti-social” bookmarking site Pinboard ( made the news in a big way.  The site experienced hyper-growth due to the news of the possible demise of   Concerns about the future of led tens of thousands of people to look for a new place to store and share their millions of bookmarks.

And quite a few of these people chose Pinboard!  During one 30 hour period around December 18th, Pinboard received over 7 million new bookmarks, more than had been put into the system during its entire life.

I was able to catch up with Maciej for an interview via email.  I wanted to find out more about how Pinboard was operated, and how this huge spike in load had affected administration of the site.  Large-scale system administration isn’t always about hundred of systems, it can also be about tens or hundreds of thousands of users, or unexpected load spikes, or just how you plan for growth.

Read the rest of this entry »


Is it time to outsource your personal IT services?

There comes a time in every syadmin’s life when they have to face facts:  just because you can do your own sysadmin at home doesn’t mean that you should.  In fact, at some point, you won’t want to and you shouldn’t.

Like a lot of sysadmins in my age bracket, I’ve been running at least one personal server at home for at least 15 years.  My own email, web server(s), Usenet News server, DNS, and some other services have run on a succession of operating systems and hardware somewhere in my home for at least that long.

I’ve run servers at home on everything from a Commodore Amiga, a DECStation, and a SPARCstation, and then several generations of commodity PC hardware.  I’ve run everything from Ultrix to SunOS 4, to several flavors of Linux.  Some of my older machines had up to 8 drives and drew up to 600-700 W, to my latest machine which has one drive (larger than all the drives in all the prior drives put together!) and draws less than 50W.

In the “olden days” you pretty much had to run your own server if you wanted private email on your own domain name.  It was that, or AOL, or email at your employer.   Commercial web hosting, outsourced email, DNS and Usenet News just weren’t available.  Besides, we all knew sendmail (and later postfix) inside and out for our day jobs, and Apache was new and exciting.  Running your own DNS was a measure of your “clue factor” and Usenet News, was, well, news.

Fortunately, you now have lot of options.  You can outsource all of these services, in many cases for free.  In some cases, free and better than you can run yourself.

After one too many hardware failures at home, my wife moved her blog to another hosting provider.  That worked out well, as I no longer have to track all the latest security issues with WordPress, I no longer have to “declare a maintenance window” at home to play with the home server, and frankly she’s getting better service.  I also don’t have to worry about backups!

When it came time to spin up a blog for my high school reunion, I put that in managed commercial hosting at a provider that specifically supports WordPress.  I still had to deal with my own installation of WordPress on their server, though.  So on my next blog, I went into full-on “blog hosting” here at

I started moving my personal email into “the cloud” last year.  Even though I have email inbound to more than ten different personal domains, I’ve been able to move many of those either directly into Gmail or into (free) Google Apps.  My main motivation was to get some better anti-SPAM and worry less about backups.  Even though I was running multiple DNS blacklists, graylisting, custom procmail and SpamAssassin, I was still getting too much SPAM.  Gmail offered a better anti-spam solution and meant that I could just stop dealing with so much useless time-wasting SPAM.  It was taking so much time to weed through the clutter each day that I had almost given up on my personal domains for email.  Gmail gave me back my inbox.

I noticed that as I “matured” in my use of outsourced services, I’ve moved up the “stack” from Infrastructure as a Service (IaaS), to Platform as a Service (PaaS) and I’m now fully comfortable at the Application as a Service (AaaS) level.

I still have some things running at home, and in the New Year I’ll be evaluating each and every one of them for outsourcing opportunities.  In fact, I’ll use the same evaluation criteria I use at work:  First, is it a core competency?  Second, is it something I understand well enough to write (or sign) a meaningful contract with measurable success criteria?  Third, will I save time or money, or become more agile by outsourcing?

(Outsourcing is almost never a pure cost savings.  It just isn’t.  You might get cost savings in the form of avoiding hiring, or getting some valuable staff time back to work on internal projects.  You might get faster turnaround time or faster time to market,  but you will rarely save money.)

Careful outsourcing, whether at home or at work, can have a very liberating effect:  It can take the well-understood (and perhaps no longer interesting) work off your plate, to give you time and energy to focus on the new, interesting and higher-value projects that you really want to take on.

Getting out of the home email and web hosting business has given me more time to spend with my family and and more time to spend on interesting new things.  And a lot less to worry about in terms of backups and uptime.  What will you get out of it?

Leave a comment

%d bloggers like this: