Archive for January, 2013

LOPSA San Diego – first meeting

LOPSA San Diego is getting underway with a Meetup next week:

Thursday Jan 24 2013

6pm until whenever

Callahan’s Pub in Mira Mesa

This will be a social, meet and greet meeting.  No presentations at this one, but come out and meet some of the fine sysadmins in the San Diego area. Come out and meet your peers, network, talk shop, commiserate and celebrate all things syasdmin.

LOPSA is an international professional society for IT people of all job descriptions.

If you’re planning to attend, please RSVP at so we can get a headcount ahead of time. Of course, if you can only make it at the last minute, you’re very welcome too! (We understand the life of a sysadmin!)

Our members manage everything from desktops to servers, storage to networks, laptops to supercomputers. Come out and get connected to the rich sysadmin community in San Diego!

Leave a comment

When will support IPv6?

Dear, when will you support IPv6?

Over the past year I’ve watched more and more web sites come online on IPv6.  Some are “the usual suspects”; the high-tech, early adopter sites that you expect to be moving aggressively onto IPv6. Some early adopters have been surprising. While the WordPress software itself works fine over IPv6, itself seems to be a no show to the IPv6 game.

In fact, the only mention I can find from WordPress about IPv6 hosting is a blog post from World IPv6 Day in 2011.

This blog (hosted at has a lot of content about IPv6, and I get about one private comment every other month pointing out the irony that the blog can’t be viewed over IPv6.  I’d rather not move to (fully IPv6 capable), or spin up my own instance of WP if I an avoid it.

So, WordPress folks, can you at least give a timeframe for IPv6 support?

Since I’m an architect on a worldwide enterprise internal IPv6 rollout, I *do* understand the challenges involved, and the uncertainty that you might have on a fixed schedule.  But could we get at least a comment that “we’re working on it”, or “sometime in Q4 2013”, or “not planning to do this for at least a few years”?


No knowledge of #IPv6 at DSL Extreme (@DSLX_Status)

I’m still on my quest for native IPv6 at home.  Today’s attempt: DSL Extreme. Result: FAIL

The pricing from DSL Extreme looks very attractive, less than half that what I’m currently paying Megapath. However, based on my call to DSL Extreme support, you get what you pay for. While Megapath doesn’t have any announced plans for IPv6, their sales people knew exactly what it is, and my SpeakEasy/Covad/Megapath DSL has had 2 outages (of less than an hour each) in 10 years.

Let’s start with the extent of the IPv6 info on the DSL Extreme web site. Searching for IPv6 leads you to a knowledgebase article on reverse DNS. First off, this page is a complete lift from another web site,, except DSL Extreme took out the formatting that makes the page readable. Second, there’s pretty much no IPv6 info on this page.

So, I decided to try their sales/support department. That phone call didn’t go well, either.

Tech support had never heard of IPv6. Period. Full stop.  Sort of.  At first they tried to tell me that their “dynamic addresses are v6 and static are the old ones.” Then I spent almost 5 minutes trying to get across the idea that there actually was this “new thing called IPv6” and that I wanted to sign up for their service if they supported it. The best I could get from them was, and I quote, “The only addresses we have are dynamic and static”. I guess they have both kinds of music too (country AND western).

I’m so very glad that my IPv6 tunnel to has been 100% rock solid. If it wasn’t for Hurricane Electric, it might be another year or more until I could start learning and use IPv6 at home. Thanks to them, I’ve already been running IPv6 at home for 13 months!

UPDATE:  The above was written Dec 28 2012.  Today (8 Jan 2013) I got a response to my query into sales, from about 10 days ago:

Thank you for your email. Unfortunately we do not provide IPv6 and as of right now we have no intentions of providing it.
DSL Extreme.

Leave a comment

No #IPv6 from Cox Communications (@CoxComm)

I spent entirely too much time the other day, trying to get IPv6 information from Cox Communications. It was not a good experience.

I spent about 10 minutes trying to find anything useful about IPv6 on their web sites.  I found a FAQ, which has some IPv6 technical information, but nothing about roll outs, trials, schedules or plans, other than “residential trials in 2013”.  At least that’s more than some other ISPs have been able to tell me.

So I decided to try Cox online chat “support”. This was so full of fail that I was compelled to share the experience. It looks like they’ve outsourced support to something called “”. While this conversation looks short, it actually took more than 8 minutes, and it looks like their support doesn’t speak English very well. I could overlook that, if they could answer my question.

Screen Shot 2012-12-28 at 12.34.10 PM


It was 30 years ago today….

… that I logged into a UNIX system for the first time. It was also three days after the ARPANET transitioned to TCP/IP, and my first day at a new job.

The place was Logicon, in San Diego. The system was Programmers Work Bench (PWB) UNIX on a DEC PDP-11/70. The system was in single user mode, since the root filesystem was corrupt and the senior programmers who might have been able to help had quit during the month before. I think that the last one was out of the country for the next 3 weeks. And, my new (completely non-technical) boss had actually been hired after I was; but he was starting at Logicon the same day. It was an interesting beginning to my new job.

I spent that first week at my new job learning enough UNIX to figure out the icheck, dcheck and ncheck commands to repair the filesystem. I eventually got the root filesystem fixed, was able to create an account for myself and bring the system back up into multi-user mode. File systems were much simpler then. So simple in fact that I later learned to use the ed editor to repair (or just change) filenames by editing the directory files.

As soon as the PWB system came back up, I learned that we were now “off net” as the ARPANET had transitioned from NCP to TCP/IP on the 1st of January, and there was no one to port and debug the needed TCP/IP stack for our system. We needed that connection to communicate with our government customer, deliver software and work on our contract for the US Navy.

Fortunately, Logicon re-hired one of the senior programmers; he spent January and part of February working on the TCP/IP code to get us back on the ARPANET. We shared an IMP with the Naval Ocean Systems Center (NOSC, now part of SPAWAR) and UCSD. I met two long-time friends, Ron (@NOSC) and Brian (@UCSD) through our ARPANET connection, and we still keep in touch.

I always remember “ARPANET flag day“, because that’s when I got my start with UNIX and the ARPANET. That led to work on the Internet, HPC, and computer security.

Strangely enough, part of my current job is to work on a transition to IPv6. I have now had direct experience with IPv4, IPv5 and IPv6.

I owe a large debt  to all the people I’ve worked with and the USENIX (and later LOPSA) community of friends. You’ve all been wonderfully helpful, often insightful, and always friendly. Thank you all.

Leave a comment

An interview with Owen DeLong, IPv6 Evangelist, Hurricane Electric

This is an email interview with Owen DeLong , the “IPv6 Evangelist” for Hurricane Electric, one of the largest  ISPs in the world.  They are also operators of, one of the largest IPv6 tunnel providers in the world.

Owen has been in the industry in many roles, including work at Sun, Exodus Communications and TellMe. In addition to his “day job”, he’s also on the advisory council for ARIN, teaches SCUBA, and makes lots of contributions to network-related mailing lists. He’s also one of the very few people I know who has an AS number for his home net!

We’ve been wanting to do this interview for quite a while, but things have been rather busy the past year. When I ran into Owen (again) at LISA in San Diego a few weeks back, we talked about IPv6 and this interview.  Owen was kind enough to work on this with me over the holiday.

Owen, you’ve been an “IPv6 Evanagelist” for Hurricane Electric for a while now.  How did you become the IPv6 Evangelist? What does that entail, and what did you do before IPv6 needed an evangelist?

I’ve been various forms of Systems Administrator, Network Engineer, Backbone Engineer, Senior Backbone Engineer, Network Architect and even Operations Manager over the years. When my last startup was on the verge of imploding, I mentioned to someone at HE who does similar work that I was interested in doing something similar. We talked extensively in Manila (APNIC conference) and the rest as they say, is history.

Can you tell me a little about Hurricane Electric?

We are actually one of the largest backbone network providers in the world. We are the number 1 most peered IPv6 network and the number 3 most peered IPv4 network according to CAIDA. We operate multiple 10-Gig backbones throughout North America, Europe, and Asia.

We’ve run into each other at several conferences such as Game Developer and LISA. What other conferences do you attend, and what’s the feedback from people about IPv6?

I try to attend (and speak at) a wide variety of events and conferences. I usually speak at at least one ION event, I’ve also spoken at Astricon, Pubcon, Future of Web Applications, various NOGs (Network Operators Groups) such as NANOG. I’ve spoken to a variety of Linux, Unix, an Open Source groups such as BALUG, NLUUG, etc. I’ve spoken to Cisco and Juniper user’s groups and more. I also speak at many of the IPv6 oriented conferences such as the North American IPv6 summit, the Texas IPv6 summit, etc.

Most of the audiences I speak to are pretty receptive. Certainly there is more and more interest and less and less skepticism over the last 3 years.

IPv6 has been a standard now for almost 18 years (if you use RFC 1883 as the reference).  Why has adoption been so slow?

Because we made the mistake of introducing NAT and people are busy.  A network not perceived as broken tends to limp along with whatever level of brokenness it has come to accept until perturbed by an additional factor such as CGN, address exhaustion, or other external pressures like the continued ability to reach destinations in Asia.

So far, IPv4 runout hasn’t really exerted those pressures to any great extent on anyone yet, but at least people are starting to see that it is coming and IPv6 adoption is starting to accelerate quite a bit now.

According to Google, IPv6 was around 0.25% of internet traffic at the beginning of 2010 and 2011. The biggest difference between the two being that at the beginning of 2010, it was about 60% 6to4/Teredo and at the beginning of 2011 it was about 80% native, but the overall growth was pretty flat. By the end of 2011, it was 0.4% and almost entirely native with 6to4 and Teredo almost unmeasurable. As we approach the end of 2012, we’re seeing more than 1.1% with 6to4 and Teredo almost eliminated.

At first blush, those seem like pretty low numbers. However, 1% of internet traffic today is a lot more than 100% of internet traffic when IPv4 was 18 years old.  Also, consider in 2011, that’s a 100% growth rate and in 2012, it’s approximately 175% growth, so not only is adoption growing, but the rate of adoption is accelerating quite rapidly.

If the curve continues to ramp towards vertical, we might see more than 5% by the end of next year and more than 20% by the end of 2014. That’s without factoring in the additional pressures from actually running out of IPv4 addresses at the service providers and the fact that ARIN will be basically out of address space most likely in the middle of next year.

On the technical side, what’s the biggest impediment to implementation?  Is that the same for ISPs, Enterprises and end users, or do they all face different challenges?

No, they each face unique challenges.

For the end user, it’s the lack of IPv6 support in consumer products. There’s a host of products that lack IPv6 support today which is severely impacting consumers. These include familiar household names like Playstation, TiVO, Yamaha, Samsung, LG, and all of the Matsushita brands (JVC, Technics, Pioneer) and more. In fact, I don’t know of a single CE manufacturer that has embraced IPv6 in their products yet. IMHO, this is inexcusable.

None of these vendors have yet seen fit to deploy IPv6 at the consumer level in their products and this is becoming a major impediment.

To look at ISPs, we really need to divide them into two principal categories… The B2B and/or Backbone-oriented ISP, such as Hurricane Electric, where there really are very few remaining technical impediments and the few remaining impediments are primarily related to educating executive management.

The other category is what I will call the last-mile ISP. The providers that serve SOHO, Residential, and Small Business customers and perhaps some medium sized businesses as well. These ISPs are still facing shortcomings in CPE and in their provisioning, management, and deployment systems. Many DSLAMs and BRAS units have limited or no IPv6 capability. Several of the CMTS vendors still have show-stopping bugs to work out (This means you, brand C and others).

For the enterprise, there are a few technical impediments, but these can mostly be addressed by current technologies. The larger hurdles in business really are the education of executive management and communicating to the enterprise that this isn’t just a networking issue.

How about on the Business side?

I think this is easier to answer. Across the board for all of those organizations, the number one business challenge is educating executive management to the point that they understand this to be a critical organization-wide issue that must be addressed starting now (if they haven’t started already).

Any organization considering IPv6 seriously needs to understand that this is an organization-wide issue and that it touches EVERY group within the organization. Systems administrators need to be involved because this will touch every application, every system, every server, and will impact how they do their jobs on a daily basis. The help desk needs to get involved because this will represent some pretty major changes in their troubleshooting scripts and will also require them to understand more about how things operate in a dual-stack environment. Application developers, maintainers, administrators, etc. are involved because any application that communicates via the network needs to be examined and tested against a dual-stack and eventually an IPv6 only environment. The mail room has to expect that there’s going to be a lot of logistical support required for getting all of these equipment updates to the right locations at the right times. Management at all levels needs to recognize the need to make this an organization-wide priority and address it in an orderly fashion before it becomes an outright emergency.

Lots of networks seem to be taking a dual-stack approach and adding IPv6 to their existing IPv4 network.  What do you think of this approach?

I think it’s the only feasible approach for the time being. We’ll reach a point where we have to turn off IPv4 and move forward to IPv6 single-stack and that’s a good thing. It should always be the end goal. However, just as we didn’t turn off Novell the day we turned on IP in that great migration of the enterprise, we’re not going to be able to turn off IPv4 and turn on IPv6 overnight. We’re going to have to live with both for some time to come.

For people that want to learn more about IPv6, or even implement it on their home networks, where should they start?

I highly recommend our on-line training available at It’s pretty easy to get through, but it does require you to not only learn about IPv6, but to demonstrate that you can actually make things work with IPv6 in order to obtain the higher certification levels.

You’ve mentioned in the past that there are new IPv6-only customers coming online in areas where the IPv4 space is already depleted. How many people are we talking about and where are they? How many potential customers could a company “miss” by delaying an IPv6 roll out?

Well, there are currently a little more than 2.5 billion people on the internet. There are more than 6.8 billion people on earth. So, over time, I expect we’ll see most of the remaining 4.3+ billion people (and growing) connecting on IPv6 and unable to get IPv4 addresses. Immediately, it’s relatively small numbers, probably a few thousand her and there. However, that number can only continue to grow as time progresses.

One of the biggest complaints we all hear is that there’s no “killer app” and “no ROI” for implementing IPv6. What risks (if any) are companies taking by delaying IPv6 adoption?

If it isn’t clear from what I’ve said above, I’ll put it this way. The IPv4 internet cannot keep growing much larger than it currently is. The IPv6 internet will, on the other hand, continue to grow and for a while, that growth will continue to accelerate. As a result, it won’t be very many years before the fraction of the internet that is reachable via IPv4 is much smaller than that portion which is reachable via IPv6. If your business depends on being connected to the whole internet, then, that’s the killer app. for IPv6. If you’re willing to live with only being connected to the part of the internet that exists today, then you may have a few years before you have to worry about that shrinking, but, I suspect it’s not as many years as a lot of people will claim.

You speak as if IPv6 is urgent even though we haven’t actually run out of IPv4 addresses yet. Why is that?

The deployment of IPv6 is no small undertaking. In fact, the effort required in most organizations, especially enterprises will be on par with the Y2K process. However, because unlike Y2K, IPv4 does not come with a certain deadline, it has been much harder to get  management to focus proper attention on the issue.

Just like Y2K, if we had waited until Dec. 30, 1999 to begin our preparations, we would have had a number of serious problems.  Instead, most organizations had multi-stakeholder task forces assigned to Y2K more than 5 years in advance. Depending on how you measure IPv4 runout, we’re somewhere between already there and less than 2 years away, so it’s already too late to start 5 years in advance. The good news is that a lot of the necessary vendor work has already been in progress. The bad news is that there’s still a whole lot of organizational work and vendor work that hasn’t yet begun.

At the very least, it’s quite urgent for each organization to go through an IPv6 gap analysis (an analysis of what portions of the infrastructure are not yet ready for IPv6 to be deployed and what is required to correct those limitations) as soon as possible. Until that process is completed, an organization has no idea even how long it will take to prepare for IPv6 or what it will cost. This close to runout, the lack of that information is a major risk to any organization which depends on the internet for its operations.

Owen, thanks for all the great IPv6 info! We’ll have to stay in touch, especially this year, as we see more IPv6 rollouts and everyone gets more operational experience.

That wraps up the interview with Owen. Stay up to date with IPv6 news and deployment strategies at:


%d bloggers like this: