Archive for category SysAdmin Interviews
This is an email interview with Owen DeLong , the “IPv6 Evangelist” for Hurricane Electric, one of the largest ISPs in the world. They are also operators of tunnelbroker.net, one of the largest IPv6 tunnel providers in the world.
Owen has been in the industry in many roles, including work at Sun, Exodus Communications and TellMe. In addition to his “day job”, he’s also on the advisory council for ARIN, teaches SCUBA, and makes lots of contributions to network-related mailing lists. He’s also one of the very few people I know who has an AS number for his home net!
We’ve been wanting to do this interview for quite a while, but things have been rather busy the past year. When I ran into Owen (again) at LISA in San Diego a few weeks back, we talked about IPv6 and this interview. Owen was kind enough to work on this with me over the holiday.
Owen, you’ve been an “IPv6 Evanagelist” for Hurricane Electric for a while now. How did you become the IPv6 Evangelist? What does that entail, and what did you do before IPv6 needed an evangelist?
I’ve been various forms of Systems Administrator, Network Engineer, Backbone Engineer, Senior Backbone Engineer, Network Architect and even Operations Manager over the years. When my last startup was on the verge of imploding, I mentioned to someone at HE who does similar work that I was interested in doing something similar. We talked extensively in Manila (APNIC conference) and the rest as they say, is history.
Can you tell me a little about Hurricane Electric?
We are actually one of the largest backbone network providers in the world. We are the number 1 most peered IPv6 network and the number 3 most peered IPv4 network according to CAIDA. We operate multiple 10-Gig backbones throughout North America, Europe, and Asia.
I try to attend (and speak at) a wide variety of events and conferences. I usually speak at at least one ION event, I’ve also spoken at Astricon, Pubcon, Future of Web Applications, various NOGs (Network Operators Groups) such as NANOG. I’ve spoken to a variety of Linux, Unix, an Open Source groups such as BALUG, NLUUG, etc. I’ve spoken to Cisco and Juniper user’s groups and more. I also speak at many of the IPv6 oriented conferences such as the North American IPv6 summit, the Texas IPv6 summit, etc.
Most of the audiences I speak to are pretty receptive. Certainly there is more and more interest and less and less skepticism over the last 3 years.
IPv6 has been a standard now for almost 18 years (if you use RFC 1883 as the reference). Why has adoption been so slow?
Because we made the mistake of introducing NAT and people are busy. A network not perceived as broken tends to limp along with whatever level of brokenness it has come to accept until perturbed by an additional factor such as CGN, address exhaustion, or other external pressures like the continued ability to reach destinations in Asia.
So far, IPv4 runout hasn’t really exerted those pressures to any great extent on anyone yet, but at least people are starting to see that it is coming and IPv6 adoption is starting to accelerate quite a bit now.
According to Google, IPv6 was around 0.25% of internet traffic at the beginning of 2010 and 2011. The biggest difference between the two being that at the beginning of 2010, it was about 60% 6to4/Teredo and at the beginning of 2011 it was about 80% native, but the overall growth was pretty flat. By the end of 2011, it was 0.4% and almost entirely native with 6to4 and Teredo almost unmeasurable. As we approach the end of 2012, we’re seeing more than 1.1% with 6to4 and Teredo almost eliminated.
At first blush, those seem like pretty low numbers. However, 1% of internet traffic today is a lot more than 100% of internet traffic when IPv4 was 18 years old. Also, consider in 2011, that’s a 100% growth rate and in 2012, it’s approximately 175% growth, so not only is adoption growing, but the rate of adoption is accelerating quite rapidly.
If the curve continues to ramp towards vertical, we might see more than 5% by the end of next year and more than 20% by the end of 2014. That’s without factoring in the additional pressures from actually running out of IPv4 addresses at the service providers and the fact that ARIN will be basically out of address space most likely in the middle of next year.
On the technical side, what’s the biggest impediment to implementation? Is that the same for ISPs, Enterprises and end users, or do they all face different challenges?
No, they each face unique challenges.
For the end user, it’s the lack of IPv6 support in consumer products. There’s a host of products that lack IPv6 support today which is severely impacting consumers. These include familiar household names like Playstation,, Yamaha, Samsung, LG, and all of the Matsushita brands (JVC, Technics, Pioneer) and more. In fact, I don’t know of a single CE manufacturer that has embraced IPv6 in their products yet. IMHO, this is inexcusable.
None of these vendors have yet seen fit to deploy IPv6 at the consumer level in their products and this is becoming a major impediment.
To look at ISPs, we really need to divide them into two principal categories… The B2B and/or Backbone-oriented ISP, such as Hurricane Electric, where there really are very few remaining technical impediments and the few remaining impediments are primarily related to educating executive management.
The other category is what I will call the last-mile ISP. The providers that serve SOHO, Residential, and Small Business customers and perhaps some medium sized businesses as well. These ISPs are still facing shortcomings in CPE and in their provisioning, management, and deployment systems. Many DSLAMs and BRAS units have limited or no IPv6 capability. Several of the CMTS vendors still have show-stopping bugs to work out (This means you, brand C and others).
For the enterprise, there are a few technical impediments, but these can mostly be addressed by current technologies. The larger hurdles in business really are the education of executive management and communicating to the enterprise that this isn’t just a networking issue.
How about on the Business side?
I think this is easier to answer. Across the board for all of those organizations, the number one business challenge is educating executive management to the point that they understand this to be a critical organization-wide issue that must be addressed starting now (if they haven’t started already).
Any organization considering IPv6 seriously needs to understand that this is an organization-wide issue and that it touches EVERY group within the organization. Systems administrators need to be involved because this will touch every application, every system, every server, and will impact how they do their jobs on a daily basis. The help desk needs to get involved because this will represent some pretty major changes in their troubleshooting scripts and will also require them to understand more about how things operate in a dual-stack environment. Application developers, maintainers, administrators, etc. are involved because any application that communicates via the network needs to be examined and tested against a dual-stack and eventually an IPv6 only environment. The mail room has to expect that there’s going to be a lot of logistical support required for getting all of these equipment updates to the right locations at the right times. Management at all levels needs to recognize the need to make this an organization-wide priority and address it in an orderly fashion before it becomes an outright emergency.
Lots of networks seem to be taking a dual-stack approach and adding IPv6 to their existing IPv4 network. What do you think of this approach?
I think it’s the only feasible approach for the time being. We’ll reach a point where we have to turn off IPv4 and move forward to IPv6 single-stack and that’s a good thing. It should always be the end goal. However, just as we didn’t turn off Novell the day we turned on IP in that great migration of the enterprise, we’re not going to be able to turn off IPv4 and turn on IPv6 overnight. We’re going to have to live with both for some time to come.
For people that want to learn more about IPv6, or even implement it on their home networks, where should they start?
I highly recommend our on-line training available at http://tunnelbroker.net. It’s pretty easy to get through, but it does require you to not only learn about IPv6, but to demonstrate that you can actually make things work with IPv6 in order to obtain the higher certification levels.
You’ve mentioned in the past that there are new IPv6-only customers coming online in areas where the IPv4 space is already depleted. How many people are we talking about and where are they? How many potential customers could a company “miss” by delaying an IPv6 roll out?
Well, there are currently a little more than 2.5 billion people on the internet. There are more than 6.8 billion people on earth. So, over time, I expect we’ll see most of the remaining 4.3+ billion people (and growing) connecting on IPv6 and unable to get IPv4 addresses. Immediately, it’s relatively small numbers, probably a few thousand her and there. However, that number can only continue to grow as time progresses.
One of the biggest complaints we all hear is that there’s no “killer app” and “no ROI” for implementing IPv6. What risks (if any) are companies taking by delaying IPv6 adoption?
If it isn’t clear from what I’ve said above, I’ll put it this way. The IPv4 internet cannot keep growing much larger than it currently is. The IPv6 internet will, on the other hand, continue to grow and for a while, that growth will continue to accelerate. As a result, it won’t be very many years before the fraction of the internet that is reachable via IPv4 is much smaller than that portion which is reachable via IPv6. If your business depends on being connected to the whole internet, then, that’s the killer app. for IPv6. If you’re willing to live with only being connected to the part of the internet that exists today, then you may have a few years before you have to worry about that shrinking, but, I suspect it’s not as many years as a lot of people will claim.
You speak as if IPv6 is urgent even though we haven’t actually run out of IPv4 addresses yet. Why is that?
The deployment of IPv6 is no small undertaking. In fact, the effort required in most organizations, especially enterprises will be on par with the Y2K process. However, because unlike Y2K, IPv4 does not come with a certain deadline, it has been much harder to get management to focus proper attention on the issue.
Just like Y2K, if we had waited until Dec. 30, 1999 to begin our preparations, we would have had a number of serious problems. Instead, most organizations had multi-stakeholder task forces assigned to Y2K more than 5 years in advance. Depending on how you measure IPv4 runout, we’re somewhere between already there and less than 2 years away, so it’s already too late to start 5 years in advance. The good news is that a lot of the necessary vendor work has already been in progress. The bad news is that there’s still a whole lot of organizational work and vendor work that hasn’t yet begun.
At the very least, it’s quite urgent for each organization to go through an IPv6 gap analysis (an analysis of what portions of the infrastructure are not yet ready for IPv6 to be deployed and what is required to correct those limitations) as soon as possible. Until that process is completed, an organization has no idea even how long it will take to prepare for IPv6 or what it will cost. This close to runout, the lack of that information is a major risk to any organization which depends on the internet for its operations.
Owen, thanks for all the great IPv6 info! We’ll have to stay in touch, especially this year, as we see more IPv6 rollouts and everyone gets more operational experience.
That wraps up the interview with Owen. Stay up to date with IPv6 news and deployment strategies at:
About a month ago, Server Fault partnered with LOPSA to give 40 Server Fault members free LOPSA memberships based on who had provided the best technical information during the month (as measured by Server Fault reputation).
Server Fault and LOPSA have a lot in common. Both are communities of system administrators, and both are committed to advancing the state of the art in IT. Both are committed to system administration as a whole, not just “Linux admins”, “Windows admins”, “network admins”, etc.
I’ve only been a Server Fault member for a little while, but I have already gotten great value from the community there. I’ve learned some technical things (my Windows-fu really sucks), and most importantly, I’ve learned more about what I would call “new school” system administration and new ways to work with users and their community.
Kyle Brandt is one of the administrators who works behind the scenes to keep Server Fault up and running smoothly, and he also writes about his experiences at the Server Fault Blog.
Server Fault will be having a one day conference for system administrators and operations people this October called Scalability. Check out http://scalability.serverfault.com/ for details!
Kyle was kind enough to take some time from his busy schedule to answer some questions about what it is like to manage such a large and busy system, that serves a community that can be rather demanding at times.
This interview was prompted by QRZ.com‘s recent move into “the cloud”. QRZ means “Who is calling me?” or “You are being called by ___”, which is very appropriate for what is widely considered to be the largest online community for amateur (“ham”) radio in the world. Moving this resource from traditional hosting into the cloud is an interesting comment on the readiness of the cloud to actually deliver for a community that has come to depend on this resource.
The computer and ham communities have a long history together. The original “hacker” community originally had quite a few ties to ham radio and computers, as all were involved with experimenting, especially with electronics. In fact, one possible origin for the term “hacker” its use by the amateur radio community from the 1950s to mean “creative tinkering to improve performance”. This continuing curiosity and desire to build and improve is a hallmark of these communities.
I’ve encountered a few system and network administrators who are hams, and vice versa. QRZ’s founder and publisher, Fred Lloyd, is no exception. Fred spent much of his career on the cutting edge of Internet adoption, working for Sun and other companies in Silicon Valley and other locations. As it turns out, he’s been a ham radio operator about as long.
Fred was kind enough to do an email interview with me earlier this week to discuss system administration, QRZ, ham radio, the Internet and his experiences in moving to the cloud.
A few weeks ago the “anti-social” bookmarking site Pinboard (http://pinboard.in/) made the news in a big way. The site experienced hyper-growth due to the news of the possible demise of Del.icio.us. Concerns about the future of Del.icio.us led tens of thousands of people to look for a new place to store and share their millions of bookmarks.
And quite a few of these people chose Pinboard! During one 30 hour period around December 18th, Pinboard received over 7 million new bookmarks, more than had been put into the system during its entire life.
I was able to catch up with Maciej for an interview via email. I wanted to find out more about how Pinboard was operated, and how this huge spike in load had affected administration of the site. Large-scale system administration isn’t always about hundred of systems, it can also be about tens or hundreds of thousands of users, or unexpected load spikes, or just how you plan for growth.