Archive for November, 2011

Recursion…. Or, the life of a sysadmin

As part of my home IPv6 project, I encountered the dreaded “Technical Debt”, which led to a “fun” recursive process to get one of my home systems upgraded. Hilarity ensued.

The entire saga is laid out in detail below, but to cut to the chase, I had to completely switch gears on my home router selection, do multiple OS upgrades, fix all kinds of configuration files (format changes), and un-make some assumptions about what I needed to do based on old/outdated/broken information.

So, what did I learn?

  1. Don’t let technical debt pile up – the whole OS upgrade thing took longer than it should have because the system was so old. I also had big changes in some config files that would have been easier to take as a set of smaller upgrades.
  2. “The net of a million lies” is sometimes lists as facts things that are at best unconfirmed rumor (Mac’s don’t do auto-config, NEED DHCP6.) or old info.
  3. The Ubuntu upgrade process is incredibly more robust than it used to be. “do-release-upgrade” actually works now!
  4. Be flexible – be prepared to switch plans based on new information.
  5. Maintain your sense of humor.

My IPv6 install journey looks like a recursive program, what with all the false starts, newly-discovered dependencies, etc.

NOTE: WordPress has completely broken nested lists, so you’re getting the PRE-formatted version

start IPv6 home project
    set up IPv6 tunnel with Hurricane Electric
    install IPv6 tunnel via on home Linux system - all is OK, but I want something I don't really have to maintain. I also need a more user friendly solution that we can give to our employees, without them having to learn a lot about networking.
    Do some research, talk to some IPv6 networking people, etc. Discover the reason that my home MacBook Pro won't connect is that my (mostly) current versions of Mac OS X (10.6.8) "won't handle IPv6 auto configuration, you need DHCP6." (More on this later)
    Buy and install new home router to replace 8 year old WRT54G, something that will also provide a DHCP6 server
    select, buy install  D-Link DIR-825
        buy DIR-825 from Amazon
        install DIR-825 - no IPv6?!? WTF?!? Amazon shipped 3 year old "A" hardware rev - no IPv6 support possible!
        read more reviews ("drops wireless 1-10 times per day")
        return DIR-825
    select, buy, install Apple Airport Extreme
        buy from Apple store
        install Airport Extreme
        IPv6 happiness!! set up tunnel to Hurricane Electric
        ping6 and traceroute6 from home Linus server "just work"
        Discover that unlike DIR-825, Airport Extreme doesn't include DHCP6 server
        decide to install DHCP6 on home server (Ubuntu Karmic Koala 9.10)
            Karmic Koala has EOL'ed (when did that happen?)
            Might as well upgrade home server
                Hmmm, no "supported" way to upgrade from 9.10, eg "do-release-upgrade" is "not supported"
                spend hours searching Google, badly written blogs, dozens of people asking the same question"
                decide to go for it
                run full backups (I'm willing to take a risk, I'm not stupid)
                "sudo do-release-upgrade"
                eat Thankgiving dinner
                go to party
                finish upgrade
                Hmmm, everything us fine except SO's email - I broke dovecot?!? Frack!
                fix dovecot
                ready to install DHCP
                    which version?
                    much Googling - leads me to ISC, as I had originally wanted to do
                    install latest DHCP server for Ubuntu - dhcp3-server
                    configure for IPv6
                    IPv6?!?! Where is it?
                    IPv6 not supported until DHCPv4?  version *4*?
                    Oh, DHCP v4 not supported until Ubuntu 11.04 Natty Narwhal - What the hell am I running? Ubuntu 10.04.3 LTS Lucid Lynx???
                    upgrade from 10.04.03 to 11.04 Natty Narwhal
                        first upgrade to 10.10 Maverick Meerkat
                            things for libc upgrade
                            grub-pc install info
                            bind db.root - take new file
                            dovecot.conf - THREE WAY MERGE - GOOD LUCK WITH THAT!
                            release-upgrades file - take new file
                            apache2.conf, available sites, ssl.conf - take new files (CHECK GALLERY FAILURE)
                            DID GALLEREY2 FAILURE KILL THE UPGRADE? - YES IT DID
                        restart upgrade (45 mins later...)
                            postfix/post-install, postfix-files - take new file
                            dovecot.conf - AGAIN? - keep local file (hopefully merged)
                            apache2 fails to start (gallery2)
                            gallery2 upgrade fails, but this time I can submit a bug report
                            and.... it restarted the upgrade AGAIN?
                        ran into the dreaded "blank screen on reboot"
                        Much Googling ensures - need i915.modeset=0 in grub.cfg
                        Hooray!  On 11.04 Natty Narwhal - still need to fix Apache2 * and Gallery2...
                        Wait, what?!? Aha, the restart of the upgrade jumped to 11.04, bypassing the finish of 10.10!
                        Fooled you do-release-upgrade!!  HAHA!
                        gallery2 is still broken
                        dovecot is broken (again)
                        decide I might as well go all the way to 11.10 Oneric Ocelot (and then fix Apache and Gallery2)
                        accidentally find the blog post that shows how to turn on acceptance of IPv6 router advertisements on MacOS
                        sysctl -w net.inet6.ip6.accept_rtadv=1
                Mac is online with IPv6, SSH works, even! I don't need no stinkin' DHCPv6 (but I will, more later)
        DONE - IPv6 (minimally) running at home. Clients can access, etc.

What a rathole! But this is sometimes the nature of system administration. Shifting requirements, incorrect documentation, need to build the tool to build the tool to do the work.


IPv6 – basic connectivity (via tunnels)

Unless you are in a very unusual location, you are going to have to use a tunnel to get IPv6 connectivity. Native IPv6 connections are pretty much impossible to find, especially for the home user.

While Comcast has been very active in the IPv6 community, they only started their first pilot IPv6 deployments just a few days ago. IPv6 information from Time Warner and Cox is also extremely limited. In any case, there are no IPv6 native cable companies in my area (San Diego CA).

As for DSL, I’ve been a very happy SpeakEasy (now MegaPath) customer for almost 9 years. In fact, I had the first DSL install in my CO. Having two senior techs and three juniors show up the install was pretty cool 🙂 And only two outages in those 9 years.

But, no IPv6.

That leaves IPv6 tunnels. The best way to get IPv6 connectivity pretty much anywhere, is via an IPv6 tunnel broker. For me the choice was pretty easy. Back in March, I met Owen from Hurricane Electric in the ARIN booth at Game Developer Conference 2011. We had some great conversations about IPv6 and its potential impact on the online game industry. Based on those conversations, it was clear that they really understand IPv6. After some later email and reviews on discussion forums, I decided to get the tunnel from Hurricane.

Getting the tunnel couldn’t have been easier. All I had to do was create an account at their tunnel broker site. I had my first tunnel allocated to me in about 30 minutes. I had a routed /64 network and all the information I needed to create my end of the tunnel. Since they handle the routing, I don’t have to worry about BGP or announcing routes.

Small problem. No IPv6 router at the house…

Leave a comment

home IPv6 – requirements

I’ve made a few posts about my minor home IPv6 successes, and they’ve generated a some questions.

Before I get into details about the setup itself, I want to mention some of the requirements. My requirements are probably not typical for someone who has been a sysadmin as long as I have, and that probably requires some explanation.

In addition to being my family’s home network, this is also a possible “reference implementation” for future “work from home” users at my company. We may not need to support home users on IPv6 in the near future, but it will come. They will be connecting to us via VPNs just as they do today, but with the IPv6 twist. The “reference implementation” requirement has all kinds of implications for what hardware and software I can use.

Here are my requirements:

  1. Stable, production quality. Not experimental. I’ll be using this as a client network to test our upcoming IPv6 deployment at work (which will include VPNs, public facing services, etc) , and I want to spend any needed time debugging the work environment (which may be very complex), not my home network, which should be simple. This is my family’s network, and it has uptime requirements 🙂
  2. Stable, readily available components for hardware and software. No bleeding-edge Linux versions, no Windows registry hacks, no custom-built or non-vendor supported router software. I have used and applaud the DDWRT and similar efforts, but I need something more mainstream.
  3. Consumer-grade networking gear. I have access to commercial kit from Cisco, Juniper, etc from work, but that’s not our target for mainstream home deployment. If you can’t buy it at Fry’s or Amazon (or maybe NewEgg) for around $100 (maybe up to $200?), then it’s not the right product.
  4. Suitable for deployment by a relatively savvy home user, but doesn’t require a system or network admin. This is a sample deployment that we may want to have our employees replicate; while there are sysadmins, programmers and the like, there are also artists and business/financial/HR people who shouldn’t need to load custom router firmware, or significantly change their home PC (or Mac) to successfully connect to us.
  5. No support required from the IT department for the home user, beyond a setup guide or two. No (or extremely little) ongoing maintenance required by the home user. More importantly, little or no support for the home network from me, since my family uses the home net for school, business and personal things.
  6. Target clients are relatively modern: Windows 7, Mac OS 10.6 (Leopard) or newer, Ubuntu 11, Centos/RedHat 6 (maybe 5?). No legacy support.
  7. Dual stack: no 6to4 or 4to6, no tunnels. Deploying this must NOT break existing networks, including home printers, consumer NAS, etc.

I think I’ve met those requirements with my home deployment, for the most part. I’ve found a few places where IPv6 support in mainstream OSes is still a little lacking, and I’ll write about those in the future…

, , ,

1 Comment

I have seen the dancing kame!

This evening I finished up the initial phase of the home IPv6 project. At this point, any client  on my home network that fully supports IPv6 can connect to any IPv6 resources on the public Internet, such as and

(More on that “fully supports” thing, later.)

This should have taken just a few hours, but I ran into a lot of incorrect documentation and obsolete software that led me down a few ratholes.  More on that later, too 🙂

For, now, it’s (dancing, IPv6) turtles, all the way down!


Leave a comment

%d bloggers like this: