Recursion…. Or, the life of a sysadmin

As part of my home IPv6 project, I encountered the dreaded “Technical Debt”, which led to a “fun” recursive process to get one of my home systems upgraded. Hilarity ensued.

The entire saga is laid out in detail below, but to cut to the chase, I had to completely switch gears on my home router selection, do multiple OS upgrades, fix all kinds of configuration files (format changes), and un-make some assumptions about what I needed to do based on old/outdated/broken information.

So, what did I learn?

  1. Don’t let technical debt pile up – the whole OS upgrade thing took longer than it should have because the system was so old. I also had big changes in some config files that would have been easier to take as a set of smaller upgrades.
  2. “The net of a million lies” is sometimes lists as facts things that are at best unconfirmed rumor (Mac’s don’t do auto-config, NEED DHCP6.) or old info.
  3. The Ubuntu upgrade process is incredibly more robust than it used to be. “do-release-upgrade” actually works now!
  4. Be flexible – be prepared to switch plans based on new information.
  5. Maintain your sense of humor.

My IPv6 install journey looks like a recursive program, what with all the false starts, newly-discovered dependencies, etc.

NOTE: WordPress has completely broken nested lists, so you’re getting the PRE-formatted version

start IPv6 home project
    set up IPv6 tunnel with Hurricane Electric
    install IPv6 tunnel via on home Linux system - all is OK, but I want something I don't really have to maintain. I also need a more user friendly solution that we can give to our employees, without them having to learn a lot about networking.
    Do some research, talk to some IPv6 networking people, etc. Discover the reason that my home MacBook Pro won't connect is that my (mostly) current versions of Mac OS X (10.6.8) "won't handle IPv6 auto configuration, you need DHCP6." (More on this later)
    Buy and install new home router to replace 8 year old WRT54G, something that will also provide a DHCP6 server
    select, buy install  D-Link DIR-825
        buy DIR-825 from Amazon
        install DIR-825 - no IPv6?!? WTF?!? Amazon shipped 3 year old "A" hardware rev - no IPv6 support possible!
        read more reviews ("drops wireless 1-10 times per day")
        return DIR-825
    select, buy, install Apple Airport Extreme
        buy from Apple store
        install Airport Extreme
        IPv6 happiness!! set up tunnel to Hurricane Electric
        ping6 and traceroute6 from home Linus server "just work"
        Discover that unlike DIR-825, Airport Extreme doesn't include DHCP6 server
        decide to install DHCP6 on home server (Ubuntu Karmic Koala 9.10)
            Karmic Koala has EOL'ed (when did that happen?)
            Might as well upgrade home server
                Hmmm, no "supported" way to upgrade from 9.10, eg "do-release-upgrade" is "not supported"
                spend hours searching Google, badly written blogs, dozens of people asking the same question"
                decide to go for it
                run full backups (I'm willing to take a risk, I'm not stupid)
                "sudo do-release-upgrade"
                eat Thankgiving dinner
                go to party
                finish upgrade
                Hmmm, everything us fine except SO's email - I broke dovecot?!? Frack!
                fix dovecot
                ready to install DHCP
                    which version?
                    much Googling - leads me to ISC, as I had originally wanted to do
                    install latest DHCP server for Ubuntu - dhcp3-server
                    configure for IPv6
                    IPv6?!?! Where is it?
                    IPv6 not supported until DHCPv4?  version *4*?
                    Oh, DHCP v4 not supported until Ubuntu 11.04 Natty Narwhal - What the hell am I running? Ubuntu 10.04.3 LTS Lucid Lynx???
                    upgrade from 10.04.03 to 11.04 Natty Narwhal
                        first upgrade to 10.10 Maverick Meerkat
                            things for libc upgrade
                            grub-pc install info
                            bind db.root - take new file
                            dovecot.conf - THREE WAY MERGE - GOOD LUCK WITH THAT!
                            release-upgrades file - take new file
                            apache2.conf, available sites, ssl.conf - take new files (CHECK GALLERY FAILURE)
                            DID GALLEREY2 FAILURE KILL THE UPGRADE? - YES IT DID
                        restart upgrade (45 mins later...)
                            postfix/post-install, postfix-files - take new file
                            dovecot.conf - AGAIN? - keep local file (hopefully merged)
                            apache2 fails to start (gallery2)
                            gallery2 upgrade fails, but this time I can submit a bug report
                            and.... it restarted the upgrade AGAIN?
                        ran into the dreaded "blank screen on reboot"
                        Much Googling ensures - need i915.modeset=0 in grub.cfg
                        Hooray!  On 11.04 Natty Narwhal - still need to fix Apache2 * and Gallery2...
                        Wait, what?!? Aha, the restart of the upgrade jumped to 11.04, bypassing the finish of 10.10!
                        Fooled you do-release-upgrade!!  HAHA!
                        gallery2 is still broken
                        dovecot is broken (again)
                        decide I might as well go all the way to 11.10 Oneric Ocelot (and then fix Apache and Gallery2)
                        accidentally find the blog post that shows how to turn on acceptance of IPv6 router advertisements on MacOS
                        sysctl -w net.inet6.ip6.accept_rtadv=1
                Mac is online with IPv6, SSH works, even! I don't need no stinkin' DHCPv6 (but I will, more later)
        DONE - IPv6 (minimally) running at home. Clients can access, etc.

What a rathole! But this is sometimes the nature of system administration. Shifting requirements, incorrect documentation, need to build the tool to build the tool to do the work.

  1. #1 by Jeff_A on December 5, 2011 - 2:36 pm

    I’ve never heard of the term “Technical Debt”.

    Were I an accountant, I would think that Technical Debt is the opportunity cost of relocating to a place with lower prices on land, electricity, labor etc. but with correspondingly less in the way of developed infrastructure (limited network connectivity, poor power availability, scarce transporation resources….)

    “After factoring in the Technical Debt that we’d incur by moving manufacturing to Kerguelen Island, we discovered we’d be better off staying in Newark! The $0.25 per hour wages aren’t all that attractive when you add in the mandatory Yeti insurance…”

    However, I *have* heard of “Yak-Shaving”:

    It misses the point (slightly) from the Ren & Stimpy sketch, but I think the central point remains. Getting ISC dhcpd to return the right boot options to some clients doesn’t yet require decorating the house with Huggies or filling one’s galoshes with coleslaw, but it’s only a matter of time…

  1. IPv6 – MacOS Snow Leopard update « Thuktun (Message)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: