Why I killed our IPv6 project…

Seven years ago we started an “IPv6 project”. The goal was to deploy IPv6 throughout our internal game studio network. After two months of analysis, I approached my boss and recommended we kill it. At least as an “IPv6 project”. It was reborn as a “clean up our network architecture, and oh by the way, add IPv6 (and a bunch of other things)”.

IPv6 at AWS – Route53

Hooray! AWS users can now serve their DNS info over IPv6. You could serve AAAA records before, but only over IPv4.

This finally gives AWS customers a way to deal with IPv6-only customers (as are appearing in Asia), who might have otherwise had to depend on ISP proxies or CGN (Carrier Grade NAT), host their DNS elsewhere, or be unable to reach services hosted in AWS.

IPv6 – now from COX (San Diego)!

As you recall, I’ve been lamenting the lack of direct IPv6 via my local ISP (COX) since 2013.

It seems that some time in the past 3 months, they silently enabled IPv6 in my area! I was preparing to reconfigure my tunnel from tunnelbroker and decided to “just check”. Cox is now correctly(!) serving IPv6.

I had to turn off my Hurricane Electric tunnel a few months ago, as Netflix began blocking as many tunnel services as they could, over geo-location “issues”.

I was able to set my Apple Airport Extreme to “auto configure” for IPv6, and I’ve got proper addresses, routers and even DNS over IPv6.

Thanks Cox!

 

 

Register4Less now supports IPv6 DNS!

I got a great followup from my domain registrar Register4less today. A few weeks ago, I had asked about when their DNS would fully support IPv6.

They’ve allowed AAAA records in their hosted DNS for years, but they only accepted queries over IPv4 until this week.

This is just another reason that I love R4l’s support. When I had asked them about IPv6 DNS before, they said it was coming “soon”, but couldn’t give a for-sure date, but would let me know.

When they turned up IPV6 DNS this week, they proactively sent me an email letting me know that the service was available, answered a few questions (within literally 5 minutes!).

Register4less.com is the official DNS provider of UserFriendly.org. If you work IT, you should know this long-running webcomic.

 

WordPress.com – still no IPv6

Three years ago I wondered when wordpress.com would begin to support IPv6. The irony of hosting a blog that talks a lot about IPv6 on an IPv4-only platform is not lost on me 😦

Here we are in 2016 with Google, Akamai and pretty much all content providers are reporting more IPv6 traffic, and WordPress.com is still stuck in 1983.

This is not an OS issue, a load balancer issue, a PHP language issue, or even a WordPress software issue. I’ve seen WP software running full dual-stack, no problems. Five years ago. I’ve done it myself, I just don’t want to run WP myself anymore.

For whatever reason, WordPress.com has not decided to commit to the future. yet.

Stop disabling IPv6 in your system images

Seriously.  Just stop that.

Stop disabling IPv6 as part of your standard OS install and network configurations.

If you’re like a lot of IT shops, you’ve probably been building “golden images” of your operating systems to use as the template for OS installation. While these images are (hopefully) on a regular patch cycle after installation, the basic configurations and options can remain unchanged for years.

The upshot of this is that there are a lot of operating system images out there that were initially created around the time that the base OS was released, and which have had minimal changes since then, other than mandatory patches.

Windows 7 and Server 2008R2 were released in 2009. Centos 5 was released in 2007. Both are still in very wide use. Even if you’ve moved up to Windows Server 2012 or Centos 6 (both released in 2011), it is not uncommon for golden images of these to retain the network and other configurations such as IPv6 from prior versions.

In other words, it is quite likely that your brand new OS install is using assumptions and configurations from 2009 or even 2007, when it was still considered good practice to disable IPv6 at every opportunity. We’re beginning to see new OS features, such as DirectAccess, that require functioning IPv6, either native or tunneled.

I have yet to find any service that’s available in the MacOS X, Centos or Ubuntu systems that can’t make use of IPv6, or is negatively impacted in any way by dual-stacking the host. I have also not found any instance where taking a dual-stack-capable host onto an IPv4-only network has caused an issue, in at least 2 years.

Here’s some more info for you Windows folks, including a list of MS services that do, or don’t use IPv6.

So just quit disabling IPv6 by default, mmmkay?

IPv6 – dealing with unwanted SLAAC addresses on servers

Are your servers getting SLAAC addresses in addition to the addresses you are manually configuring? If so, read on…

You need to find and turn off the “A” bit in the Prefix Length option of your Router Advertisement packets. The “A” bit is on by default on most network routers, and the documentation that describes the interactions between the “M”, “O” and “A” bits is scattered across at least a half dozen RFCs.

When we first set up our IPv6 lab, we went through several phases. Initially we just did client subnets and hosts and let all the stations auto-configure (SLAAC). This all happened “magically” with the default behavior of all the operating systems and network gear we tested.

Then we split the clients and servers onto separate subnets. When we did the split we added a DHCPv6 server and turned ON the M and O bits for the client subnets. For the server subnets, we turned OFF the M and O bits and statically configured the IPv6 (and IPv4) addresses.

The client hosts did everything exactly as expected, gathering IPv6 addresses and other options, exactly as they would have using DHCP and IPv4.

But, we never could quite get the servers to stop creating and configuring SLACC addresses, even with M & O bits turned ON or OFF on their subnets. Making sure that we did NOT have DHCPv6 clients configured on these servers, we tested all  four states with nearly identical results.

In other words, each server would always end up with three IPv6 addresses:

1. a globally unique (global scoped) static assigned address, the one we configured at boot time
2. a globally unique (global scoped) SLAAC address, usually based on its MAC address
3. the usual and expected link-local address (fe80::)

So, what else was going on? Most of the documentation we found (especially RFCs) described these two bits in excruciating and often contradictory fashion! Take a look at RFC 4861 for the format of the Router Advertisements, and you’ll see the M and O bits right there in section 4.2). If there are other option bits that might control this, shouldn’t they be shown here?

By the way, the M and O bits are always OFF by default on all the networking gear we’ve seen so far (Cisco, Juniper and HP).

4.2. Router Advertisement Message Format

   Routers send out Router Advertisement messages periodically, or in
   response to Router Solicitations.

      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |     Type      |     Code      |          Checksum             |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     | Cur Hop Limit |M|O|  Reserved |       Router Lifetime         |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                         Reachable Time                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                          Retrans Timer                        |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |   Options ...
     +-+-+-+-+-+-+-+-+-+-+-+-

But in all four combinations of the M and O bits, and IF you aren’t running a DHCPv6 client, you get a SLAAC address in addition to the address you statically (manually) configure.  How do you turn off “auto conf” if it isn’t controlled by flags in the Router Advertisement???

It turns out that there are actually three bits in the RA that control host configuration, not two, and so there are 8 possible cases of M, O and “A”, not four. So where is this mysterious “A” bit hiding?

The “A” bit is “hidden” in a Router Advertisement option (“Prefix Information”), which is described in section 4.6.2, about 10 pages farther along in the RFC. This option’s purpose is to tell you about the length of the valid address prefix that’s available on the current subnet, but it also has “A”  that controls whether or not a station on that subnet should do SLAAC. And unlike M and O, A seems to always be set ON by default.

4.6.2. Prefix Information

       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |     Type      |    Length     | Prefix Length |L|A| Reserved1 |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                         Valid Lifetime                        |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                       Preferred Lifetime                      |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                           Reserved2                           |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      +                                                               +
      |                                                               |
      +                            Prefix                             +
      |                                                               |
      +                                                               +
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

   Fields:

      Type           3

      Length         4

      Prefix Length  8-bit unsigned integer.  The number of leading bits
                     in the Prefix that are valid.  The value ranges
                     from 0 to 128.  The prefix length field provides
                     necessary information for on-link determination
                     (when combined with the L flag in the prefix
                     information option).  It also assists with address
                     autoconfiguration as specified in [ADDRCONF], for
                     which there may be more restrictions on the prefix
                     length.

      L              1-bit on-link flag.  When set, indicates that this
                     prefix can be used for on-link determination.  When
                     not set the advertisement makes no statement about
                     on-link or off-link properties of the prefix.  In
                     other words, if the L flag is not set a host MUST
                     NOT conclude that an address derived from the
                     prefix is off-link.  That is, it MUST NOT update a
                     previous indication that the address is on-link.

      A              1-bit autonomous address-configuration flag.  When
                     set indicates that this prefix can be used for
                     stateless address configuration as specified in
                     [ADDRCONF].

So, that’s where the mysterious server SLAAC addresses come from. They are caused by the default-on “A” bit that is in the Prefix Information option to the Router Advertisement.  Clear this A bit on your server subnets, and you’ll get only the IPv6 addresses that you configure, and no more SLAAC addresses as an extra bonus.

After I figured out what was going on, I also found these web pages which each shed some light on the situation:

• http://blog.ipspace.net/2012/11/ipv6-router-advertisements-deep-dive.html
• http://blogs.cisco.com/enterprise/ipv6-automatic-addressing/
• http://blogs.technet.com/b/teamdhcp/archive/2009/03/03/dhcpv6-understanding-of-address-configuration-in-automatic-mode-and-installation-of-dhcpv6-server.aspx

IPv6 transition and the pit of doom

In case anyone hasn’t noticed, I’m not a big fan of the IPv6 transition technologies, such as Teredo, 6to4, DNS64, etc. I also don’t like “big bang”, everything today, or “flag day” cutovers either. Fortunately, for IPv6 there’s a good middle ground, dual-stack.

Consumer-facing may be different, but here’s the strategy that I like for transitioning an internal, corporate or .EDU network to IPv6. You don’t have to build anything that you won’t be using for the future, so no spending on technological cul-de-sacs or band-aids. You won’t have to install, maintain, and then eventually turn off any transition mechanisms.  Just a steady, always forward, straightforward path that will eventually lead you to 100% IPv6, and eventually allow you to turn off IPv4, if you ever so desire. You won’t have to, you just may not have many people to talk to on IPv4 in a few years.

You can begin this transition today, given time and a steady plan to refresh your technology over the course of a few years. If you have an urgent need for IPv6, you can go faster. If you don’t, or are time or money constrained, you can go slower, taking years to make the switch.

1. boot to the head to anyone who insists on any transition technology for our internal networks, without a very convincing argument
2. dual stack the (internal, client) networks + provide “outbound” external dual-stack paths to the public Internet
3. dual-stack DNS and DHCP and any thing else I need to manage and operate the clients, such as Active Directory
4. dual stack the clients – if an OS doesn’t do IPv6, chuck it or confine it to the legacy pit of doom, an IPv4-only subnet (or two)
5. dual stack the servers – do this on your regular refresh/upgrade cycle or as-needed, or if it can’t be upgraded, it goes into the pit
6. dual stack any remaining services – your software vendors and internal developers have had enough time to sort out dual-stack network calls, or they go into the pit
7. At this point your network, clients, servers and services are all dual-stacked. The users can reach everything on the public Internet, old or new, as well as all your internal services on both IPv4 and IPv6. And guess what? By this time, you’ll only have two reasons to keep IPv4 around:
   • Your users might still need to reach old crufty web sites and services out there in the real world
   • You might still need to talk to some things in the pit of doom
8. Finally, after years, you can look into turning off IPv4. Start by pouring petrol into the pit of doom and lighting it on fire.
9. Then start turning off IPv4, first on the the services and servers, then the clients, and eventually the network. Take your time, you’re in no hurry.  You can take years, if you like.

We’re doing 1,2 and 3 this year and next. We’ve started on 4 and that will continue for the next year or so. We can start on 5 as soon as we have time. New servers will be born dual-stack starting early next year. A year or three from now we clean up 6, and we’ll be at stage 7.

Then, and only then, do we think about Step 8, turning off IPv4. If that begins by 2017, I’ll be surprised. If Step 9 doesn’t start until 2020, I don’t care.

As long as I’ve got 1 – 4 done, I have a lot less pressure and I can proceed on a rational, non-panic’ed basis to replace servers and services as part of a regular refresh cycle.  For many companies, this is 3-4 (or even 5) years.

The consumer-facing case has completely different drivers and requirements, it gets a completely different plan and schedule. But I bet it won’t have any transition technologies in it, if I can help it.

This is rather cold-blooded and (I freely admit) a bit of a pipe dream. But if I can make this work, I’ll never have to justify, pay for, create, debug, support, and then decommission any of the transition technologies. I’ll always have a fall back if there’s a problem with a particular IPv6 implementation or if we run into a show stopper on vendor support for IPv6.

I hate building things that I know I’m only building as a temporary bridge. Spending time (and money) putting in the transition strategies can be better spent just moving forward, not sideways.

IPv6 just isn’t has hard as some might lead you to believe. There are still some implementation glitches here and there, but every day there’s better vendor support and fewer bugs.

Don’t be afraid to move forward, it’s easier than you think.

IPv6 transition – just get on with it!

A recent DEFCON presentation highlights the need to accelerate adoption of IPv6 on your network. You can either turn off IPv6 on all your hosts (which will break things), or get on with it and deploy IPv6 “for real”.

(TL;DR: your hosts already have dual-stack activated, not having IPv6 supported in your network opens up a  man-in-the-middle (MITM) attack. Though long known, there is now a “one click” exploit available.)

There’s been a lot of discussion on various IPv6-related mailing lists with how to drive the transition, how to transition, and which (if any) of the transition technologies should be used.

In general, NONE of the transition technologies (other than dual-stack) address this particular MITM attack. They (for the most part) leave old IPv4 nodes as-is on your network, and try to translate protocols and hide IPv6 from those old nodes (and vice versa).

Personally, I find it quite heartening that many are making good business cases for aggressive adoption of native IPv6. Some are also providing good historical evidence that we’ve made similar transitions in the past, without extensive transition technologies, with good success:

On 8/8/13 1:40 PM, Ray Hunter wrote (v6ops):
Actually I think your reasoning and reference to the IPX and Appletalk
phase out would suggest it’s easier to make a bold call: move to IPv6
ASAP for critical systems via dual stack, and for the rest you draw a
box around it and call it legacy and run it on IPv4 until it dies a natural death.

IMHO Going half way with NAT64/DNS64 just prolongs the pain and locks
you into a transition technology that is expensive and difficult to
operate for the life cycle of that box, and which has to remain in place
until the last app is migrated or switched off.

I’ve been in a fair number projects where you sometimes just have to
dare to cut the cord whilst maintaining a process to find out what has
broken. So one valid IPv6 only migration strategy might be: “If it’s
important, they’ll migrate before a flag day date. Otherwise they get
cut off.”

I cannot agree enough with the “prolongs the pain”  and “locks you into a transition technology “observations.

At work, we’re going on the assumption that we’ll be able to go dual-stack and not need any translation. So far, that looks viable for our internal networks. When we get to the consumer-facing stuff, well, we’ll see.

IPv6 adoption – still doubling every year

Good News, everyone!

IPv6 adoption continues to double, year on year.  Of course, that’s only three years of baseline, but things are certainly moving in the right direction.

As this article points out, if this rate continues, more than half of Internet users could have IPv6 within 6 years. This goes along with estimates of IPv6-only customers reaching 20% by 2017.

It remains to be seen if this adoption rate can continue.  However, events such as Switzerland moving from 3% to 10% adoption in a single month are interesting. They show that a single large ISP can quickly make a huge difference in adoption rates, as they turn up large portions of IPv6 connectivity in large deployment events. I expect Comcast to quickly begin to have a similar impact on US IPv6 availability later this year.

This CircleID article calls out some winners and losers in the IPv6 adoption race. My favorite IPv6 (tunnel) provider, Hurricane Electric gets high marks.