Archive for December, 2011

IPv6 – enabling the last services – SSH, HTTP, SMTP

In this post, I’ll finish up the “usual services” for my home network. So far I’ve got IPv6 routing and DNS. Now I just want to confirm that I’ve got the rest of my “core services” accessible via IPv6.

(I’ve decided that I don’t need DHCP6 for my particular network, so I’ll be skipping that.)

My remaining core services are: SSH, HTTP and SMTP.

The SSH daemon (sshd) has been configured to listen on both IPv4 and IPv6 by default for years. In fact, it attempts to listen on the IPv6 port, even if you don’t have IPv6 enabled on the host OS. In my case specifically (OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011), I was able to “ssh ::1” as soon as I had eth0 set up with an IPv6 address.

As for HTTP, Apache 2 (Apache/2.2.20 (Ubuntu)), there were no config changes needed. Apache2 will listen on all the addresses (IPv4 and IPv6) that are configured when the daemon starts. All that was needed was a “server apache2 restart” once the IPv6 address was configured, and the web server began answering IPv6 requests.

SMTP turns out to be a little harder. Postfix doesn’t listen on IPv6 ports by default.  You need a few config file changes in

# listen on IPv4 and IPv6
inet_protocols = all
# add IPv6 networks to mynetworks
mynetworks = [::1]/128 [fe80::]/10 [2001:470:67:84::]/64

Then make sure you have an MX record that leads to a AAAA record, do a quick “server postfix reload”, and you’re good to go.

This wraps us the series on my home IPv6 network. There will continue to be IPv6-related posts, and I’ll be writing about our work IPv6 network beginning in mid January.

Thanks for stopping by, and if you liked the IPv6 series, please let me know or +1 this post. You can also find me on Twitter as @tomperrine



Leave a comment

IPv6 – MacOS Snow Leopard update

In this earlier post, I alluded to MacOS X Snow Leopard not supporting IPv6 out of the box. I mentioned that you needed these two commands to make IPv6 work:

# ip6 -a
# sysctl -w net.inet6.ip6.accept_rtadv=1

The first command was mentioned on a blog post as needed to “fully enable IPv6 features, beyond what is enabled via the Control Panel”. The second command enables the acceptance of IPv6 Router Advertisements.

This turns out to NOT be be needed at all. I did a complete new Snow Leopard install from the DVD this evening on a spare MacBook Pro, and everything IPv6 worked perfectly, out of the box. IPv6 was enabled by default, and fast visits to and showed full native IPv6 connectivity.

I can only surmise that somewhere along the way, my regular MacBook Pro had had IPv6 turned off in some unusual way. Or it could be that my original MacBook Pro was originally a Leopard install, which was upgraded to Snow Leopard.

So, MacOS X Snow Leopard completely IPv6 ready, out of the box. I’ll be testing Lion in January…

, , , , , ,


IPv6 DNS Part 3 (authoritative DNS via IPv6 transport)

In this post I’ll finish off DNS by ensuring that I have publicly accessible IPv6 DNS servers. As I pointed out in the first two IPv6 posts, there are three parts of getting to “IPv6 DNS”:

  1. The first is to get AAAA (quad-A) records into your DNS system. At that point clients can ask for the AAAA records over IPv4 and everything will work just fine.
  2. The second is for you to actually serve your DNS zones over IPv6.
  3. The third is to get hooked into the global IPv6 DNS system, so that you (and others) can resolve your IPv6 addresses.

In this post, we will deal with the third part, ensuring that all the DNS servers needed to resolve my AAAA records are IPv6 capable. This step isn’t strictly necessary since, as I pointed out before, there’s nothing wrong with serving your AAAA records via IPv4.

First, let’s take a look at the symptoms of my problem:

$ dig -6 +short +trace aaaa
 ;; connection timed out; no servers could be reached

What has happened here is that there is no authoritative DNS server that can be reached via IPv6. So, what’s the problem?

One thing that I’ve never mentioned is that “my” local DNS server is a hidden master. It holds all the zone files, but is not advertised. My advertised public DNS servers are elsewhere, and they pick up my zone data via AXFR whenever I make changes and they are sent a NOTIFY. So, while my local server has all the zone data, it will never be queried during a normal DNS lookup. The advertised DNS servers, the slaves, actually serve all the answers.

It turns out that there are two problems here:

  1. My external slave name servers aren’t IPv6 capable;
  2. My resolv.conf has no IPv6 name servers listed.

My external nameservers are run by a friend at his organization’s datacenter. They aren’t prepared to serve DNS over IPv6, and won’t be any time soon. The fastest way to fix this is to move my external DNS to a DNS hosting provider that is IPv6 capable. Fortunately, I can get IPv6 DNS from the same place that I get my IPv6 tunnel: Hurricane Electric.

Using their DNS slave server setup page, I can easily make Hurricane’s DNS servers be my public slave DNS servers. I do have to ensure that their DNS servers can do zone transfers from my hidden master, which is mostly handled by using the the allow-transfer statement in my Bind config files, which is left as an exercise for the student.

After setting up the slave servers at Hurricane, I can wait for them to zone transfer my data to their servers. This can take 5-10 minutes for the first transfer.

At this point, there are three completely independent sets of name servers that have correct data for my domain:

  1. My home hidden master
  2. The current advertised DNS servers at my friend’s organization
  3. The as-yet unadvertised DNS servers at Hurricane Electric

All that remains is to remove the advertisements for the “old” servers and to add advertisements for the new servers. This is done at your domain registrar. Fortunately, my registrar is Register4Less, and their DNS system can take IPv6 addresses. In fact, it was easier than I expected. Unlike some other registrars I’ve used in the past, at Register4Less, you enter the names of your DNS servers, not the IP addresses. Register4Less resolved the hostnames and created NS records for both the IPv4 and IPv6 addresses.

There’s still one step left, and that’s to add some IPv6 name servers to my resolv.conf file.  I could use the five DNS servers that Hurricane Electric has advertised, but I’ll go one step farther. I’ll use the anycast DNS server address that Hurricane gave me when I established my tunnel:

$ more /etc/resolv.conf
nameserver                     ;; IPv4 nameservers from my ISP
nameserver 2001:470:20::2                 ;; IPv6 anycast name server from HE.NET

And now here’s a full “trace” of IPv6 DNS resolution:

$ dig -6 +trace aaaa

; <<>> DiG 9.7.3 <<>> -6 +trace aaaa
;; global options: +cmd
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
.            79981    IN    NS
;; Received 509 bytes from 2001:470:20::2#53(2001:470:20::2) in 31 ms

org.            172800    IN    NS
org.            172800    IN    NS
org.            172800    IN    NS
org.            172800    IN    NS
org.            172800    IN    NS
org.            172800    IN    NS
;; Received 436 bytes from 2001:7fe::53#53( in 157 ms        86400    IN    NS        86400    IN    NS        86400    IN    NS        86400    IN    NS
;; Received 112 bytes from 2001:500:c::1#53( in 188    28800    IN    AAAA    2001:470:67:84::10
;; Received 62 bytes from 2001:470:500::2#53( in 28 ms

And I’m done with DNS. Now I can go on to some other services, like SSH, FTP, HTTP, etc.

, , , ,

1 Comment

IPvFox, my favorite new plug-in

Now that I have a functioning IPv6 network, I can actually “see” how much of the public Internet (or at least web sites) are IPv6. Before I had the home net on IPv6, I was limited to just using DNS queries for AAAA records (over IPv4).

My new favorite FireFox plug-in is IPvFox, which gives me IPv6/IPv4 information right in the URL “awesome bar”.  I can tell at a glance, if the current page’s data was served over IPv6, IPv4 or mixed.

Here are a few images showing which sites/pages are loaded via IPv6, IPv4, or both.

This first one is interesting, As you can see from the image, the main page (URL) is IPv6 (big green “6”), but other parts of the page loaded via IPv4 (little red “4”). Clicking on the 6/4 image in the URL bar shows you which parts loaded which way. The main URL is IPv6, but the other parts of the page loaded over IPv4. Note that loads over IPv4.

This next one is Again the main page loads via IPv6, but the other content on the page is loaded from a combination of other sites running IPv4 and IPv6.

Here’s another IPv6 test site, This one uses IPv4 for the main site, and then pulls elements over IPv6 and IPv4.

As one of the newest of Google’s Internet properties, it is not unexpected that loads over IPv6, at least in this example. Go back and look at the first example, however, where it loaded over IPv4. Strange…. However, the “+1” system is still IPv4:

As I do my daily browsing, it’s interesting which sites come up over IPv6, and which don’t. I’m seeing more media and social sites on IPv6, and very few vendor sites. I had expected to see much more IPv6 from the big network kit vendors, but they are noticeably missing. Some of them “do” IPv6 on a separate host (, for example).

Not surprisingly, the main DREN web site is 100% IPv6.

Cisco, Juniper, IBM, Apple and Dell are all 100% IPv4. Many Mozilla sites, and a few US Government sites (Department of Education), and even Fark! are all solidly IPv6.

I wonder if the social media sites will lead the charge, or the vendors? Right now, I’m not seeing a lot of commitment from companies that I would hope have a lot more IPv6 experience.

They are going to want my company’s money for new network gear in the coming year, and I’m going to be asking hard questions about why they don’t have their own main sites running IPv6.

, , ,


IPv6 – DNS Part 2 (serving DNS over IPv6)

As I pointed out in the last IPv6 post, there are three parts of getting to “IPv6 DNS” and we only accomplished Step 1.

  1. The first is to get AAAA (quad-A) records into your DNS system. At that point clients can ask for the AAAA records over IPv4 and everything will work just fine.
  2. The second is for you to actually serve your DNS zones over IPv6.
  3. The third is to get hooked into the global IPv6 DNS system, so that others can resolve your IPv6 addresses.

Here we go with Part 2, to get our own Bind name server to listen (and answer) on IPv6 transport.

Depending on your particular version of Bind and Linux, you may already be ready to listen for queries on IPv6. Look in /etc/bind for the config file that contains the Bind options. In my version of Ubuntu, this is /etc/bind/named.conf.options

All you need is the following directive:

listen-on-v6 {any};

This tells bind to listen on all IPv6 interfaces. There’s an implied “listen-on {any};” which does the same for IPv4. In both cases, you can also use the “listen” directive to select specific interfaces or ports for Bind to listen to in IPv4 and IPv6.

After changing the config file and restarting Bind, we can query DNS over IPv6 from the same computer:

server$ dig -6 +short @::1 aaaa         #::1 is the IPv6 loopback address

Let’s try it from a client:

client$ dig -6 +short @2001:470:67:88::10 aaaa       # that IPv6 address is the address of my DNS server

And, we’re done with Step 2. But what happens if we don’t provide a specific DNS name server?

client$ dig -6 aaaa
; <<>> DiG 9.6-ESV-R4-P3 <<>> -6 aaaa
;; global options: +cmd
;; connection timed out; no servers could be reached

My DNS server is fine, BUT there’s something wrong when I query starting from the ROOT name servers. And that’s what we’ll look into next time.


IPv6 is catching on

IPv6 is getting some traction. You can see this in (almost) real time by following these web sites:

And most interesting, here’s a list that is driven by the Alexa top 1,000,000 web sites:

It’s interesting to see who is already “just using” IPv6, in that they are running IPv6 on their main domain web server. It’s also interesting to see who isn’t fully running IPv6.

On the “good list” I found lots of open source projects, lots of Universities, lots of US government web sites, etc. But some vendors are notably missing.  Cisco doesn’t have any IPv6 web presence I could find, and Juniper Networks serves IPv6 at, not on its main site.

Hopefully I’ll be on the “good” list by Christmas with at least one of the home web sites.

In the meantime, here’s a quote from Google network engineer Irena Nikolova: “At some point, we stopped talking about IPv6 as the ‘new protocol’ and started calling IPv4 the ‘old protocol’,” in an interview about a paper presented at LISA earlier this month.

Leave a comment

IPv6 – DNS Part 1 (AAAA records)

With the clients now all speaking IPv6 (with IP addresses from stateless auto-config), and the server now having a global-scope static IPv6 address, it’s time to make this much more useful.

With IPv6 address being 128 bits (32 Hex characters), it’s just not practical to expect anyone to remember IP addresses. DNS becomes much more important, not only for servers (with static addresses) but for clients. Clients will in general get their “real” IPv6 address via DHCP6 and do dynamic DNS updates. (There’s a special “stateless” DHCPv6 that just listens for the auto-config’ed IP addresses and put them into DNS.)

There are three parts of getting to “IPv6 DNS”.

  1. The first is to get AAAA (quad-A) records into your DNS system. At that point clients can ask for the AAAA records over IPv4 and everything will work just fine.
  2. The second is for you to actually serve your DNS zones over IPv6.
  3. The third is to get hooked into the global IPv6 DNS system, so that others can resolve your IPv6 addresses.

In this installment, we’ll just do Step 1.

Lets do the AAAA records and test some queries. If you’re this far along, editing Bind zone files and using “dig” should be second nature for you, so I’m only going to show snippets from the zone files:

;;; services
www         a       ;; original IPv4 address
www         aaaa    2001:470:67:88::10 ;; NEW IPv6 address, same name
ipv6        aaaa    2001:470:67:88::10 ;; NEW ipv6 address, new name for ease in testing

I’ve added two new records, a second “www” entry and a completely new “ipv6” entry. The “ipv6” entry is so that I have a hostname that has only an IPv6 address, and no IPv4 addresses. Let’s see what I can get (after I reload the zone)…

$ dig +short              # 1 asking for the "A" record for "ipv6" - NO AAAA records exist
$ dig +short aaaa         # 2 asking for the AAAA record - SUCCESS
$ dig +short               # 3 asking for the "A" record for "www" - SUCCESS
$ dig +short aaaa          # 4 ...and the AAAA record - SUCCESS
dig -4 +short aaaa         #5 force IPv4 query (which is actually the default) - SUCCESS
$ dig -6 +short aaaa       #6 force query over IPv6 transport - NO RESPONSE
^C            #hangs

Two notes:

  1. By default “dig” queries for “A” records if no other record type is given.
  2. Be default “dig” queries over IPv4.

This explains why query #1 returns no data and why #3 returns the “A” record (only). To get the “AAAA” records, you have to explicitly ask for them with a record type. Finally, query #6 attempts to force the DNS queries to use IPv6 for transport, which hangs since there are no know IPv6 DNS resolvers configured in the system.

At this point we’ve achieved step 1, we have AAAA records in our DNS, and we can retrieve them via IPv4.

Next step, having our own DNS server answer queries over IPv6 transport.


IPv6 – assigning a global static address for a server

In this installment, we assign a global, static IPv6 address to an interface on the home Linux Server. This is the first step to go from “my first ping” to a (mostly) fully functional IPv6 home Linux server with SSH, HTTP and other services.

To get from the all-client network to a functioning server, we need to get a persistent static IPv6 address on at least one of the server’s interfaces and get that address into DNS. Then we need the service daemons to bind to that IPv6 address.

In addition to being much larger than IPv4 addresses, IPv6 addresses are also more complicated. Like IPv4, IPv6 has unicast, multicast and anycast addresses, but there are no broadcast addresses. You’ll use either anycast or multicast instead. IPv6 addresses also have scope. The two scopes that we need to be concerned with for this setup are link-local and global.

A big change in IPv6 is that you need to think in terms of “interfaces” instead of “hosts” having IPv6 addresses, and every interface will have multiple IPv6 addresses as a matter of course.

IPv6 has “auto configuration” built in, which will give you two addresses: a link-local unicast address and a global EIU-64 unicast address. Both are created by using the interface’s hardware (MAC) address as the lower 64 bits of the full IPv6 address.

At this point in my project, all the home systems are using “link local” and modified EIU-64 IPv6 addresses. You can see this on the Linux server via “ifconfig”:

server$ ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:30:1b:82:cb:42  
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::230:1bff:fe82:cb42/64 Scope:Link
          inet6 addr: 2001:470:67:88:230:1bff:fe82:cb42/64 Scope:Global

This interface has three IP addresses: an RFC-1918 IPv4 address, a link-local address (the fe80:: address) and a global EIU-64 auto configured address that is made by concatenating the /64 IPv6 prefix (2001:470:67:88) and the interface hardware address (230:1bff:fe82:cb42).

At this point, we could put the EIU-64 address into DNS, but that is not a good practice. Changing the interface’s hardware (or the whole server) would change the address, requiring a DNS change. DNS names should be tied to services, not hardware…

Getting an IPv6 address onto the eth0 interface is relatively easy; just edit /etc/network/interfaces to add something like this:

### Start IPV6 static configuration
 iface eth0 inet6 static
 pre-up modprobe ipv6
 address 2001:470:67:88::10
 netmask 64
 ### END IPV6 configuration

Reset the interface and check its addresses:

server# ifup --force eth0
server# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:30:1b:82:cb:42
          inet addr:  Bcast:  Mask:
          inet6 addr: fe80::230:1bff:fe82:cb42/64 Scope:Link
          inet6 addr: 2001:470:67:88::10/64 Scope:Global

And now the interface has a global, static IPv6 address. You can demonstrate this with a quick SSH to the global address, from another client on the home network:

laptop$ ssh -v tep@2001:470:67:88::10
 OpenSSH_5.2p1, OpenSSL 0.9.8r 8 Feb 2011
 debug1: Reading configuration data /etc/ssh_config
 debug1: Connecting to 2001:470:67:88::10 [2001:470:67:88::10] port 22.
 debug1: Connection established.

And we’re done. Next, DNS AAAA records and services.


Wells Fargo/Wachovia merger – culture is key

One of this morning’s keynotes at Gartner Datacenter Conference (#gartnerdc) was an on-stage interview with Scott Dillon, EVP, Head of Technology Infrastructure for Wells Fargo. He was interviewed about the Wells/Wachovia merger,  and the challenges faced by the organization.

While the talk was full of sound bytes about scale, talk about merger strategies and budgets, the discussion came back to culture over and over.

On the scale and technology side, there were tidbits like these:

  • Wells Fargo employs 1 in 500 in the US;
  • IT had 10,000 change events per month, before the merger;
  • They have a physical presence within 2 miles of 50% of the US population.

But it was on the management side that I found the most interesting information.

Before the merger, there were clear guidelines, such as “if we have two systems, A and B that are doing the same thing, we will pick the best, either A or B. No C options.” This was a merger of equals, at least in terms of the technology. They chose the best of the two orgs, then committed to making that the One True New System for everyone. They ended up with an almost 50/50 split of technology from the two companies.

But, no matter where the talk went in management and technology, it just kept coming back to culture. Building one culture from the best of both was a top management priority for the entire company. Just as they (IT) selected the best tech from each, they (Executive management) worked to take the best of the culture from both, to be the foundation moving forward. They had a great advantage, as both companies share almost all of their core values, so this was a little easier than merging the technology. But there was an explicit decision to do this, it wasn’t left to chance.

Management made “culture” a number one priority. They focused on merging the culture as much as they focused on merging the technology.  They made building communications between the employees an early priority. Very early on, they even created a “culture group” to look at the two cultures and make specific decisions about how to foster the culture merger.

Part of their culture involves employee value. Every company does “exit interviews” when employees leave. Wells does “stay interviews” where they engage with employees to actually gather their concerns, let them know how much the company values and appreciates them. Isn’t that better, to find any issues before key people leave? To constantly work to make the work environment better, instead of waiting until it’s too late?

In IT we often get too focused on the technology, and we can claim that “the business” is too focused on profits, or stock price, or some other “business” area.

When was the last time you heard a business, a bank, even, put their culture as one of their highest priorities?

More importantly, as IT, when was the last time we put “culture” high on our priority list?

, , , ,

Leave a comment

IPv6 – setting up the tunnel on the home router

At this point, there is an IPv6 tunnel from Hurricane Electric to my home Linux server. That’s OK, but not what I need for “production”. As I mentioned in the requirements, I want to use a commercial home router solution so that it can be easily replicated by some of our non-technical staff.

Based on recommendations from some Navy DREN folks, I selected an Apple Airport Extreme. Some of them have been doing IPv6 testing (and home IPv6 tunnels) for upwards of six-seven years.  Based on their comments, the Airport has quite good IPv6 functionality. All that is needed is to load the tunnel configuration information from the web page into the router, using the Airport Utility. You can see the instructions for this here.

The only drawback to the Airport is that every, and I mean every configuration change forces a reboot, which interrupts connectivity for about 30-50 seconds. Other than that, rock solid.

Leave a comment

%d bloggers like this: