fewer chat choices leads to stronger more collaborative global community

We reduced the number of choices for online chat and ended up with a more concentrated, focused and collaborative global community. This wasn’t planned, it happened organically, and it’s still in progress. But it shows the value of allowing and encouraging people to concentrate into common systems, even ignoring any financial considerations.

As recently as four years ago we had a multitude of separate group and direct chat applications. We had multiples of everything from private in-house Internet Relay Chat (IRC), Jabber and similar servers to several generations of Microsoft products. At one point I counted no fewer than 8 different chat servers that I needed to use myself, just to reach my customers and peers. For the record, that was two different Internet Relay Chat (IRC) servers, two different Jabber servers (the original open source XMPP server), two different Slack instances, Microsoft Lync, and Microsoft Office Communicator (which is now Skype for Business).

(This doesn’t even get into the multiple WebEx, Zoom, GoToMeeting and special conferencing systems.)

Completely ignoring the efforts and costs of running the in house IRC and Jabber servers, the major problem was people wasting time trying to remember (or figure out) which app or server they needed to use to reach a particular person or group. This led to lots of conversations like this…

“Does Susan use Slack?” “No, she’s on Microsoft.” “I don’t see her in Lync?” “She’s in Communicator”.

“Dear IT – I’m in Tokyo and I can’t reach UK R&D over Slack. Please fix.” “R&D is in the other Slack.” “What other Slack?”

This was bad enough for individuals, but finding out that perhaps 5 people who needed to collaborate were “homed” on three different platforms required getting them all accounts on a single platform just for that particular project or need. This led to an even worse explosion of per-user accounts, as now everyone had to have multiple accounts on multiple platforms.

This is similar to, but worse than the fracture in social media. While the “do I find you on Facebook or Google+ or Reddit or Twitter or Instagram or email or SMS?” question is annoying for individuals, having this in a global company is untenable in the long term.

Slack started as a grass roots effort among several of our internal communities. These groups “routed around” the IT-provided solutions and adopted Slack independently, and in most cases without knowing about each other. Obviously, this initially made the problem worse! But, as these groups found out about each others’ Slack instances, they began to talk about whether it would be good for them to merge their communities into fewer (or a single) Slack instance.

Importantly, all the IT organizations around the world realized what was happening and helped and encouraged the move. They made Slack a supported and preferred solution, instead of fighting it.

Over the past year some of our groups around the world have been organically moving their communities to Slack, retiring old services on their own. All the IRC servers and most of the Jabber servers have been decommissioned. Multiple Slack instances have been merged into a new “main Slack”, with more planned to move this year. More importantly several “new” chat systems have NOT been launched; those communities have agreed to adopt Slack instead.

This only worked because people wanted “one place” to gather and Slack offered a “good enough” experience. It’s not important that they selected Slack, it’s important that they all want to be in “one place”, and that “they” selected Slack, it wasn’t imposed. And frankly Slack was better than almost all of the legacy systems.

It’s not clear that we’ll stay on Slack forever, as some of the promised Microsoft solutions may offer better integration with Active Directory, desktop voice and video chat, etc. But until those come, all our users have the option of a single place to collaborate.

WordPress.com – still no IPv6

Three years ago I wondered when wordpress.com would begin to support IPv6. The irony of hosting a blog that talks a lot about IPv6 on an IPv4-only platform is not lost on me 😦

Here we are in 2016 with Google, Akamai and pretty much all content providers are reporting more IPv6 traffic, and WordPress.com is still stuck in 1983.

This is not an OS issue, a load balancer issue, a PHP language issue, or even a WordPress software issue. I’ve seen WP software running full dual-stack, no problems. Five years ago. I’ve done it myself, I just don’t want to run WP myself anymore.

For whatever reason, WordPress.com has not decided to commit to the future. yet.

USA! USA! (unless you want affordable high-speed Internet service)

“In comparison with the rest of the developed world, the US has slower broadband speeds and higher broadband prices than just about anybody.”

No surprises here.  US ISPs and cable companies (among many other industries) continue to rock record profits, and instead of investing, just buy back their stock, or sit on the cash.

On the technology front, this means that instead of upgrading backbones, or delivering native IPv6, or a higher quality of service, they are deploying stopgap measures. Some examples of this are Carrier Grade NAT (CGN) instead of native IPv6. “Dumb” DVRs that are less programmable, and less usable than some home grown solutions. No investment in technical support.  Man-in-the-middle ad networks, DNS hijacking, abusive legislation, and other interference with their customers’ data.

As long as the last mile is a de-facto monopoly, that’s just what we’re stuck with.

IPv6 adoption – still doubling every year

Good News, everyone!

IPv6 adoption continues to double, year on year.  Of course, that’s only three years of baseline, but things are certainly moving in the right direction.

As this article points out, if this rate continues, more than half of Internet users could have IPv6 within 6 years. This goes along with estimates of IPv6-only customers reaching 20% by 2017.

It remains to be seen if this adoption rate can continue.  However, events such as Switzerland moving from 3% to 10% adoption in a single month are interesting. They show that a single large ISP can quickly make a huge difference in adoption rates, as they turn up large portions of IPv6 connectivity in large deployment events. I expect Comcast to quickly begin to have a similar impact on US IPv6 availability later this year.

This CircleID article calls out some winners and losers in the IPv6 adoption race. My favorite IPv6 (tunnel) provider, Hurricane Electric gets high marks.

Having “the talk” with your vendors

My company buys a fair amount of IT kit each year. We visit and are visited by vendors almost weekly. Lately, “the talk” has become part of the conversation: “How’s your IPv6 support?”

We’ve had this discussion with our network vendors for quite a while, now we’re talking to the rest of the vendors: storage, cloud services, middleware platforms, monitoring, and security.

A very select few of them have answered immediately: “Of course, we’ve had it for years. What can we do to help you with your IPv6 transition?”

Others have said, “It’s on our roadmap, about X months out, would you like to be in the Beta?”

But too many have responded, “IPv6? Is that important to you? You’re the first customer who has asked about it. We’ll get back to you…”

I predict a rocky next 5 years for the vendors in the the last group.  Smaller, more agile, more forward-thinking upstarts are going to make life “interesting” for those folks.

You should have “the talk” with your vendors. If they can’t help you move forward on IPv6, you’ll need to find alternatives that can.

 

Total Cost of Ownership of dual-stack IPv4/IPv6 and Carrier Grade NAT (CGN)

INET Denver was held last month at the same time and location as the North American IPv6 Summit.  The theme was “IPv4 Exhaustion and the Path to IPv6″.

I missed the morning talks as they conflicted with my advanced IPv6 class, but I did catch the afternoon sessions.

It seemed like there were three camps at INET Denver: people already embracing the future by deploying IPv6, people trying to avoid IPv6 as long as possible, and people who planned to make money from both of the other two camps.

Let’s talk about the “wait as long as possible” camp.

For almost two decades the argument from the “business side” of IT was there was “no compelling business reason to move to IPv6“. (That article is is from 2009 by the way, but I haven’t seen a new argument since then.) It’s true, there’s been no “killer app” that everyone demanded that was only available via IPv6. It’s also true that doing nothing was a legitimate strategy for quite a while. After all, what good is it to have a telephone (IPv6), if no one else has one? Until recently, moving to IPv6 truly didn’t have a compelling business argument. After all, doing nothing costs nothing.  Mostly.

The Internet has changed. And we’re (almost) out of IPv4 addresses, so you have to do something. Sadly, too many ISPs have tried to do what they think is the cheapest and most minimal amount of work they could get away with. That’s Carrier Grade NAT (CGN).

The economics have changed, too.  Lee Howard of Time Warner Cable had a very interesting talk where he deconstructed, and then destroyed the myth that CGN is cheaper to deploy than dual-stack.  Since he’s the Director of Network Technology for Time Warner Cable, I guess he knows more about the ISP business than most people.

Mr Howard’s talk shows that CGN will cost you in (unhappy) customers, support costs, and only delay the inevitable, when you’ll have to move to dual-stack anyway.  His talk effectively demonstrates that the infrastructure and operational costs of a CGN network are more expensive than dual-stack.

There’s your business case. Deploy IPv6 and save money. Done. Now get to work.

LOPSA San Diego – Tomorrow

LOPSA San Diego Meeting

Thursday Feb 28 2013

6pm until whenever

Callahan’s Pub in Mira Mesa

This will be a social, meet and greet meeting.  Come and meet some of the fine sysadmins in the San Diego area. Come out and meet your peers, network, talk shop, grab a bite and/or a beer and celebrate all things syasdmin.

LOPSA is an international professional society for IT people of all job descriptions.

If you’re planning to attend, please RSVP at Meetup.com so we can get a headcount ahead of time. Of course, if you can only make it at the last minute, you’re very welcome too! (We understand the life of a sysadmin!)

Our members manage everything from desktops to servers, storage to networks, laptops to supercomputers. Come out and get connected to the rich sysadmin community in San Diego!

IPv6 – CGN and Teredo Considered Harmful

There, I said it. The so-called “IPv6 transition strategies” are making it harder, more complicated and less secure to deploy IPv6 than just “doing the right thing”.

Carrier Grade NAT (CGN) and Teredo (among others) are the last gasps of an IPv4 world, and have no place in the modern Internet. While they may have short-term advantages to network operators, they will cause problems for their end users until they are finally phased out. Dual stack would be a better transition process, especially for customers.

CGN is, as much as anything else, a way for carriers with a large network or large installed base of end users to make the fewest (and hopefully least expensive) changes in their networks. They are betting that by introducing a small number of large-scale NAT devices on the border between their networks and the Internet that they can avoid making sweeping internal network changes, or upgrading CPE (Customer Premise Equipment).

At best, even when working correctly, CGN breaks end-user accountability, geo-location and the end user experience. On top if that, it will slow IPv6 adoption, and force “true IPv6” users to adopt a host operational work-arounds and complicate deployment of next generation mobile and Internet applications.

CGN is inherently selfish on the part of the network operators that deploy it. They are saying “I want to spend less money, so I’m going to force everyone else to make changes or suffer in order to continue to talk to my customers.”

Or, as Owen Delong put it in his excellent look at the tradeoffs in CGN:

Almost all of the advantages of the second approach [transition to CGN and avoid investing in IPv6 deployment] are immediate and accrue to the benefit of the provider, while almost all of the immediate drawbacks impact the subscriber.

The next part of my rant has to do with Teredo, a “last resort transition technology”.

Like CGN, Teredo promises to allow end-user equipment to connect to the public IPv6 Internet over IPv4. It does this by “invisibly” tunneling your IPv6 traffic over the public Internet, to a “Teredo gateway”. A Teredo gateway performs a 4to6 network translation and passes your traffic onto the desired IPv6 destination. Teredo is implemented transparently in some Microsoft operating systems and can by default provide an IPv4 tunnel to the outside world for your IPv6 traffic.  It can, also provide an “invisible” tunnel from the outside world back into the heart of your network. And of course, all your network traffic could be intercepted at the Teredo gateway.

Teredo security has been a hot topic for years, with some concerns being raised shortly after Teredo’s standardization in 2006, and RFC6169 finally providing IETF consensus in 2011. Sadly, Teredo security must still be discussed, even though it is 0.01% of network traffic to dual-stacked resources. Fortunately, there’s a move in IETF to declare 6to4 technologies (including Teredo) as “historic”. Teredo will complicate network security until it is gone.

I for one, cannot wait for both CGN and Teredo to be consigned to the dustbin of history.

LOPSA San Diego – first meeting

LOPSA San Diego is getting underway with a Meetup next week:

Thursday Jan 24 2013

6pm until whenever

Callahan’s Pub in Mira Mesa

This will be a social, meet and greet meeting.  No presentations at this one, but come out and meet some of the fine sysadmins in the San Diego area. Come out and meet your peers, network, talk shop, commiserate and celebrate all things syasdmin.

LOPSA is an international professional society for IT people of all job descriptions.

If you’re planning to attend, please RSVP at Meetup.com so we can get a headcount ahead of time. Of course, if you can only make it at the last minute, you’re very welcome too! (We understand the life of a sysadmin!)

Our members manage everything from desktops to servers, storage to networks, laptops to supercomputers. Come out and get connected to the rich sysadmin community in San Diego!

When will WordPress.com support IPv6?

Dear WordPress.com, when will you support IPv6?

Over the past year I’ve watched more and more web sites come online on IPv6.  Some are “the usual suspects”; the high-tech, early adopter sites that you expect to be moving aggressively onto IPv6. Some early adopters have been surprising. While the WordPress software itself works fine over IPv6, WordPress.com itself seems to be a no show to the IPv6 game.

In fact, the only mention I can find from WordPress about IPv6 hosting is a blog post from World IPv6 Day in 2011.

This blog (hosted at WordPress.com) has a lot of content about IPv6, and I get about one private comment every other month pointing out the irony that the blog can’t be viewed over IPv6.  I’d rather not move to Blogger.com (fully IPv6 capable), or spin up my own instance of WP if I an avoid it.

So, WordPress folks, can you at least give a timeframe for IPv6 support?

Since I’m an architect on a worldwide enterprise internal IPv6 rollout, I *do* understand the challenges involved, and the uncertainty that you might have on a fixed schedule.  But could we get at least a comment that “we’re working on it”, or “sometime in Q4 2013”, or “not planning to do this for at least a few years”?