Posts Tagged IPv6
No surprises here. US ISPs and cable companies (among many other industries) continue to rock record profits, and instead of investing, just buy back their stock, or sit on the cash.
On the technology front, this means that instead of upgrading backbones, or delivering native IPv6, or a higher quality of service, they are deploying stopgap measures. Some examples of this are Carrier Grade NAT (CGN) instead of native IPv6. “Dumb” DVRs that are less programmable, and less usable than some home grown solutions. No investment in technical support. Man-in-the-middle ad networks, DNS hijacking, abusive legislation, and other interference with their customers’ data.
As long as the last mile is a de-facto monopoly, that’s just what we’re stuck with.
Are your servers getting SLAAC addresses in addition to the addresses you are manually configuring? If so, read on…
You need to find and turn off the “A” bit in the Prefix Length option of your Router Advertisement packets. The “A” bit is on by default on most network routers, and the documentation that describes the interactions between the “M”, “O” and “A” bits is scattered across at least a half dozen RFCs.
When we first set up our IPv6 lab, we went through several phases. Initially we just did client subnets and hosts and let all the stations auto-configure (SLAAC). This all happened “magically” with the default behavior of all the operating systems and network gear we tested.
Then we split the clients and servers onto separate subnets. When we did the split we added a DHCPv6 server and turned ON the M and O bits for the client subnets. For the server subnets, we turned OFF the M and O bits and statically configured the IPv6 (and IPv4) addresses.
The client hosts did everything exactly as expected, gathering IPv6 addresses and other options, exactly as they would have using DHCP and IPv4.
But, we never could quite get the servers to stop creating and configuring SLACC addresses, even with M & O bits turned ON or OFF on their subnets. Making sure that we did NOT have DHCPv6 clients configured on these servers, we tested all four states with nearly identical results.
In other words, each server would always end up with three IPv6 addresses:
- a globally unique (global scoped) static assigned address, the one we configured at boot time
- a globally unique (global scoped) SLAAC address, usually based on its MAC address
- the usual and expected link-local address (fe80::)
So, what else was going on? Most of the documentation we found (especially RFCs) described these two bits in excruciating and often contradictory fashion! Take a look at RFC 4861 for the format of the Router Advertisements, and you’ll see the M and O bits right there in section 4.2). If there are other option bits that might control this, shouldn’t they be shown here?
By the way, the M and O bits are always OFF by default on all the networking gear we’ve seen so far (Cisco, Juniper and HP).
4.2. Router Advertisement Message Format
Routers send out Router Advertisement messages periodically, or in response to Router Solicitations. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Code | Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Cur Hop Limit |M|O| Reserved | Router Lifetime | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reachable Time | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Retrans Timer | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options ... +-+-+-+-+-+-+-+-+-+-+-+-
But in all four combinations of the M and O bits, and IF you aren’t running a DHCPv6 client, you get a SLAAC address in addition to the address you statically (manually) configure. How do you turn off “auto conf” if it isn’t controlled by flags in the Router Advertisement???
It turns out that there are actually three bits in the RA that control host configuration, not two, and so there are 8 possible cases of M, O and “A”, not four. So where is this mysterious “A” bit hiding?
The “A” bit is “hidden” in a Router Advertisement option (“Prefix Information”), which is described in section 4.6.2, about 10 pages farther along in the RFC. This option’s purpose is to tell you about the length of the valid address prefix that’s available on the current subnet, but it also has “A” that controls whether or not a station on that subnet should do SLAAC. And unlike M and O, A seems to always be set ON by default.
4.6.2. Prefix Information
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type | Length | Prefix Length |L|A| Reserved1 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Valid Lifetime | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Preferred Lifetime | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Reserved2 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | + + | | + Prefix + | | + + | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Fields: Type 3 Length 4 Prefix Length 8-bit unsigned integer. The number of leading bits in the Prefix that are valid. The value ranges from 0 to 128. The prefix length field provides necessary information for on-link determination (when combined with the L flag in the prefix information option). It also assists with address autoconfiguration as specified in [ADDRCONF], for which there may be more restrictions on the prefix length. L 1-bit on-link flag. When set, indicates that this prefix can be used for on-link determination. When not set the advertisement makes no statement about on-link or off-link properties of the prefix. In other words, if the L flag is not set a host MUST NOT conclude that an address derived from the prefix is off-link. That is, it MUST NOT update a previous indication that the address is on-link. A 1-bit autonomous address-configuration flag. When set indicates that this prefix can be used for stateless address configuration as specified in [ADDRCONF].
So, that’s where the mysterious server SLAAC addresses come from. They are caused by the default-on “A” bit that is in the Prefix Information option to the Router Advertisement. Clear this A bit on your server subnets, and you’ll get only the IPv6 addresses that you configure, and no more SLAAC addresses as an extra bonus.
After I figured out what was going on, I also found these web pages which each shed some light on the situation:
This post begins a series that will create a sample IPv6 address plan for a medium sized company with multiple office sites and multiple hosting locations.
In prior posts, I’ve written about IPv6 address plans in general and shown an especially interesting plan from UCSD.EDU. This post begins to “do the math” for a hypothetical company to produce a concrete addressing plan. This company is similar to my employer but I’ve thrown in a few things that we don’t have that many other more typical companies will have.
For the purpose of this thought experiment, we’ll use a company that looks like this:
- six office locations (cities)
- three co-location facilities that host consumer-facing services
- is part of a larger multi-national, but there is no higher-level single global network
For the office locations we’ll use these criteria:
- around 5000 employees total
- some locations have multiple large groups
- most users have at least four devices that need an IP address, many have six and some have 10
- two of the locations have their own Internet connections as well as a connection on the internal WAN
- some locations have multiple Internet connections from multiple providers
The co-location facilities look like this:
- up to 5000 hosts (or instances) per location
- multiple Internet connections from multiple providers
- each location houses sub-facilities that are leased to other business units (separate private clouds and the like)
- each sub-facility needs to be independently routable via separate ISPs (not all use the same ISPs)
- also used for Disaster Recovery (DR) for the offices
With this information, we can begin to create the IPv6 address plan. Creating the plan begins with determining the Assigned Global Routing Prefix and subnet sizes and concludes with subnet numbering and routing plans.
The constraints we have to work within are:
- The assignment of the Global Routing Prefix is gone by an Internet Registrar according to their policies. You must justify the size of the allocation you request.
- Subnets are “always” on a /64 boundary (host identifiers are “always” 64 bits)
- “Sites” are groups of subnets on a /48 boundary
- Only networks with prefixes at /48 or larger are considered “publicly routable” by most ISPs. They won’t announce routing data for anything smaller.
Next time, I’ll look at these constraints and how they factor into the size of a desired Global Routing Prefix and creating a site and subnet plan.
Is IPv6 getting enough traction to be called mainstream, yet? Sort of. Lots of groups are tracking world wide IPv6 adoption through various means, often looking at the percentage of web sites that are IPv6 reachable. But is this the right metric?
World IPv6 Launch Day did “prove” that IPv6 is viable and that more people are using it. But, does it matter that Romania has 8.64% adoption or that the US has 1.77%, or that France has 4.61%? How does that relate to a “real user”? The answer of course is that it doesn’t. I don’t (often) visit Romanian or French web sites, and the experience of Internet users in those countries is affected by the use (or lack) of IPv6 elsewhere in the world. Facebook, Google, Twitter and others are all worldwide communities.
One way to see how (if) IPv6 adoption is affecting you is to look at which web sites that you visit every day are IPv6 capable.
I took the last 30 days of browser history from my laptop and looked at the IPv6 reachability for the sites that I actually use on a regular basis. Here are the results.
I started with the Firefox add-in “History Export 0.4” and exported my history for the past 30 days. This showed that I visited over 10,000 URLs in the recent past. This raw data was massaged by some Perl and Emacs macros to process, sort and extract unique domain names. Finally I used a bash script to do DNS lookups for A and AAAA records for all the unique hostnames that remained.
Here are the results:
- 10202 URLs in 30 day history
- 1310 unique host names (there were lots and lots and lots of Gmail URLs!)
- 125 of the unique host names had AAAA records indicating IPv6 reachability
- 2(!) hosts had AAAA records and no A records – IPv6 ONLY sites! W00t!
So about 9.5% of the sites that I visited in the past 30 days are IPv6 capable. That’s more than I had expected and more than the general Google IPv6 stats would suggest. Now, since I am doing IPv6 work I would expect to be an outlier, but am I an outlier for that reason?
Of the 125 IPv6 sites:
- 27 are Google properties (Google, Gmail. Blogger, Google Code, Youtube, Android Market and similar)
- 11 are IPv6 information or test sites
- 10 are US .gov (more on this later)
- 5 are notable open source projects (ISC Bind, ISC DHCP, Mozilla, Ubuntu, Fedora)
- 8 are larger .EDUs like Stanford, UCLA, UCSD
- 6 Internet governance and operations sites (IANA, IETF, Internet Society, ARIN, APNIC)
- 6 are blogs hosted at blogspot.com
- 5 of the sites are Facebook properties
- 2 Netflix sites
- 2 Wikipedia
- 2 Yahoo properties
- the rest (almost 40) are singleton sites – individually hosted blogs, news and aggregation sites (political, tech)
This means that 91% of the IPv6 sites I visit are probably typical for a “regular” Internet user. Most of the most popular properties are represented in IPv6-land: Google, Facebook, Wikipedia, NetFlix and similar.
It also shows that while individual sites are making progress (those 37 singletons), many hosted sites will get upgraded through no action of their own when their hosting provider (or cloud provider) makes the switch. Between then, Blogger and Blogspot are now hosting thousands of personal blogs that are IPv6 capable.
Personally, I can’t wait for WordPress.com to make the switch.
As part of the IPv6 sprint at work last month, I ended up doing a lot of IPv6 research. For my part, I spent a lot of time researching “customer issues” and MacOS issues in addition to the purely technical work.
When I started the sprint, my laptop was on MacOS X Snow Leopard, which I used for all my home IPv6 work. Halfway through the sprint, I upgraded to MacOS X Lion.
The upgrade to Lion went well, but Apple has changed the behavior of some IPv6 features, and I personally would have to consider Snow Leopard as a better IPv6 platform than Lion.
Apple didn’t “break” IPv6 in Lion, but they did introduce a new problem, which has been dubbed “hampered eyeballs”.
I’ve noticed some newly-hampered IPv6 web browsing since the upgrade. Some sites that came back solidly on IPv6 100% of the time, now come back as IPv4 up to 20% of the time. (Thanks IPvFox!)
This has lots of implications for how consumers will see the new Internet, especially during the transition. According to some anecdotal remarks on some IPv6 mailing lists, this is being used as an excuse by some companies to delay (even more) any IPv6 transition or even dual stacking!
This last week was Game Developer Conference in San Francisco, next week is a global IPv6 meeting in New Jersey. I should have lots more “corporate” IPv6 info on the next 10 days.
The last two weeks at work have been some of the most fun in the past few years. A few months ago I moved from management back to my first love: deep technical work. In my new position I’m responsible (with a co-worker) for technical strategy, creating our Enterprise Architecture, and forward-looking technical projects. We’re also tasked with finding new ways to collaborate and take on projects as well as take a hard look to ensure that IT is supporting the rest of the business.
For some of these, we act as facilitators for IT projects, even though we aren’t in the management chain.
IPv6 has been one of my “back burner” projects for almost a year. There is a business mandate that we must have IPv6 connectivity to one of the inter-corporate networks by 1 April. A select set of our internal users need to have IPv6 connectivity to business applications that will only be available over IPv6 via this network.
To prepare for this, we had a need to ramp up IPv6 knowledge from almost nothing, to ready to plan a limited IPv6 deployment next month.
We decided to try a new project methodology (loosely) based on agile concepts: we performed IPv6 testing and deployment preparation as a “sprint”. We got 12 of our most senior system and network admins together in a large conference room with a pile of hardware, a stack of OS install disks, a new IPv6 transit connection and said, “Go!”.
No distractions, no email, no phone calls. Just 12 people off in a completely different building, in a big room with a pile of gear and the mandate to “explore IPv6” and learn enough to be comfortable planning a limited IPv6 deployment at the end.
It was great seeing people from different IT departments who usually specialize in Linux, MS Windows, VMWare, networking, security, etc. all come together to explore IPv6 on all these platforms, bring up services, test, find vendor bugs 🙂 and in general build a standalone IPv6 lab from scratch.
We truly did start from scratch; we started with an empty room, a bunch of tables and chairs, two pallets of PCs, assorted network kit, three boxes of ethernet cables and installation media.
Along the way, all of these people stepped out of their comfort zones, learned about each others’ specializations, and worked together for a common goal that we all created together.
At the end of the 2 weeks, we had a fully functioning dual-stack IPv4/IPv6 network:
- Routers and switches, firewall and IPv4/6 transit from a new provider
- Fully functioning Windows infrastructure: AD, DNS, DHCP, IIS, Exchange, etc.
- Linux infrastructure: DNS, DHCP, syslog, apache, Splunk, Puppet (mostly)
- Windows Server 2008 and 2008 R2, Windows 7 clients
- Linux Centos 5 and 6 servers and desktop
- MacOS Snow Leopard and Lion clients
All the results and everything we learned is documented in a wiki full of IPv6 configurations, hints and tips, debugging info, links to IPv6 info, lessons learned and plans for IPv6 next steps to production. I think we generated about 50-60 pages of new documentation along the way on IPv6, and about 6 pages of notes on the sprint experience itself.
The sprint wasn’t perfect, and we had a few stumbles along the way. But we learned a lot about how to run these kinds of sprints, and we’re pretty sure that we’ll have more of them in the future.
We also had two full weeks of face time with our colleagues from four sites in two states. In some cases we had never met each other in person, but had been exchanging email and tickets for years.
It was incredibly productive two weeks. We learned a lot about IPv6, each other and found new ways to work together.
Last year brought us World IPv6 (test) day on June 8. Dozens of content providers, network backbones and other technical groups came together to do a live test of IPv6 in production. Results were very good, and provided enough evidence that planning for a real, permanent cutover to full “dual stack” was practical.
However, there were enough issues that many of the participants took down their IPv6 sites after the experiment.
But this year, it’s gonna be real. June 6 2012 is World IPv6 Launch Day. The same big names and many other are participating. More importantly, some of the major providers of CPE (customer premise equipment) AKA “home routers” are committed as well.
Cisco and D-Link are committed to shipping “home equipment” with compliant IPv6 stacks and Ipv6 enabled by default by this date. Facebook, Google, Bing and Yahoo! will all permanently enable IPv6 for their main sites. In the US, AT&T, Comcast and Time-Warner will activate IPv6 for at “significant” portions of their home wireline customers.
And this time, it’s permanent. Unlike the 24 hour experiment last year, this is a permanent change. I expect that all the participants will have to shake out configuration issues and software bugs after the launch, but at least now they are committed to making IPv6 work for everyone, from now on.
The only thing that might make this better would be commitments from the operating system vendors. Apple, Microsoft and the Linux community already have known issues that will need to be addressed. Having the home router providers commit to some level of IPv6 support (firmware upgrades) for at least some currently shipping products would also be good, but I suspect they would rather sell new gear.
- World IPv6 Launch on June 6, 2012, To Bring Permanent IPv6 Deployment (internetsociety.org)
In this post, I’ll finish up the “usual services” for my home network. So far I’ve got IPv6 routing and DNS. Now I just want to confirm that I’ve got the rest of my “core services” accessible via IPv6.
(I’ve decided that I don’t need DHCP6 for my particular network, so I’ll be skipping that.)
My remaining core services are: SSH, HTTP and SMTP.
The SSH daemon (sshd) has been configured to listen on both IPv4 and IPv6 by default for years. In fact, it attempts to listen on the IPv6 port, even if you don’t have IPv6 enabled on the host OS. In my case specifically (OpenSSH_5.8p1 Debian-7ubuntu1, OpenSSL 1.0.0e 6 Sep 2011), I was able to “ssh ::1” as soon as I had eth0 set up with an IPv6 address.
As for HTTP, Apache 2 (Apache/2.2.20 (Ubuntu)), there were no config changes needed. Apache2 will listen on all the addresses (IPv4 and IPv6) that are configured when the daemon starts. All that was needed was a “server apache2 restart” once the IPv6 address was configured, and the web server began answering IPv6 requests.
SMTP turns out to be a little harder. Postfix doesn’t listen on IPv6 ports by default. You need a few config file changes in main.cf:
# listen on IPv4 and IPv6 inet_protocols = all # add IPv6 networks to mynetworks mynetworks = 127.0.0.0/8 192.168.1.0/24 [::1]/128 [fe80::]/10 [2001:470:67:84::]/64
Then make sure you have an MX record that leads to a AAAA record, do a quick “server postfix reload”, and you’re good to go.
This wraps us the series on my home IPv6 network. There will continue to be IPv6-related posts, and I’ll be writing about our work IPv6 network beginning in mid January.
# ip6 -a # sysctl -w net.inet6.ip6.accept_rtadv=1
The first command was mentioned on a blog post as needed to “fully enable IPv6 features, beyond what is enabled via the Control Panel”. The second command enables the acceptance of IPv6 Router Advertisements.
This turns out to NOT be be needed at all. I did a complete new Snow Leopard install from the DVD this evening on a spare MacBook Pro, and everything IPv6 worked perfectly, out of the box. IPv6 was enabled by default, and fast visits to test-ipv6.com and ipv6-test.com showed full native IPv6 connectivity.
I can only surmise that somewhere along the way, my regular MacBook Pro had had IPv6 turned off in some unusual way. Or it could be that my original MacBook Pro was originally a Leopard install, which was upgraded to Snow Leopard.
So, MacOS X Snow Leopard completely IPv6 ready, out of the box. I’ll be testing Lion in January…
- The first is to get AAAA (quad-A) records into your DNS system. At that point clients can ask for the AAAA records over IPv4 and everything will work just fine.
- The second is for you to actually serve your DNS zones over IPv6.
- The third is to get hooked into the global IPv6 DNS system, so that you (and others) can resolve your IPv6 addresses.
In this post, we will deal with the third part, ensuring that all the DNS servers needed to resolve my AAAA records are IPv6 capable. This step isn’t strictly necessary since, as I pointed out before, there’s nothing wrong with serving your AAAA records via IPv4.
First, let’s take a look at the symptoms of my problem:
$ dig -6 +short +trace ipv6.thuktun.org aaaa ;; connection timed out; no servers could be reached
What has happened here is that there is no authoritative DNS server that can be reached via IPv6. So, what’s the problem?
One thing that I’ve never mentioned is that “my” local DNS server is a hidden master. It holds all the zone files, but is not advertised. My advertised public DNS servers are elsewhere, and they pick up my zone data via AXFR whenever I make changes and they are sent a NOTIFY. So, while my local server has all the zone data, it will never be queried during a normal DNS lookup. The advertised DNS servers, the slaves, actually serve all the answers.
It turns out that there are two problems here:
- My external slave name servers aren’t IPv6 capable;
- My resolv.conf has no IPv6 name servers listed.
My external nameservers are run by a friend at his organization’s datacenter. They aren’t prepared to serve DNS over IPv6, and won’t be any time soon. The fastest way to fix this is to move my external DNS to a DNS hosting provider that is IPv6 capable. Fortunately, I can get IPv6 DNS from the same place that I get my IPv6 tunnel: Hurricane Electric.
Using their DNS slave server setup page, I can easily make Hurricane’s DNS servers be my public slave DNS servers. I do have to ensure that their DNS servers can do zone transfers from my hidden master, which is mostly handled by using the the allow-transfer statement in my Bind config files, which is left as an exercise for the student.
After setting up the slave servers at Hurricane, I can wait for them to zone transfer my data to their servers. This can take 5-10 minutes for the first transfer.
At this point, there are three completely independent sets of name servers that have correct data for my domain:
- My home hidden master
- The current advertised DNS servers at my friend’s organization
- The as-yet unadvertised DNS servers at Hurricane Electric
All that remains is to remove the advertisements for the “old” servers and to add advertisements for the new servers. This is done at your domain registrar. Fortunately, my registrar is Register4Less, and their DNS system can take IPv6 addresses. In fact, it was easier than I expected. Unlike some other registrars I’ve used in the past, at Register4Less, you enter the names of your DNS servers, not the IP addresses. Register4Less resolved the hostnames ns1-ns5.e.net and created NS records for both the IPv4 and IPv6 addresses.
There’s still one step left, and that’s to add some IPv6 name servers to my resolv.conf file. I could use the five DNS servers that Hurricane Electric has advertised, but I’ll go one step farther. I’ll use the anycast DNS server address that Hurricane gave me when I established my tunnel:
$ more /etc/resolv.conf nameserver 188.8.131.52 ;; IPv4 nameservers from my ISP nameserver 184.108.40.206 nameserver 220.127.116.11 nameserver 2001:470:20::2 ;; IPv6 anycast name server from HE.NET
And now here’s a full “trace” of IPv6 DNS resolution:
$ dig -6 +trace ipv6.thuktun.org aaaa ; <<>> DiG 9.7.3 <<>> -6 +trace ipv6.thuktun.org aaaa ;; global options: +cmd . 79981 IN NS g.root-servers.net. . 79981 IN NS j.root-servers.net. . 79981 IN NS a.root-servers.net. . 79981 IN NS i.root-servers.net. . 79981 IN NS h.root-servers.net. . 79981 IN NS l.root-servers.net. . 79981 IN NS c.root-servers.net. . 79981 IN NS b.root-servers.net. . 79981 IN NS d.root-servers.net. . 79981 IN NS m.root-servers.net. . 79981 IN NS f.root-servers.net. . 79981 IN NS e.root-servers.net. . 79981 IN NS k.root-servers.net. ;; Received 509 bytes from 2001:470:20::2#53(2001:470:20::2) in 31 ms org. 172800 IN NS b0.org.afilias-nst.org. org. 172800 IN NS a2.org.afilias-nst.info. org. 172800 IN NS d0.org.afilias-nst.org. org. 172800 IN NS c0.org.afilias-nst.info. org. 172800 IN NS a0.org.afilias-nst.info. org. 172800 IN NS b2.org.afilias-nst.org. ;; Received 436 bytes from 2001:7fe::53#53(i.root-servers.net) in 157 ms thuktun.org. 86400 IN NS ns5.he.net. thuktun.org. 86400 IN NS ns4.he.net. thuktun.org. 86400 IN NS ns2.he.net. thuktun.org. 86400 IN NS ns3.he.net. ;; Received 112 bytes from 2001:500:c::1#53(b0.org.afilias-nst.org) in 188 msipv6.thuktun.org. 28800 IN AAAA 2001:470:67:84::10 ;; Received 62 bytes from 2001:470:500::2#53(ns5.he.net) in 28 ms
And I’m done with DNS. Now I can go on to some other services, like SSH, FTP, HTTP, etc.