IPv6 Renumbering: A Pain in the …

If you’re following my blog you probably know that I am using IPv6 everywhere. Everything in my lab is dual-stacked if not already IPv6-only. Great so far.

A few months ago my lab moved to another ISP which required to change all IP addresses (since I don’t have PI space yet). Shit. While it was almost no problem to change the legacy IPv4 addresses (only a few NATs), it was a huge pain in the … to change the complete infrastructure with its global unicast IPv6 addresses. It turned out that changing the interface IPv6 addresses was just the first step, while many modifications at different services were the actual problem. And this was *only* my lab and not a complete company or the like.

Following is a list of changes I did for IPv6 and for legacy IP. Just an overview to get an idea of differences and stumbling blocks.

TL;DR: Changing the public IPv4 addresses was done in a few minutes, while it took me a couple of weeks until all services were running with the new IPv6 addresses likewise. It was not only changing interface IPv6 addresses, but lots of configuration details in almost all services/appliances where IPv6 GUAs were used as well.

However, please note that this is NOT a problem of IPv6 in general, but due to the fact that we can use “public” IPv6 address (GUAs) everwhere. This crap here is only needed when you’re forced to change your IPv6 prefix which you should avoid completely! If you would use public IPv4 addresses all over your network (as it was intended a couple of decades ago), you would have the same problems for IPv4, too. It’s the NAT for IPv4 that saves us from renumbering our networks when changing the ISP. I won’t start the “don’t use ULAs” discussion here for now. ;) Just the reminder: Use PI space!

You might want to look at RFC 5887 “Renumbering Still Needs Work”. While most of this RFC focuses on the renumbering of interface addresses, at least section 5.3.4 “Management Issues” discusses a few scenarios in which IPv6 addresses are used in other configuration files that pose problems. (And of course: changing an IPv6 prefix for a client-only network using SLAAC or stateful DHCPv6 is no problem at all. Probably even with cascaded routers using DHCPv6-PD. But this post is focused on servers.)

Understanding my Lab

My lab consists of many different hardware devices and virtual appliances along with open source software. A (probably not complete) list of it is: Palo Alto Networks firewall, Fortinet FortiGate firewall, Dell switch, Cisco routers and switches, Pulse Connect Secure remote access VPN, F5 BIG-IP load balancer, Cisco ESA Email Security Appliance, Perle console server, VMware ESXi Server, couple of Raspberry Pis, Ubuntu server VMs, BIND name servers (authoritative as well as recursive caching), NTP servers, syslog-ng server, RIPE Atlas probe, Postfix mail transfer agent, MRTG monitoring server, ntopng real-time network analyzer, Apache web server.

My prefix changed from 2003:51:6012::/48 to 2003:de:2016::/48. I absolutely love my new prefix since it is that easy to remember. Oh yeah. The first hextet “2003” is quite common, the second hextet “de” is just as the TLD for Germany, and the third hextet “2016” is just like the year 2016. Lovely. ♥♥♥ However, since the third hextet changed from “6012” to “2016”, some typos were inevitable. Since most (but not all) systems were dual-stacked, I almost always used an IPv4 admin connection to change the IPv6 stuff.

One positive side effect during this renumbering was that I actually enforced my host ID structure. Formerly I was quite lazy and simply increased the rightmost bits beginning with ::1, whereas I used some kind of logic within my new IPv6 prefix. (Blogpost following.)

Changes Overview

The following table summarizes the number of changes I did for both Internet protocols. In fact it is a list that compares the usage of IPv6 GUA addresses vs. IPv4 private addresses in server configurations. Some more details below.

 IPv6Legacy IP
Host/Device/VM
Interface Addresses
23x (address + gateway + DNS server)none! LOL. Everything with private addresses
Firewall: Interfaces7x2x
Firewall: Static Routes10x2x
Firewall: Host/Network Objects29x9x
Firewall: NATsNOT A SINGLE ONE! STRIKE! This is why we're all here. ;)1x outgoing, 0x incoming since objects (row above) were used
Firewall: VPNs (RA & S2S)3x10x (since most VPNs are over v4)
Firewall: Miscellaneous6x (custom reports, LLDP profile, DNS-Proxy, RDNSS)none
DNS Names AAAA and A47x23x
DNS Authoritative Zones: masters, also-notify5x4x
DNS Glue Record: ns1.weberdns.de1x1x
Reverse DNS PTR Records40x (incl. new zone for new v6 range)none (since internal RFC 1918 addresses haven't changed)
Syslog Forwarding11xnone
NTP Restricts (for access from MRTG)6xnone
SNMP read access2x1x
Pulse Connect Secure: Admin Auth Policy1x1x
Pulse RA VPN: Client Addressing1xnone
ntopng: local-networks1xnone
DNS caching BIND: allow-recursion1xnone
RIPE Atlas measurements2x1x
Postfix: relayhost, mynetworks2xnone
Cisco ESA: Host Access Table (HAT), Relaylist4xnone
MRTG: Targets w/ static IPs3xnone
FileZilla FTP Server: NAT external IPnone1x

Note the “none”s for IPv4, while for IPv6 I had to adjust several configuration files. Counting the mere numbers it’s 205x changes of IPv6 addresses vs. 56x changes of legacy IPv4 addresses. But it’s not just the numbers that count, but the dependencies:

Some Details

While the mere changes of interface IPv6 addresses of all devices was quite obvious, here are some examples where those addresses were used in other configurations that did not work until I corrected them to my new IPv6 range. Some of them came immediately into my mind, while some others did not work for weeks until I encountered some smaller errors.

Postfix

Straight forward: Replacing the old IPv6 addresses for the relayhost and mynetworks. Easy:

I tried using an FQDN for the relayhost, but it used only the v4 address of the DNS record. Since I wanted to force v6 transmission, I was stuck using the raw v6 address.

Cisco ESA Email Security Appliance

Another example which required configuration changes was my email routing on the Cisco ESA. Outgoing mails through the RELAYLIST as well as SMTP routes used the old v6 addresses. At least for the SMTP routes I was able to use FQDNs rather than IP(v6) addresses, while the tested FQDNs for the HAT didn’t work:

BIND caching DNS Server

Uh, that was a hard one. I am using a BIND server as my caching DNS server. Of course I am not allowing anyone to query it, but only my own IP ranges. Now here is the point: I totally forgot to adjust the “allow-recursion” statement AND I did not recognize it for a couple of weeks. :D Why? Because only some monitoring servers are using this server and I was not looking at them at the time. Just after I saw some errors on one of those servers, I troubleshoot the issue and finally found the error on the BIND server. Of course, in an user environment, this would have been reported within minutes. ;) This is the corrected line:

 

syslog-ng

Though there was no problem sending syslog messages to the new address, all logs were stored in newly created folders since I am using the source IP addresses for the folder generation. That is: searching for logfiles from certain devices now implies using the old and the new hierarchy of folders. Bad.

This listing shows my syslog-ng root folder. Line 16-34 are from the old prefix; starting with line 35 the new prefix kicks in:

 

Palo Alto Firewall Services

Just two examples from my Palo Alto Networks firewall in which IPv6 GUAs are used as well: email server profile (for outgoing alert emails through the Cisco ESA appliance) and the RDNSS option within the router advertisements for network interfaces:

RIPE Atlas Measurements

I am using a couple of RIPE Atlas measurements, e.g. for my authoritative DNS servers. Since the IP addresses (in this case: v6 and v4) have changed for one of those servers, some areas turned red and did never recover. It ended up starting new measurements resolving to the new addresses.

Conclusion

Long story short: Avoid renumbering! Go for PI (provider-independent) space! This is the common advise: reference1 (Ivan), reference2 (Enno), reference3 (Greg). As I showed, there are many situations where we’re using IPv6 addresses in configuration files that end up with interrupted services after renumbering. Some of them will remain faulty for a long period (such as my allow-recursion for BIND) since it’s almost impossible to prove that all your services are fully migrated to the new prefix. And this post was only about my lab and not about a historically grown company.

Scheiße 97/366” by Dennis Skley is licensed under CC BY-ND 2.0

Citing RFC 5887 again: “It should be noted that the management and administration issues (i.e., tracking down, recording, and updating all instances where addresses are stored rather than looked up dynamically) form the dominant concern of managers considering the renumbering problem. This has led many sites to continue the pre-CIDR (Classless Inter- Domain Routing) approach of using a provider-independent (PI) prefix.”

One hint at the end: Use FQDNs rather than raw addresses! This is not possible everywhere, for example, in almost all of my config files IP(v6) addresses are mandatory since the config file is NOT able to do a DNS query before it gets read out by the service itself. At least for some cases such as my MRTG monitoring server sending SNMP get message, the usage of FQDNs rather than raw IPv6 addresses is the better solution.

Cheers. ;)

Featured image “Safety In Numbers” by Craig Rodway is licensed under CC BY-NC-ND 2.0.

3 thoughts on “IPv6 Renumbering: A Pain in the …

  1. Hmm, in some recent post or article you mentioned also the risks of VPN traffic escaping unencrypted when using GUAs. Somehow I have the impression you recommend global IPv6 addresses throughout an enterprise network?
    Personally, I would use ULAs where private IPv4 is used. This creates a clear separation between internal and DMZ systems, that everybody in the organization would understand.
    NAT is another issue, but proxies, loadbalancers or ALGs should always be preferred over NAT, which sometimes creates more issues then it actually resolves in an enterprise network IMHO.

  2. I’m curious for your IPv6-only network(s) are you using any type of v4 translation such as NAT64? If so what implementation are you using?

Leave a Reply

Your email address will not be published. Required fields are marked *