Incorrect Working IPv6 NTP Clients/Networks

During my analysis of NTP and its traffic to my NTP servers listed in the NTP Pool Project I discovered many ICMP error messages coming back to my servers such as port unreachables, address unreachables, time exceeded or administratively prohibited. Strange. In summary, more than 3 % of IPv6-enabled NTP clients failed in getting answers from my servers. Let’s have a closer look:

This article is one of many blogposts within this NTP series. Please have a look!

I saw those ICMP packets in my traces for a while but did not think about it until I read this article from Heiko Gerstung: How to NOT use the NTP Pool. “[…] It sends an NTP client request, waits for a specific amount of time for the response and, if the response does not arrive within this time frame, closes the port and stops listening. If an NTP response arrives after the device stopped waiting for it, an ICMP Port Unreachable error message is sent to the sender of the NTP response, creating even more unnecessary traffic […].”

I wanted to have a deeper look into it. I captured all NTP traffic for 24 hours coming into my four NTP servers listed at the NTP Pool Project. (I was using my ProfiShark 1G in front of the FortiGate FG-100D that had those servers behind it.) I was quite astonished how many different ICMP error codes I found. Note that since I am using IPv6-only, you’ll only see ICMPv6 messages rather than legacy IP ones. Following is my analysis:

TL;DR: Within 24 hours my four NTP servers received 10923 ICMPv6 error messages from 5193 sources. This is about 3.11 % of all NTP clients. Seven different failure types were received. I did not expect so many and different error types and codes.

ICMP Errors ‘n Errors ‘n Errors

This is how my NTP tracefile looks in Wireshark. Different errors distributed all over the trace. Second screenshot filtered for “icmpv6” and a special custom column “ipv6.dst” but with 2nd field occurrence to display the original source that triggered the ICMPv6 error as my servers tried to reply:

Leveraging tshark I was getting some stats about it:

  • During that 24 h period 166471 unique source addresses queried my NTP servers
  • My servers received ICMPv6 error packets from 5193 different sources! (That is: unique routers OR end nodes.)
  • Doing the math it’s about 3.11 % of all NTP clients. Quite high to my mind.
  • In summary, they received 10923 ICMPv6 error messages.

The distribution of those ICMPv6 types and codes was:

WHAT? This is almost every destination unreachable code that is available! ;D Uhm. This was not expected.

For the sake of completeness: This is how I used tshark along with sort, uniq, etc.:

 

Why?

To my mind, this is not only related to failures in NTP clients but to different IPv6 misbehavior in general. (For further reading, have a look at one of many other articles teaching the different types in general.)

  • type 1, code 0: no route to destination: It seems to be a generic IPv6 problem based on the router to which the end-user network is attached to. Or the source address was spoofed and never valid at all. Looking into my trace: Many different sources were affected by this error while it was almost one single v6 router that sent out these errors. So maybe a single central router having routing issues?
  • type 1, code 1: communication with destination administratively prohibited: Maybe some middleboxes (routers, firewalls) that do not work stateful? Probably not an NTP client problem but an administrative network issue. Looking into my trace this is almost one single source that triggered those errors. Hence no general problem.
  • type 1, code 3: address unreachable: Again, generic IPv6 problem. For example when the layer 2 address is not resolvable (neighbor solicitation). Again, probably the original source address was spoofed, hence it did never exist. But that many? I have no idea why.
  • type 1, code 4: port unreachable: Ok, this seems to be related to bad NTP client configurations such as shown here.
  • type 1, code 5: source address failed ingress/egress policy: Why should someone block my source address?
  • type 1, code 6: reject route to destination: Found that single one in my trace: The original source for this NTP query was “fdef:ffc0:4fff:1:dd91:bb7:2a77:d84”. This was discarded by my firewall on purpose. Again an IPv6 issue since someone configured this invalid source address.
  • type 3, code 0: hop limit exceeded in transit: Again a generic IPv6 routing issue. But those affected clients can’t reach the Internet at all, can’t they?

If you want to have a look at those error messages you can download the trace file (with only the errors). 7zipped, 517 kb, 10923 packets:

(Trivia: Wireshark Filtering)

Note that I ran into some problems using Wireshark and tshark with its display filters since those are used for the ICMPv6 packets itself as well as the quoted original IPv6 packet (in my case: NTP). Complicated discussion about this on Twitter:

Some possible solutions:

Conclusion

While I “just wanted to have a quick look at the ICMP errors” it took me a couple of hours to get just a little bit of knowledge out of this trace file. Sigh. I am not sure whether all my thoughts are 100 % correct.

Anyway, there are many different scenarios that lead to failures when it comes to IPv6 enabled NTP clients. To my mind, an error rate of more than 3 % is quite bad.

Featured image “3 o’clock” by Hani Amir is licensed under CC BY-NC-ND 2.0.

One thought on “Incorrect Working IPv6 NTP Clients/Networks

  1. Some ICMP explanations.

    NTP client programs such as ntpdate or sntp, these come from the NTP reference implementation, are often the cause of ICMP port unreachable messages. By design these programs run for a few seconds or less. NTP responses arriving after the client program has terminated may result in ICMP errors.

    Organization may have more restrictive policies for incoming traffic than for outgoing traffic.

Leave a Reply to Steven Sommars Cancel reply

Your email address will not be published. Required fields are marked *