Wherever you’re running an NTP server: It is really interesting to see how many clients are using it. Either at home, in your company or worldwide at the NTP Pool Project. The problem is that ntp itself does not give you this answer of how many clients it serves. There are the “monstats” and “mrulist” queries but they are not reliable at all since they are not made for this. Hence I had to take another path in order to count NTP clients for my stratum 1 NTP servers. Let’s dig in:
Now that you have your own NTP servers up and running (such as some Raspberry Pis with external DCF77 or GPS times sources) you should monitor them appropriately, that is: at least their offset, jitter, and reach. From an operational/security perspective it is always good to have some historical graphs that show how any service behaves under normal circumstances to easily get an idea about a problem in case one occurs. With this post I am showing how to monitor your NTP servers for offset, jitter, reach, and traffic aka “NTP packets sent/received”.
During my work with a couple of NTP servers, I had many situations in which I just wanted to know whether an NTP server is up and running or not. For this purpose, I used two small Linux tools that fulfill almost the same: single CLI command while not actually updating any clock but only displaying the result. That is: ntpdate & sntp. Of course, the usage of IPv6 is mandatory as well as the possibility to test NTP authentication.
It’s not always this simple DNS thing such as “single query – single answer, both via UDP”. Sometimes you have some more options or bigger messages that look and behave differently on the network. For example: IP fragmentation for larger DNS answers that do not fit into a single UDP datagram (hopefully not after the DNS flag day 2020 anymore), or DNS via TCP, or some newer options within the EDNS space such as “EDNS Client Subnet” (ECS) or DNS cookies.
I won’t explain any details about those options, but I am publishing a pcap with that kind of packets along with some Wireshark screenshots. Feel free to dig into it.
Since my last blogposts covered many 6in4 IPv6 tunnel setups (1, 2, 3) I took a packet capture of some tunneled IPv6 sessions to get an idea how these packets look like on the wire. Feel free to download this small pcap and to have a look at it by yourself.
A couple of spontaneous challenges from the pcap round things up. ;)
Yes, I know I know, the Juniper ScreenOS devices are Out-of-Everything (OoE), but I am still using them for a couple of labs. They simply work as a router and VPN gateway as well as a port-based firewall. Perfect for labs.
For some reasons I had another lab without native IPv6 Internet. Hence I used the IPv6 Tunnel Broker one more time. Quite easy with the SSGs, since HE offers a sample config. But even through the GUI it’s just a few steps:
Of course, you should use dual-stack networks for almost everything on the Internet. Or even better: IPv6-only with DNS64/NAT64 and so on. ;) Unfortunately, still not every site has native IPv6 support. However, we can simply use the IPv6 Tunnel Broker from Hurricane Electric to overcome this time-based issue.
Well, wait… Not when using a Palo Alto Networks firewall which lacks 6in4 tunnel support. Sigh. Here’s my workaround:
For some reason, I am currently using a FortiGate on a location that has no native IPv6 support. Uh, I don’t want to talk about that. ;) However, at least the FortiGate firewalls are capable of 6in4 tunnels. Hence I am using the IPv6 Tunnel Broker from Hurricane Electric again. Quite easy so far.
But note, as always: Though FortiGate supports these IPv6 features such as a 6in4 tunnel or stateful/-less DHCPv6 server, those features are NOT stable or well designed at all. I had many bugs and outages during my last years. Having “NAT enabled” on every new IPv6 policy is ridiculous. Furthermore, having independent security policies for legacy IP and IPv6 is obviously a really bad design. One single policy responsible for both Internet protocols is a MUST. Anyway, let’s look at the 6in4 tunnel:
One of my readers sent me this question:
Let’s have a look:
Yes, ScreenOS is end-of-everything (EoE), but for historical reasons I still have some of them in my lab. ;D They simply work, while having lots of features when it comes to IPv6 such as DHCPv6-PD. However, using IPv6-only NTP servers is beyond their possibilities. :(
Anyway, I tried using NTP authentication with legacy IP. Unfortunately, I had some issues with it. Not only that they don’t support SHA-1 but MD5, this MD5 key was also limited in its length to 16 characters. Strange, since ntp-keygen per default generates 20 ASCII characters per key. Let’s have a look:
I initially wanted to show how to use NTP authentication on a Pulse Connect Secure. Unfortunately, it does not support NTP over IPv6, which is mandatory for my lab. Ok, after I calmed down a bit, a configured it with legacy IP and got NTP authentication running. ;) Here’s how:
Configuring NTP authentication on the Infoblox Grid Master is quite simple. Everything is packed inside the single “NTP Grid Config” menu. You just have to enter the NTP keys respectively key IDs and enable authentication on the appropriate servers. In the case of incorrect authentication values an error message is logged. Very good, since this is not the case on some other network security devices (Palo, Forti).
Too bad that it only supports MD5 while SHA-1 should be used instead of.
A security device such as a firewall should rely on NTP authentication to overcome NTP spoofing attacks. Therefore I am using NTP authentication on the FortiGate as well. As always, this so-called next-generation firewall has a very limited GUI while you need to configure all details through the CLI. I hate it, but that’s the way Fortinet is doing it. Furthermore the “set authentication” command is hidden unless you’re downgrading to NTPv3 (?!?) and it only supports MD5 rather than SHA-1. Not that “next-generation”!
Finally, you have no chance of knowing whether NTP authentication is working or not. I intentionally misconfigured some of my NTP keys which didn’t change anything in the NTP synchronization process while it should not work at all. Fail!
Everyone uses NTP, that’s for sure. But are you using it with authentication on your own stratum 1 servers? You should since this is the only way to provide security against spoofed NTP packets, refer to Why should I run own NTP Servers?. As always, Palo Alto has implemented this security feature in a really easy way, since it requires just a few clicks on the GUI. (Which again is much better than other solutions, e.g., FortiGate, which requires cumbersome CLI commands.) However, monitoring the NTP servers, whether authentication was successful or not, isn’t implemented in a good way. Here we go:
This is how you can use NTP authentication on Cisco IOS in order to authenticate your external NTP servers respectively their NTP packets. Though it is not able to process SHA-1 but only MD5, you’re getting an authentic NTP connection. Let’s have a look: