Why should I run own NTP Servers?

… since we all can use pool.ntp.org ? Easy answer: Many modern (security) techniques rely on accurate time. Certificate validation, two-factor authentication, backup auto-deletion, logs generation, and many more. Meanwhile we use an unauthenticated protocol (via stateless UDP) from unauthenticated sources (NTP pool) to rely on! Really?

TL;DR: If you want to operate a secure environment you should use your own on-site stratum 1 NTP servers along with authentication. This is the only way to eliminate time spoofing attacks from the outside. Don’t reduce your overall security to a stateless and unauthenticated (read: easy-to-spoof) network protocol!

If you are using couple of different NTP sources it might be not that easy for an attacker to spoof your time – though not unfeasible at all. And think about small routers with VPN endpoints and DNSSEC resolving enabled, or IoT devices such as cameras or door openers – they don’t even have a real-time clock with battery inside. They fully rely on NTP.

This is what this blogpost series is all about. Let’s dig into it. ;)

Why am I starting a long series about NTP? Because I have many customers and colleagues that are way too lazy when it comes to NTP. “It’s working with some servers on the Internet, so why should I care?”

Why NTP matters

Because almost everything in secure environments relates to timestamps. If the time isn’t contributed precisely all over your enterprise you will run into different kind of errors or security failures or troubleshooting burdens. Do not miss NTP as it might be the weakest link in your overall security strategy.

I basically want to cite this mail from the NANOG (North American Network Operators Group) mailing list:

“How about you set the time on your server ahead by 5 years. Got any idea what would happen?

  • Most of your passwords would expire.
  • All your SSL certs would expire.
  • All your TOTPs, like Google Authenticator would fail.
  • All your IPSEC tunnels would drop, and refuse to restart.
  • Many of your cron jobs would got nuts, possibly deleting all your logs.
  • Much of your DNSSEC would expire.
  • Many of your backups would be deleted since they ‘expired’.

Until recently, setting your iPhone to 1 Jan 1970 would brick it. I’m sure there are many more examples, but likely you can no longer log in, via SSH or HTTPS, and your iPhone is dead. I think any of those would qualify as more than an annoyance.”

Think about it. 😕

Want some more resources? Have a look at this paper: Bypassing HTTP Strict Transport Security [PDF] which almost only leverages a man-in-the-middle tool for NTP. Uh. Or this one: “The most common cause for secure connection errors turns out to be user systems having the wrong time“, Panos Astithas at Feeling safer online with Firefox.

And something from Geoff Huston: “What can we say about time and the Internet? The safest assumption is that most systems will be in sync with a UTC reference clock source as long as the definition of “in sync” is a window of 24 hours! If the ‘correct’ behaviour of an Internet application relies on a tighter level of time convergence with UTC time than this rather large window, then it’s likely that a set of clients will fall outside of the application’s view of what’s acceptable“, at APNIC Labs – What’s the Time?.

And what about the Internet of Things? IoT devices merely rely on NTP since they don’t have a built-in battery-powered real-time clock. That is: After booting they don’t have a time at all until they get one over NTP. Spoofing the NTP servers or packets would immediately mess up their time.

And one more example out of a networker’s daily business: Taking multipoint packet captures without synchronized times is a nightmare! You have to sync the time on your capture devices to have a chance to find your relevant packets all over your traces.

What to do? Usage of own Servers & Authentication!

You should use three on-site NTP servers as well as NTP authentication. That is:

  1. Use your own NTP servers inside your own network with different external clock sources. Advantages:
    • You have your own stratum 1 NTP servers that are independent of other NTP instances on the Internet. Unknown NTP servers might be out of sync or not reachable over time.
    • Using own antennas for external sources omits the dependency on single third parties. You should use at least two different external source such as DCF77 and GPS. That is: You should not only rely on the time codes of a single foreign country.
    • You can rely on your own hardware and their validity because it is not easy to alter them at all (as long as they reside in your own physically secure data center). You can uniformly control your servers, such as inserting leap seconds.
    • Your whole company gets almost exactly the same time since they’re using the same NTP servers throughout your enterprise which is not the case if every single device uses the NTP pool on its own which results in using hundreds of different NTP servers.
    • All NTP connections are routed within your own network, hence no one on the Internet can spoof your packets. Furthermore, you have low network latency and jitter compared to routed packets through the Internet.
    • Your own devices can alert you in case of failures, e.g. by mail, SNMP, syslog.
  2. Use three independent NTP servers. Why? Because of redundancy. One server might fail, hence you need at least two. However, if one out of two fails (that is: delivers a falsified timestamp) your clients don’t know which of those two sources to trust. Hence you should use at least three NTP servers. If one out of three fails, your clients are still able to get the correct time, since two correct sources win against one faulty. For further redundancy you should position them in different physical locations as well as logical IP subnets.
  3. Use NTP authentication on your NTP servers and clients. With an authentication mechanism leveraging hash functions such as SHA-1, your NTP clients can validate that they are talking to the correct source and are not spoofed by an attacker using a man-in-the-middle (MITM) attack. NTP authentication works with symmetric pre-shared keys. You need to generate different keys (at least per security zone, better per device) on the server in order to configure the clients appropriately. If one key gets lost, you can replace it while not loosing the security of all other keys/devices.
Using unknown NTP sources on the Internet opens at least two attack vectors: 1) The NTP source itself could deliver invalid timestamps while 2) the UDP packets could be spoofed on the way through the Internet. Using your own servers with NTP authentication thwarts both vectors.

Blogpost Series w/ Raspberry Pis and Meinberg LANTIME

J30A2051.jpg” by Furcifer pardalis is licensed under CC BY-NC 2.0

For this blogpost series I primarily used a couple of Raspberry Pis. Simply because they are cheap, have small power consumption, and you can easily use the IO ports to connect other devices such as a GPS or DCF77 receiver modules. Works fine. However: You should NOT rely on those Pis in your production environment. Why? Because they don’t have a real-time clock which keeps the own time accurate. Hence they merely rely on their stratum 0 input. If this signal is lost, you will drift in time. Furthermore, they are not stable at all, hard to update (as you will see during my series) and everything needs to be configured by hand. Hence: Only use dedicated NTP appliances that are made to serve precise timestamps all over your network. For my posts I am additionally using a Meinberg LANTIME M-200 device which I am comparing to the Pis at some points.

This blogpost series is split into three parts:

  1. Building stratum 1 NTP servers: How to use Raspberry Pis for NTP servers, advantages of purpose-built NTP appliances, load-balancing NTP, and so on.
  2. Using NTP authentication: Securing NTP packets with symmetric keys on several devices such as firewalls.
  3. Monitoring NTP servers: Getting telemetry data and stats from NTP servers.

All posts are related to my three IPv6-only stratum 1 NTP servers:

  • ntp1.weberlab.de, a Raspberry Pi with a DCF-77 receiver,
  • ntp2.weberlab.de, a Raspberry Pi with a GPS receiver and
  • ntp3.weberlab.de, a Meinberg LANTIME M200 NTP Server with DCF77 and phase modulated PRNG.

Feel free to use them via ntp.weberlab.de only. This FQDN will list only the IP addresses of those servers that are available. Of course you should only use them for test purposes, NOT for your enterprise, as this article is all about running your own servers. ;)

Throughout this series I am only covering NTP but not PTP (Precision Time Protocol). Furthermore, I am not attacking NTP via MITM attacks (at least not now) but only show how to prevent them with NTP authentication, nor do I attack the external time sources DCF77 or GPS. And I am not diving into DDoS attacks that use NTP servers (reflection & amplification attack), since I assume that you’ll primarily use your servers only within your own network and not on the public Internet.

Further Reading

As always you can read even more. ;) If you’re interested in the deep details about the network time protocol itself you should have a look at the Internet Protocol Journal volume 15, number 4 [PDF] and its article “Protocol Basics: The Network Time Protocol” from Geoff Huston. You should also have a look at this Recent NTP papers website with some really deep dive papers about NTP. Wow. For example this one: Challenges in Time Transfer using the Network Time Protocol (NTP).

Anyway, here are some more links:

As well as security related stuff:

Disclaimer

I am not set against the NTP pool project in general. It is a great community driven project that I am using client-side (for many small devices such as routers, Raspberry Pis, IoT, …) as well as server-side since I am contributing with at least one online NTP server. However, my focus with this blogpost series is the enterprise network. And you should not use the NTP pool within your enterprise as at all, but own independent stratum 1 NTP appliances along with NTP authentication.

Featured image “Breitling Super Avenger” by W.A.Smith is licensed under CC BY-NC-ND 2.0.

3 thoughts on “Why should I run own NTP Servers?

  1. Love it, but additional benefit you didn’t mention: combine your own NTP with a gateway NTP hijack rule so internal (compromised) devices could never participate in an outbound NTP reflection attack.

    Both device types benefit:

    1) Clean devices benefit from getting a *local* answer even if they try to get NTP update from time.apple.com, for example

    2) Infected devices benefit the network owners in being unable to launch NTP reflection attacks.

    Johannes, I would like to add that the same principles apply to any other UDP-based unauthenticated protocol, including DNS. Offer it locally, hijack/redirect port 53 and everybody is happier and safer.

  2. Hallo Johannes,

    in diesem Artikel erklärst du aus nachvollziehbaren Gründen, warum man 3 NTP-Server betreiben sollte.

    Nehmen wir an ein Angreifer kann das DCF-77 Signal verfälschen (da gabs
    sogar mal ein Raspberry-PI Projekt dafür, wie man das nur mit einem
    Raspberry alleine bewerkstelligen soll, in meinem Versuch hatte es nicht
    geklappt) und 2 eigene NTPs basieren auf DCF-77. Die beiden Server mit
    der vermeintlich richtigen Zeit überstimmen den Server mit der richtigen
    GPS-Zeit.

    Umgekehrt bei 2 GPS-Empfängern und Versuche diese zu manipulieren. Auch
    GPS-Signale lassen sich fälschen, wie dieses Video
    https://www.youtube.com/watch?v=VqLSKPYKeXU eindrucksvoll beweist.

    Selbst wenn man nur 2 Server hätte, wird es dann gefährlich, wenn ein
    Angreifer sowohl GPS-, als auch DCF-77 Zeit falsch aussendet.

    Um vor diesem Angriff wirklich sicher zu sein, sehe ich spontan 2 Maßnahmen.

    1. Alle größeren Zeitsprünge in der Zeitquelle werden durch den
    NTP-Server verworfen, bis zum manuellen Eingriff wird lediglich die
    interne Uhr als NTP-Quelle verwendet

    2. Zur Verhinderung einer zu langsamen oder schnellen externen GPS oder
    DCF-77 Empfangszeit: Implementation eines Vergleichsalogrithmus. Wie
    schnell laufen interne Referenzuhr und externe Uhr auseinander. Bei zu
    großer Differenz gibt es einen Alarm.

    Genereller Schutz: Ein eigener vertrauenswürdiger NTP-Server an einem
    “geheimen” Ort, der via VPN abgefragt wird und bei zu großen
    Abweichungen in der Zeit wird ebenfalls Alarm geschlagen und man muss
    manuell der Sache nachgehen.

Leave a Reply

Your email address will not be published. Required fields are marked *