Having accurate time on your entire technology stack is critical, yet so often is misconfigured, or simply forgotten about.

Maybe time keeping is critical to your business' core operation.

But equally likely, you're experiencing clock drift on a SQL database server. Or some of your machines on your Active Directory domain are losing their trust relationships. If things get really bad, maybe you're getting certificate errors.

Or simply you're here because that one guy keeps showing up 3 minutes early to Teams calls – who swears he didn't start it early - and it's driving you mad...

I'll share with you today, in my opinion, the best way to implement timekeeping infrastructure for a typical business.

Where to get time?

The single most important element to any timekeeping system is the actual source of time. The clock that all the other clocks synchronise with. The source of truth. In computer time keeping speak, that would be the Stratum 1 source.

The device on the network that syncs with an external - non-computer - time source, like GPS. (The very same GPS that you use for your satnav).

This time is then provided to all other devices over a protocol called NTP (Network Time Protocol) or in really critical nano-second sensitive implementations, PTP (Precision Time Protocol) - (note: this article is about NTP specifically and won't cover PTP).

For some basic environments, like at home, the ntp.org volunteer servers will do just fine - https://support.ntp.org/Servers (just be sure to pick a Stratum 1 source).

However, my view is that time is such a crucial part of effective business infrastructure, that for any serious infrastructure project, I'll specify a GPS/GNSS time Stratum 1 time source. It's cost effective to implement and really only requires a single product to purchase.

Why do this? You can get super accurate, ultra-reliable time - that doesn't rely on volunteer projects or internet connectivity to function. You also aren't placing part of your core infrastructure into the hands of someone you don't control or have a business relationship with.

Personally, I really like TimeTools Limited's NTP servers.

I have no relationship with TimeTools nor am I getting any form of kickback for including them here. The links to their products in this article are NOT affiliate links. When I reference a specific product, it's because I think it's genuinely great. Not because of some incentive, financial or otherwise.

They are a UK based manufacturer of timekeeping devices, who've supplied to the best in the business. I've used their kit on a few different projects. They supply everything you need to get going in the box: antenna & mounting bracket, a decent length of GNSS antenna cable, and the device itself.

Their 1U rack mount systems - the TimeTools T300 and T550 - are ultra-reliable and easy to configure. The T550 is extra cool because it'll sync with more GNSS (Global Navigation Satellite Systems) than just the USA's GPS (yes there are others, the EU, China and Russia all have their own respective GNSS).

They even have an analogue wall clock that syncs with NTP and is powered by Power-over-Ethernet (PoE) (which if nothing else is an interesting conversation starter, yes I do have conversations about this in real life!).

The only real consideration is that you need somewhere to mount the antenna, and you need to run the antenna cable.

TimeTools claim that their antennas are good enough to get GPS/GNSS lock indoors in some scenarios, but I've never tried that. In my experience, they need to be mounted outside with clear view of the horizon.

Normally this means installing the antenna on the outside wall of the building with the dome peeking just over the edge of the roof. Like I say, the antenna mount, suitable for outdoor install, comes in the box.

You then need to run the cable to your rack where you plan to install the device. It's a pretty decent cable provided, it's not exactly fragile - but you'll want to be sure - especially if you're using really long runs - that you're keeping it away from any particularly noisy electrical sources.

When you first boot the TimeTools T300 or T550, you'll see on the display it gives you the devices IP address (from DHCP) and the status of the GNSS signal lock. Don't panic - it can take 15 minutes to get GNSS lock on boot up.

So you've got your source of truth. Now what? Networking and DNS.

First things first, you need to get your NTP server booted up and connected to your network. You need to put it on a fixed IP, where any device on the network can reach it over port UDP/123. The device needs a fixed IP. So you need to do a DHCP reservation, or really for something so critical, I would suggest setting a static IP on the device.

You might need to create a firewall rule. Lets say, we've got the device online, on a static IP address 10.182.10.123, IP subnet 10.182.10.0/24. You therefore need an ALLOW rule on any firewalls in the environment permitting communication on 10.182.10.123 UDP/123, and you need to make sure all of your devices at all sites know where to find the 10.182.10.0/24 IP subnet.

Next, because easily understood DNS (with PTR records... that's a topic for another time) makes for a more supportable environment, let's give it a DNS name.

Firstly, we need an A record with the hostname as per your environments naming standard. For the example, lets call this IOP-NTP01 (if you're curious, that stands for infrastructure, on-premise, physical, separator, network time protocol, and a tie-breaker).

Now create a second record, a CNAME, time.int.infrastructureinsider.co.uk, that points to the A record we just created. Our int.infrastructureinsider.co.uk DNS zone is our friendly, user-facing DNS space that we use for other internal web services.

This two-tier DNS record structure gives us two key benefits.

Firstly, we can easily see in DNS where our time sync is being pointed at.

Secondly, we can change the time source without having to change it on every server in the environment. We can change where the CNAME record is pointed, wait for the TTL to expire, and boom, different source.

Networking done. Now lets configure AD.

Next, Active Directory.

Time is incredibly important for Active Directory.

Yes it is necessary to have that in big font. If your time drifts by as little as 5 minutes, machines on the network will have trouble authenticating domain user logons, which means users getting issues logging onto their machines, being unable to connect to your server RDP console, problems with AD replication, amongst many other things. Basically, your life will suck a lot until you fix it.

Ancient versions of Active Directory used something called a Primary Domain Controller (PDC) as the source of time for all of the computers on the domain. These days, that function is part of the PDC Emulator FMSO role, held by one of the Active Directory Domain Controllers (AD DCs) in your environment.

Therefore, in all AD environments, you must, must, must make sure that the PDC Emulator FSMO role holder is reaching out to an external time source.

I've seen too many environments where the PDC Emulator AD DC is looking nowhere for time. It's relying on the BIOS clock.
And that is a REALLY, REALLY BAD IDEA for ANY virtual machines, let alone a PDC Emulator FSMO role holder.

How do we ensure our PDC Emulator uses our NTP source for time?

We could configure it manually, but if the PDC Emulator FSMO role holder changes to another AD DC, we could be in for an unscheduled very bad time.

So, create a GPO to synchronise the PDC Emulator specifically to our time source.

So the way I like to do this, is with a targeted Group Policy object. You specify the time source, and then specify using a WMI filter to only apply to the PDC Emulator role holder.

The WMI filter required is below (you'll need to first create the WMI filter before you can create a GPO that uses it):

Select * from Win32_ComputerSystem where DomainRole = 5

Save that filter as "Domain PDC FSMO Role Holder"
Now, create a new GPO.
Call it something that follows your naming standard, in this case we'll call it

GPO_DC_PDCe_TimeSync

We'll need to attach this GPO to the default Domain Controllers container within AD.

This is a computer policy. Disable the User Policy half of the policy. We're not setting anything there.

The important part is setting the time source correctly, so lets do that. The specific GPO option is

Computer Configuration > Policies > Administrative Templates > System > Windows Time Service. Select Time Providers > Configure Windows NTP Client

and you'll want to set that to your time.int.infrastructureinsider.co.uk equivalent CNAME record, that we created earlier. We'll also need to specify the option flags for this, usually 0x8 is the correct choice, but review the GPO help displayed, it's actually pretty decent for this GP option.

The type of time source is NTP.

time.int.infrastructureinsider.co.uk,0x8

Note: Definitely do not set your time provider to literally exactly time.int.infrastructureinsider.co.uk. There's no NTP server there. It won't work. Specify YOUR time server.

Next, syncing Domain Computers with your PDC Emulator for time

Now, we need the rest of the domain environment to sync up with that. By default, on a nice vanilla AD environment, it will.

But it won't. Are there really any AD environments out there where some previous administrator hasn't already run a bunch of GPOs, which they've then removed incorrectly, leaving stale registry changes on machines?

Set it explicitly.

Create another GPO, again following your naming convention. In our example we'll call it

GPO_COM_TimeSyncToAD

COM is for computers - this is a computer policy. Disable the User Policy half of the policy. We're not setting anything there.

Again we want the exact same Group Policy option

Computer Configuration > Policies > Administrative Templates > System > Windows Time Service. Select Time Providers > Configure Windows NTP Client

Except this time we want to set to synchronise to Active Directory. How do we do that? We specify the domain name as the time source.

activedirectory.infrastructureinsider.co.uk,0x9

You'll also want to set the type to NT5DS (that's AD Domain Services)

Now because we can really fuck things up if we don't check, we need to make sure our GPO actually works.

Attach the GPO to your test computers OU and then, from a machine within that OU, run a

gpupdate /force

followed by these two

w32tm /resync
w32tm /query /status

You should see that your computer has syncronised it's time from Active Directory. If it has, awesome.

Now we need to attach the OU to our organisation's various Computer Object OUs (you probably have a servers OU that also contains computer objects at least?).

Right, that's Active Directory covered, now what? Everything else that needs to sync!

Everything else in your environment needs to sync with your NTP server.

I'm talking: desktops, servers, linux boxes, ESXI, UPSes, PDUs, alarm panels, network switches, printers, firewalls, load balancers, clocking-in machines, coffee machines... you get the point.

You'll need to consult your device's specific instructions to establish how to configure, but normally there's simply a Time option and you can specify "Use NTP Server", and specify the time.int.infrastructureinsider.co.uk equivalent DNS address (the CNAME that you made).

Debian-based Linux, such as Ubuntu

You need timesyncd installed to sync with NTP. Once installed, run the service.

sudo apt install systemd-timesyncd 
systemctl start systemd-timesyncd.service

On first run, a file is created at /etc/systemd/timesyncd.conf .Open this in your preferred editor (lets assume you're not some linux wizard who likes to use vi/vim)

sudo nano etc /etc/systemd/timesyncd.conf

Edit as follows, substituting my addresses for yours of course.

[Time]
NTP=time.int.infrastructureinsider.co.uk

And for VMware ESXi

ESXi hosts need accurate time, and so does vCenter Server Appliance (and any other VMware appliances you might have - Replication, vROps, etc.

Both ESXi & vCenter Server MUST use the exact same time source.

This is a perfect time to use a little bit of PowerCLI to configure en-masse. Here's how to do it for ESXi - you'll need to change up the server hostname for your own.

#Initialise & connect to vSphere
Import-Module VMware.PowerCLI
$adm_creds = Get-Credential
#Set-PowerCLIConfiguration -InvalidCertificateAction Ignore #Ignore vSphere certificate errors - you might need this
Connect-VIServer -Server IOV-VCA01.ad.infrastructureinsider.co.uk -Credential $adm_creds

#Specify hosts to perform action against, sort a-z
$HostsToAction = Get-VMHost | Sort-Object Name #all hosts

#Check that NTP port 123 allowed through ESXi firewall
$HostsToAction | Get-VMHostFirewallException | Where-Object {$_.Name -eq "NTP client"}

# Stop NTP service
$HostsToAction | Get-VMHostService | Where-Object {$_.key -eq "ntpd" } | Stop-VMHostService

# Remove existing NTP Server settings and Add correct NTP server
$HostsToAction | Remove-VMHostNtpServer -NtpServer '1.uk.pool.ntp.org', '2.uk.pool.ntp.org'
$HostsToAction | Add-VMHostNtpServer -NtpServer 'time.int.infrastructureinsider.co.uk'

# Start NTP Service and set to Automatic start
$HostsToAction | Get-VMHostService | Where-Object {$_.key -eq "ntpd" } | Start-VMHostService
$HostsToAction | Get-VMHostService | Where-Object {$_.key -eq "ntpd" } | Set-VMHostService -policy "automatic"

# Get the NTP Servers of ESXi Hosts
$HostsToAction | Select-Object Name, @{N="NTPServer";E={$_ |Get-VMHostNtpServer}}, @{N="ServiceRunning";E={(Get-VmHostService -VMHost $_ | Where-Object {$_.key-eq "ntpd"}).Running}}

#Set current time to match that of workstation for initial set
$HostsToAction | ForEach-Object {$t = Get-Date; (Get-View $_.ExtensionData.configManager.DateTimeSystem).UpdateDateTime((Get-Date($t.ToUniversalTime()) -format u)) }

You've done it!
You've solved the question of time.

Lets review what we did with a sweet draw.io diagram which we can stick into our organisation's documentation pack/wiki/Confluence/bathroom stall wall.

Troubleshooting

The key elements to successful NTP synchronization, if you've configured everything as per this article are:

• IOP-NTP01 (or your stratum 1 source) having network connectivity, 
• Firewall rules permitting port 123 (NTP) connectivity to IP address 10.182.10.123 (or to your stratum 1 source if external)
• time.int.infrastructureinsider.co.uk CNAME (or equivalent) configured correctly
• For domain joined machines, DNS name resolution of at least one Active Directory Domain Controller must be working correctly
• Devices configured to use time.int.infrastructureinsider.co.uk as time source
• The PDC Emulator FMSO role must be held by a domain controller that's turned on and connected to the network
• Clear view of the sky for the GNSS antenna & IOP-NTP01 Server achieving GNSS "lock" - wait 15 minutes after boot of the NTP server for that to happen.
  

Lessons Learned

What could we do to improve our deployment?

Well first off, if you really need accurate accurate time you could switch to using PTP instead of NTP. That's outside the scope of this article though. There's a whole load of additional configuration and considerations that you need for that. And not all devices support it, so you probably will need both PTP and NTP running on your network in any case.

What if we needed better resilience?

What I would recommend, is getting a second and third TimeTools 1U Rack NTP server and sticking them at another physical datacentres. Then, potentially get yourself a load balancer in your main datacentre (or even in the cloud), and have high-availability NTP (or skip the load balancer part and have time, time2 and time3 DNS entries).

Important to note, if you have more than one externally synchronised NTP server, you need at least three to know if it's working correctly. How else will you know which of the two is reporting correctly?

What if some of my devices frequently go off network - say my user's laptops? Won't they be unable to sync?

Yeah correct, they won't be able to synchronise time with the NTP server when they are off-network. If you are managing user laptops directly via Active Directory and Group Policy, doing so doesn't work well without an "always-on VPN", and this is a good example of that in practice.
If you don't have always-on VPN for your devices, end user devices should be managed via an alternative device management solution (MDM) such as InTune.
However, I can definetely appreciate that simply saying "you have done it wrong, re-do the whole thing" isn't always helpful.
If you're unable to migrate over to InTune for device management, one option you do have is to configure a fall-back NTP server, something like time.windows.com (you can specify a fall-back server policy within a GPO, so create a new GPO specific for laptops - GPO_COM_TimeSyncToAD_FallBackForLaptops , which you attach to your laptops OU, or use a WMI filter for your specific laptop models. The Microsoft server will keep your laptops in sync enough so that they can be fully corrected when back on-network.
Another solution would be to make your NTP server public, so that all devices off and on network could access it. However, there's a lot to consider there - firewalls, rate-limiting, and other security mitigations most prominently. Sticking it behind Cloudflare WAF or similar is probably a good idea. You'll also need public DNS entries, and if you're going to this extent you'll probably be looking at implimenting a 3-node load balanced NTP solution too.

Written by Ryan Drake
Infrastructure Insider - Editor-in-Chief

Copyright 2023, All Rights Reserved.

No AI writers or writing assistants here.
GPT free.