Pivotal Cloud Foundry (PCF) all versions
The virtual machine's (VM's) system clock is incorrect or skewed. You can run "date" on the VM to confirm its current time does not match the expected time.
The Ubuntu stemcell has a crontab entry that tries to sync its time with an NTP server by running the "ntpdate" command every 15 minutes.
Usually NTP, including the `ntpdate` command, uses the privileged (<1024) UDP port 123 as its source port for outgoing packets. Therefore, returning packets from the NTP server would come back to the UDP port 123.
This becomes a problem in a network where a firewall or other network device does not allow such packets from outside of the network to privileged ports. Since the VM cannot communicate with the NTP server, its time will not remain in sync with the NTP server and the clock will likely become skewed.
To confirm that you are experiencing this issue, please perform the following steps:
- Log in to the VM in question and become the root user. Then run the command /usr/sbin/ntpdate (the full path to the command is necessary), specifying an NTP server (for example, 0.pool.ntp.org):
# /usr/sbin/ntpdate 0.pool.ntp.org
- If you see an error message saying “no server suitable for synchronization found”, then traffic from the NTP server is not being received by the VM.
- Now, run the ntpdate command with the -u option. For example: /usr/sbin/ntpdate -u 0.pool.ntp.org.
- This will repeat the same request but use a nonprivileged port. If this works successfully, it is likely that your network does not allow traffic coming back to UDP port 123 from the NTP server that you tested.
To resolve this issue, you must change the network or firewall configurations so that they allow all traffic coming into the VM at UDP port 123.