Sunday, July 1, 2012

Java leap second bug - how to fix your broken servers


Great way to spend a Saturday night. So much fuss about Y2K and turns out a single second can turn a simple night into chaos.


Simply put, here's a possible fix that might save the night:
/etc/init.d/ntp stop; date; date `date +"%m%d%H%M%C%y.%S"`; date

 After doing this, see if the cpu usage falls down. If not, a simple restart of the service should be enough.

Good luck


Edit: There are some advices about leaving ntp off for a while

9 comments:

  1. Thanks - we have many many servers, which have problems. Some of them related to java, but others to splunk and other processes. A service restart is not enough - a reboot solves the problem, but your fix is far better!

    ReplyDelete
    Replies
    1. We noticed that MySQL replication starts lagging as well. Same fix applies there. If typing lengthy date format is annoying - # date "`date`" - does the trick just fine.

      Delete
  2. Another Java basher, learn to have your servers updated, that includes your Java VMs. Java has a time database the same that Linux/Unix servers that need to be updated. Out if using OpenJDK it is patched to use tzdata from the OS, but still you need to restart those processes

    ReplyDelete
  3. for me it says /etc/init.d/ntp: No such file or directory can you please help?

    ReplyDelete
    Replies
    1. use: /etc/init.d/ntpd

      Great fix! Thanks!

      Delete
  4. Ravi, chances are your NTP daemon is run with ntpd, not ntp. Depends on OS and version...

    ReplyDelete
  5. Don't forget to start NTP afterwards. Mozilla's fix implies systems are running puppet that restarts it automatically.

    ReplyDelete
  6. Thanks for all the information ,it was very helpful i really like that you are providing information on core and advance java ,being enrolled in
    advance and core java http://www.wiziq.com/course/1779-core-and-advance-java-concepts i was looking for such information on advance and core java and your information helped me a lot.

    ReplyDelete