[Freeipa-users] Kerberos Clock Skew too great

[Rakesh Rajasekharan]

Hi,

I am using a Freeipa 4.2.0 server.

I sometimes see, "clock skew too great" errors in /var/log/krb5kdc.log. And
when this happens, usually logins or new ipa-cleint-install fails.

When I checked on one of the hosts for which the clock skew was reported,

#> ntpq -p
remote refid st t when poll reach delay offset
jitter
==============================================================================
*ip-10-10-1-150.e 171.66.97.126 2 u 869 1024 377 0.448 0.047
0.142


Does the above o/p looks fine interms of the ntp sync

Whats the max sync time difference thats allowed for a client.

Thanks
Rakesh

[Jakub Hrozek]

Post by Rakesh Rajasekharan
Hi,
I am using a Freeipa 4.2.0 server.
I sometimes see, "clock skew too great" errors in /var/log/krb5kdc.log. And
when this happens, usually logins or new ipa-cleint-install fails.
When I checked on one of the hosts for which the clock skew was reported,
#> ntpq -p
remote refid st t when poll reach delay offset
jitter
==============================================================================
*ip-10-10-1-150.e 171.66.97.126 2 u 869 1024 377 0.448 0.047
0.142

In general, 5 minutes is OK at least. But are you sure the server is also
in sync or just the client against an NTP server (iow, are you sure you
are checking the difference between a client and the KDC as well?)

[Rakesh Rajasekharan]

yes on the IPA server as well.. the offset isn't that high

remote refid st t when poll reach delay offset
jitter
==============================================================================
*ip-10-10-1-150.e 132.163.4.101 2 u 119 128 377 0.431 -0.279
0.348

So, my NTP server, the ipa client and the IPA master.. all seems to not
have a high offset or a jitter.

There were about 1500 hosts that were alerting for "clock skew" and the
issue went away only after I did a resync using ntpdate on all those hosts

Is it possible that so many higher number of minor offsets adds up and
causes it. Coz from the individual offset it looks much below the 5min limit

Or, is there a way to tell whats the offset limit its actually looking for.

Thanks,
Rakesh

Post by Rakesh Rajasekharan

Post by Rakesh Rajasekharan
Hi,
I am using a Freeipa 4.2.0 server.
I sometimes see, "clock skew too great" errors in /var/log/krb5kdc.log.

And

Post by Rakesh Rajasekharan
when this happens, usually logins or new ipa-cleint-install fails.
When I checked on one of the hosts for which the clock skew was reported,
#> ntpq -p
remote refid st t when poll reach delay offset
jitter
============================================================

==================

Post by Rakesh Rajasekharan
*ip-10-10-1-150.e 171.66.97.126 2 u 869 1024 377 0.448 0.047
0.142

In general, 5 minutes is OK at least. But are you sure the server is also
in sync or just the client against an NTP server (iow, are you sure you
are checking the difference between a client and the KDC as well?)
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Jakub Hrozek]

Post by Rakesh Rajasekharan
yes on the IPA server as well.. the offset isn't that high
remote refid st t when poll reach delay offset
jitter
==============================================================================
*ip-10-10-1-150.e 132.163.4.101 2 u 119 128 377 0.431 -0.279
0.348
So, my NTP server, the ipa client and the IPA master.. all seems to not
have a high offset or a jitter.
There were about 1500 hosts that were alerting for "clock skew" and the
issue went away only after I did a resync using ntpdate on all those hosts
Is it possible that so many higher number of minor offsets adds up and
causes it. Coz from the individual offset it looks much below the 5min limit
Or, is there a way to tell whats the offset limit its actually looking for.

Sorry, I'm a bit out of my depth here, the only other suggestion I have
is to try kinit with KRB5_TRACE=/dev/stderr when that happens, which
should at least dump which KDC is the client talking to (if you have
multiple masters..)

Post by Rakesh Rajasekharan
Thanks,
Rakesh

Post by Rakesh Rajasekharan

Post by Rakesh Rajasekharan
Hi,
I am using a Freeipa 4.2.0 server.
I sometimes see, "clock skew too great" errors in /var/log/krb5kdc.log.

And

Post by Rakesh Rajasekharan
when this happens, usually logins or new ipa-cleint-install fails.
When I checked on one of the hosts for which the clock skew was reported,
#> ntpq -p
remote refid st t when poll reach delay offset
jitter
============================================================

==================

Post by Rakesh Rajasekharan
*ip-10-10-1-150.e 171.66.97.126 2 u 869 1024 377 0.448 0.047
0.142

In general, 5 minutes is OK at least. But are you sure the server is also
in sync or just the client against an NTP server (iow, are you sure you
are checking the difference between a client and the KDC as well?)
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Robbie Harwood]

Post by Rakesh Rajasekharan
There were about 1500 hosts that were alerting for "clock skew" and the
issue went away only after I did a resync using ntpdate on all those hosts

Great, glad it's fixed! Are these VMs? If not, you may wish to
(re?)configure automatic syncing.

Post by Rakesh Rajasekharan
Is it possible that so many higher number of minor offsets adds up and
causes it. Coz from the individual offset it looks much below the 5min limit

Not as such, if I understand you correctly? This should only be a
problem between any two machines that need to communicate (including the
freeipa KDC).

Post by Rakesh Rajasekharan
Or, is there a way to tell whats the offset limit its actually looking for.

5 minutes almost certainly. The parameter to configure it is
"clockskew" in the config files, but I don't think IPA touches that.

Hope that helps,
--Robbie

[Rakesh Rajasekharan]

Hi There,

Sorry could not get back on this earlier,

Post by Robbie Harwood
Great, glad it's fixed! Are these VMs? If not, you may wish to
(re?)configure automatic syncing.

yes these are AWS instances. How do I reconfigure auto syncing . Is there
a documentation I can follow.
Sorry, haven't done this before and not much info on that part


Apart from this , I also have a correlation between the "Clock skew" issue
and an earlier issue that I posted in another thread.
Basically , noticed that whenver I see clock skew errors, I see a lot of
connections in SYNC_RECV state.

this is the list of SYNC_RECV connections

tcp 0 0 10.0.8.45:88 10.0.30.49:42695 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.15.72:44991 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.2.82:53265 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.31.253:57682 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.34.208:53488 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.27.17:47245 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.17.53:54504 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.24.78:47796 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.4.246:33607 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.27.91:34190 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.27.248:38012 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.15.139:51319 SYN_RECV
tcp 0 0 10.0.8.45:88 10.0.15.175:41188 SYN_RECV


Thanks,
Rakesh

Post by Robbie Harwood

Post by Rakesh Rajasekharan
There were about 1500 hosts that were alerting for "clock skew" and the
issue went away only after I did a resync using ntpdate on all those

hosts
Great, glad it's fixed! Are these VMs? If not, you may wish to
(re?)configure automatic syncing.

Post by Rakesh Rajasekharan
Is it possible that so many higher number of minor offsets adds up and
causes it. Coz from the individual offset it looks much below the 5min

limit
Not as such, if I understand you correctly? This should only be a
problem between any two machines that need to communicate (including the
freeipa KDC).

Post by Rakesh Rajasekharan
Or, is there a way to tell whats the offset limit its actually looking

for.
5 minutes almost certainly. The parameter to configure it is
"clockskew" in the config files, but I don't think IPA touches that.
Hope that helps,
--Robbie

[Robbie Harwood]

Post by Robbie Harwood
Great, glad it's fixed! Are these VMs? If not, you may wish to
(re?)configure automatic syncing.

yes these are AWS instances. How do I reconfigure auto syncing . Is
there a documentation I can follow.

During install of the IPA server, it will set up an NTP server (unless
you ask it not to). During enrollment of each IPA client, it will
configure NTP against that server (unless you ask it not to). Disabling
it is the -N flag in both cases.

[Rakesh Rajasekharan]

thanks for the inputs..


one more question I was curious is.. when does the krb5kdc.log get entries
. .. I mean is it only when someone makes an attempt to login to a server
that the log file krb5kdc.log on the IPA master gets updated or there are
other scenarios as well

Thanks
Rakesh

Post by Robbie Harwood

Post by Robbie Harwood
Great, glad it's fixed! Are these VMs? If not, you may wish to
(re?)configure automatic syncing.

yes these are AWS instances. How do I reconfigure auto syncing . Is
there a documentation I can follow.

During install of the IPA server, it will set up an NTP server (unless
you ask it not to). During enrollment of each IPA client, it will
configure NTP against that server (unless you ask it not to). Disabling
it is the -N flag in both cases.

[Robbie Harwood]

Post by Rakesh Rajasekharan
one more question I was curious is.. when does the krb5kdc.log get entries
. .. I mean is it only when someone makes an attempt to login to a server
that the log file krb5kdc.log on the IPA master gets updated or there are
other scenarios as well

It's controlled by /etc/kdc.conf ; take a look at the "[logging]" section in
`man 5 kdc.conf` for more information.

[Rakesh Rajasekharan]

I was seeing a lot of entries in the krb5kdc.log like below

"krb5kdc[10403](info): TGS_REQ (4 etypes {18 17 16 23}) 10.1.4.219: ISSUE:
authtime 1485450918, etypes {rep=18 tkt=18 ses=18}, host/my-***@MYDOMAIN"

On one env.. where users rarely log in... even there I see a lot of such
requests.


Finally , I think I was able to track this down.. there are few local
accounts ( non freeipa ) on my hosts . These are used to run some custom
scripts through cron and run frequently ( every few mins ).
So, I feel whenever thers a request for "su - <localuser>" or a sudo to
the local user, that would also end up calling the Kerbros service.. and
since it runs so frequently on all the hosts.. they would be choking the
IPA master / replica with so many requests..

Please correct me If I am wrong in the above assumption.

Going by the above logic.. I have added filter_users section with these
users in the sssd.conf . Hopefully I would see a drop in the number of
requests

Post by Rakesh Rajasekharan

Post by Rakesh Rajasekharan
one more question I was curious is.. when does the krb5kdc.log get

entries

Post by Rakesh Rajasekharan
. .. I mean is it only when someone makes an attempt to login to a server
that the log file krb5kdc.log on the IPA master gets updated or there

are

Post by Rakesh Rajasekharan
other scenarios as well

It's controlled by /etc/kdc.conf ; take a look at the "[logging]" section in
`man 5 kdc.conf` for more information.