[Freeipa-users] kinit admin not working anymore (LOCKED_OUT: Clients credentials have been revoked)

[Torsten Harenberg]

Dear all,

I cannot get an "admin" kerberos token anymore on our main IPA server:

[***@ipa log]# kinit admin
kinit: Clients credentials have been revoked while getting initial
credentials

Sep 03 11:02:30 ipa.pleiades.uni-wuppertal.de krb5kdc[1351](info):
AS_REQ (6 etypes {18 17 16 23 25 26}) 132.195.124.12: LOCKED_OUT:
***@PLEIADES.UNI-WUPPERTAL.DE for
krbtgt/PLEIADES.UNI-***@PLEIADES.UNI-WUPPERTAL.DE, Clients
credentials have been revoked

also login via HTTP is not possible anymore:

Sep 03 11:04:52 ipa.pleiades.uni-wuppertal.de krb5kdc[1351](info):
AS_REQ (6 etypes {18 17 16 23 25 26}) 132.195.124.12: NEEDED_PREAUTH:
HTTP/ipa.pleiades.uni-***@PLEIADES.UNI-WUPPERTAL.DE for
krbtgt/PLEIADES.UNI-***@PLEIADES.UNI-WUPPERTAL.DE, Additional
pre-authentication required
Sep 03 11:04:52 ipa.pleiades.uni-wuppertal.de krb5kdc[1351](info):
closing down fd 11
Sep 03 11:04:52 ipa.pleiades.uni-wuppertal.de krb5kdc[1351](info):
AS_REQ (6 etypes {18 17 16 23 25 26}) 132.195.124.12: ISSUE: authtime
1441271092, etypes {rep=18 tkt=18 ses=18},
HTTP/ipa.pleiades.uni-***@PLEIADES.UNI-WUPPERTAL.DE for
krbtgt/PLEIADES.UNI-***@PLEIADES.UNI-WUPPERTAL.DE
Sep 03 11:04:52 ipa.pleiades.uni-wuppertal.de krb5kdc[1351](info):
closing down fd 11
Sep 03 11:04:52 ipa.pleiades.uni-wuppertal.de krb5kdc[1351](info):
AS_REQ (6 etypes {18 17 16 23 25 26}) 132.195.124.12: LOCKED_OUT:
***@PLEIADES.UNI-WUPPERTAL.DE for
krbtgt/PLEIADES.UNI-***@PLEIADES.UNI-WUPPERTAL.DE, Clients
credentials have been revoked

while the same works on the secondary server.

I read

http://web.mit.edu/kerberos/krb5-devel/doc/admin/lockout.html

but this did not give me a clue how to get out of this.

I am pretty sure that I never entered a wrong password, but of course
someone could have tried to log in on the Web interface.

Any idea how this can be resolved?

Kind regards

Torsten
--
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<> <>
<> Dr. Torsten Harenberg ***@physik.uni-wuppertal.de <>
<> Bergische Universitaet <>
<> FB C - Physik Tel.: [phone] <>
<> Gaussstr. 20 Fax : [phone] <>
<> 42097 Wuppertal <>
<> <>
<><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><>

[Torsten Harenberg]

Sorry for self-replying, I was able to solve it by using the 2nd IPA server:

[***@ipa2 ~]# kinit admin
Password for ***@PLEIADES.UNI-WUPPERTAL.DE:
[***@ipa2 ~]# ipa user-status admin
-----------------------
Account disabled: False
-----------------------
Server: ipa.pleiades.uni-wuppertal.de
Failed logins: 0
Last successful authentication: 20150903090946Z
Last failed authentication: 20150903090808Z
Time now: 2015-09-03T09:09:47Z

Server: ipa2.pleiades.uni-wuppertal.de
Failed logins: 0
Last successful authentication: 20150903090946Z
Last failed authentication: 20150903090851Z
Time now: 2015-09-03T09:09:47Z
-------------------------------------
Anzahl der zurückgegebenen Einträge 2
-------------------------------------
[***@ipa2 ~]# ipa user-unlock admin
-----------------------------
Konto »admin« wurde entsperrt
-----------------------------
[***@ipa2 ~]#


and now it works again on the primary:

[***@ipa ~]# kinit admin
Password for ***@PLEIADES.UNI-WUPPERTAL.DE:
[***@ipa ~]# klist
Ticket cache: KEYRING:persistent:0:0
Default principal: ***@PLEIADES.UNI-WUPPERTAL.DE

Valid starting Expires Service principal
03.09.2015 11:11:07 04.09.2015 11:11:04
krbtgt/PLEIADES.UNI-***@PLEIADES.UNI-WUPPERTAL.DE
[***@ipa ~]#


(Sorry for the german messages, my working machine is set to german).


Is there any to find out why the admin user was unlocked on the primary
machine? And would it be also possible to unlock the "admin" user with
one of the accounts inside the "admins" group? I am a bit afraid that we
will lock out ourselves next time that happens.

Thanks

Torsten
--
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<> <>
<> Dr. Torsten Harenberg ***@physik.uni-wuppertal.de <>
<> Bergische Universitaet <>
<> FB C - Physik Tel.: [phone] <>
<> Gaussstr. 20 Fax : [phone] <>
<> 42097 Wuppertal <>
<> <>
<><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><>

[Janelle]

You will find, if you check in the ns-slapd "errors" log that this
server may no longer be handling replication correctly.

Look in /var/log/dirsrv/slapd-INSTANCE..../errors

Look for errors where replication is not starting correctly because of
credential problems. You may have to re-init this replica.
The reason "admin" is locked out is that something gets screwed up with
the keytab file that was originally installed (I have not found the
cause yet, only experienced the exact same thing)

Once the keytab file is messed up, others servers can't authenticate and
therefore the ADMIN account gets locked out. If you restart the server,
it will clear for a little while, but go rgiht back to being locked out.

Solution - delete the replica and recreate.

~J

Post by Torsten Harenberg
Dear all,
kinit: Clients credentials have been revoked while getting initial
credentials
credentials have been revoked
pre-authentication required
closing down fd 11
AS_REQ (6 etypes {18 17 16 23 25 26}) 132.195.124.12: ISSUE: authtime
1441271092, etypes {rep=18 tkt=18 ses=18},
closing down fd 11
credentials have been revoked
while the same works on the secondary server.
I read
http://web.mit.edu/kerberos/krb5-devel/doc/admin/lockout.html
but this did not give me a clue how to get out of this.
I am pretty sure that I never entered a wrong password, but of course
someone could have tried to log in on the Web interface.
Any idea how this can be resolved?
Kind regards
Torsten

[Rob Crittenden]

Post by Janelle
You will find, if you check in the ns-slapd "errors" log that this
server may no longer be handling replication correctly.
Look in /var/log/dirsrv/slapd-INSTANCE..../errors

This probably doesn't have anything to do with replication. Lockout is
per-master because failed (and successful) logins are not replicated due
to the performance issues that would bring. Image 500 people all logging
in at the same time in the morning how busy all the masters would be
replicating the successes and failures.

So this is a perfectly reasonable scenario where on one master the admin
has violated the password lockout policy and is locked out but can still
log in to other masters.

ipa user-status <user> will show the lockout attributes by master. And
ipa user-unlock <user> will unlock them.

rob

Post by Janelle
Look for errors where replication is not starting correctly because of
credential problems. You may have to re-init this replica.
The reason "admin" is locked out is that something gets screwed up with
the keytab file that was originally installed (I have not found the
cause yet, only experienced the exact same thing)
Once the keytab file is messed up, others servers can't authenticate and
therefore the ADMIN account gets locked out. If you restart the server,
it will clear for a little while, but go rgiht back to being locked out.
Solution - delete the replica and recreate.
~J

Post by Torsten Harenberg
Dear all,
kinit: Clients credentials have been revoked while getting initial
credentials
credentials have been revoked
pre-authentication required
closing down fd 11
AS_REQ (6 etypes {18 17 16 23 25 26}) 132.195.124.12: ISSUE: authtime
1441271092, etypes {rep=18 tkt=18 ses=18},
closing down fd 11
credentials have been revoked
while the same works on the secondary server.
I read
http://web.mit.edu/kerberos/krb5-devel/doc/admin/lockout.html
but this did not give me a clue how to get out of this.
I am pretty sure that I never entered a wrong password, but of course
someone could have tried to log in on the Web interface.
Any idea how this can be resolved?
Kind regards
Torsten

[Janelle]

Sorry Rob - I beg to differ here. I can replicate this with my replica
failures. It happens that a replica simply loses it's mind. Somehow the
keytab gets mucked up and further connections for replication fail -- it
shows a failed "admin" login and they add up because the other servers
continue. It only happens on the failed replica -- AND I am the only one
with the admin PW and there are ZERO failed attempts over SSH.

As soon as I get another failed replica in this state (about once every
2-3 weeks) I will post the logs and open a ticket. On one server, I
simply did a reboot, and when it came back, the keytab was wrong and the
replica now claimed that it was no longer a member of the replica list.
Let me get more information and logs to open a ticket.

~J

Post by Rob Crittenden

Post by Janelle
You will find, if you check in the ns-slapd "errors" log that this
server may no longer be handling replication correctly.
Look in /var/log/dirsrv/slapd-INSTANCE..../errors

This probably doesn't have anything to do with replication. Lockout is
per-master because failed (and successful) logins are not replicated
due to the performance issues that would bring. Image 500 people all
logging in at the same time in the morning how busy all the masters
would be replicating the successes and failures.
So this is a perfectly reasonable scenario where on one master the
admin has violated the password lockout policy and is locked out but
can still log in to other masters.
ipa user-status <user> will show the lockout attributes by master. And
ipa user-unlock <user> will unlock them.
rob

Post by Janelle
Look for errors where replication is not starting correctly because of
credential problems. You may have to re-init this replica.
The reason "admin" is locked out is that something gets screwed up with
the keytab file that was originally installed (I have not found the
cause yet, only experienced the exact same thing)
Once the keytab file is messed up, others servers can't authenticate and
therefore the ADMIN account gets locked out. If you restart the server,
it will clear for a little while, but go rgiht back to being locked out.
Solution - delete the replica and recreate.
~J

Post by Torsten Harenberg
Dear all,
kinit: Clients credentials have been revoked while getting initial
credentials
credentials have been revoked
pre-authentication required
closing down fd 11
AS_REQ (6 etypes {18 17 16 23 25 26}) 132.195.124.12: ISSUE: authtime
1441271092, etypes {rep=18 tkt=18 ses=18},
closing down fd 11
credentials have been revoked
while the same works on the secondary server.
I read
http://web.mit.edu/kerberos/krb5-devel/doc/admin/lockout.html
but this did not give me a clue how to get out of this.
I am pretty sure that I never entered a wrong password, but of course
someone could have tried to log in on the Web interface.
Any idea how this can be resolved?
Kind regards
Torsten

[Torsten Harenberg]

Janelle,

Post by Janelle
As soon as I get another failed replica in this state (about once every
2-3 weeks) I will post the logs and open a ticket. On one server, I
simply did a reboot, and when it came back, the keytab was wrong and the
replica now claimed that it was no longer a member of the replica list.
Let me get more information and logs to open a ticket.

May I ask you to post a link to the ticket here once it's open? I am
really intereted to follow this issue.

Besides only two people having the password here, we have a two-factor
authentication on ssh, so there shouldn't be login failures via ssh to
valid accounts. I posted my "ipa user-show" output earlier.

But we run IPA to authenticate users to a compute cluster of about 3000
job slots, so there are in fact a lot of ssh connections to be handled.
And if a flood of jobs is started more or less at the same time, these
ssh connections will spread out in parallel. So that could match what
Rob was saying.

Hope we can find out at the end what is really causing this..

Best regards

Torsten
--
<><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><><>
<> <>
<> Dr. Torsten Harenberg ***@physik.uni-wuppertal.de <>
<> Bergische Universitaet <>
<> FB C - Physik Tel.: [phone] <>
<> Gaussstr. 20 Fax : [phone] <>
<> 42097 Wuppertal <>
<> <>
<><><><><><><>< Of course it runs NetBSD http://www.netbsd.org ><>