[Freeipa-users] Clients are reading AD info inconsistently

[Guertin, David S.]

I have three IPA servers set up (master and two replicas) and they're all behaving normally. AD users can log in, AD group restrictions are honored, etc. Now I'm trying to set up clients, and running into problems. I have three clients set up, and all three behave differently.

On one of the clients, users can log in like they can on the servers. On the other two, users can't log in, but these two behave differently from each other.

Client 1 and servers (this is correct):

# id 'MIDD\juser'
uid=435021613(***@middlebury.edu) gid=435021613(***@middlebury.edu) groups=435021613(***@middlebury.edu),435330225(computer science lab ***@middlebury.edu),435231589(***@middlebury.edu),435208664(miis labfiles ***@middlebury.edu),435032463(mcms no ***@middlebury.edu),435000513(domain ***@middlebury.edu),435286826(***@middlebury.edu),435461517(ipa ***@middlebury.edu<mailto:***@middlebury.edu>)

Client 2 (AD groups are not listed):

# id 'MIDD\juser'
uid=435021613(***@middlebury.edu) gid=435021613(***@middlebury.edu) groups=435021613(***@middlebury.edu<mailto:***@middlebury.edu>)

Client 3 (user not found):

# id 'MIDD\juser'
id: MIDD\juser: No such user

On each client I have cleared the sssd cache (rm -f /var/lib/sss/db/*) and restarted sssd, with no effect. I have also uninstalled and re-installed the client, also with no effect.

What else can I try?

David Guertin

[Dmitri Pal]

Post by Guertin, David S.
I have three IPA servers set up (master and two replicas) and they're
all behaving normally. AD users can log in, AD group restrictions are
honored, etc. Now I'm trying to set up clients, and running into
problems. I have three clients set up, and all three behave differently.
On one of the clients, users can log in like they can on the servers.
On the other two, users can't log in, but these two behave differently
from each other.
# id 'MIDD\juser'
# id 'MIDD\juser'
# id 'MIDD\juser'
id: MIDD\juser: No such user
On each client I have cleared the sssd cache (rm --f
/var/lib/sss/db/*) and restarted sssd, with no effect. I have also
uninstalled and re-installed the client, also with no effect.
What else can I try?
David Guertin

What are the platforms and package versions of SSSD on these clients?

[Guertin, David S.]

Post by Dmitri Pal
What are the platforms and package versions of SSSD on these clients?

Client 1:
RHEL 6.6
sssd-1.11.6

Client 2:
RHEL 6.6
sssd-1.11.6

Client 3:
RHEL 5.11
sssd-1.5.1

David Guertin

[Guertin, David S.]

Follow-up: today I tried clearing the sssd cache and restarting sssd on all three clients, and all three lost their AD users:

# rm -f /var/lib/sss/db/*
# service sssd restart
Stopping sssd: [ OK ]
Starting sssd: [ OK ]
# id 'MIDD\juser'
id: MIDD\juser: No such user

David Guertin

[Simo Sorce]

Post by Guertin, David S.
# rm -f /var/lib/sss/db/*
# service sssd restart
Stopping sssd: [ OK ]
Starting sssd: [ OK ]
# id 'MIDD\juser'
id: MIDD\juser: No such user
David Guertin

This is normal, users are "loaded in" when they actually try to Log In.

Simo.

[Dmitri Pal]

Post by Simo Sorce

Post by Guertin, David S.
# rm -f /var/lib/sss/db/*
# service sssd restart
Stopping sssd: [ OK ]
Starting sssd: [ OK ]
# id 'MIDD\juser'
id: MIDD\juser: No such user
David Guertin

This is normal, users are "loaded in" when they actually try to Log In.
Simo.

Yes. The ability to look up AD users that never authenticated was added
in 7.1 and 6.7 (i.e. SSSD 1.12)
--
Thank you,
Dmitri Pal

Sr. Engineering Manager IdM portfolio
Red Hat, Inc.

[Sumit Bose]

Post by Simo Sorce

Post by Guertin, David S.
# rm -f /var/lib/sss/db/*
# service sssd restart
Stopping sssd: [ OK ]
Starting sssd: [ OK ]
# id 'MIDD\juser'
id: MIDD\juser: No such user
David Guertin

This is normal, users are "loaded in" when they actually try to Log In.
Simo.

Yes. The ability to look up AD users that never authenticated was added in
7.1 and 6.7 (i.e. SSSD 1.12)

I would like to just clarify tis a bit. The support to lookup up
secondary groups (the group list the id command shows) for user which
never authenticated was added in 7.1/6.7.

The plain user lookup as e.g. done with the 'getent passwd username'
always worked.

David, the IPA clients will connect the IPA server to get the user data.
This means if the server cannot resolve the user the clients cannot
either. So the IPA server should be checked first.

You said that you have three IPA servers (master and replicas). Did you
run ipa-adtrust-install on all server? If not, please do. If you are not
sure, running ipa-adtrust-install multiple times does not so any harm.

Since you are using RHEL-6 clients I assume your IPA servers are on
RHEL-6 as well. In this case please try if the command

wbinfo -n 'MIDD\juser'

returns the SID of the user on the IPA server.

HTH

bye,
Sumit

--
Thank you,
Dmitri Pal
Sr. Engineering Manager IdM portfolio
Red Hat, Inc.
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Guertin, David S.]

I would like to just clarify tis a bit. The support to lookup up secondary groups
(the group list the id command shows) for user which never authenticated
was added in 7.1/6.7.

Thanks. This makes sense, and indeed with Client 1 I can indeed log in, and "id 'MIDD\juser'" shows all the groups again.

However, logins to Client 2 (also running RHEL 6.6 and sssd 1.11.6) still fail, and "id 'MIDD\juser'" on that client shows only local IPA groups, not AD groups.

And logins to Client 3 also fail, and "id 'MIDD\juser'" there shows "No such user". (This is the RHEL 5 box with sssd 1.5.1.) So we're back to my original problem of three clients all behaving differently.

David, the IPA clients will connect the IPA server to get the user data.
This means if the server cannot resolve the user the clients cannot either. So
the IPA server should be checked first.

All three servers can resolve the user. The user can log in to all the servers and "id 'MIDD\juser'" shows the correct AD groups.

You said that you have three IPA servers (master and replicas). Did you run
ipa-adtrust-install on all server? If not, please do. If you are not sure, running
ipa-adtrust-install multiple times does not so any harm.

Yes, the trust relationship is set up correctly on all three servers, and "ipa trust-find --all" gives identical results on all three servers, correctly showing the trust relationship with our AD domain.

Since you are using RHEL-6 clients I assume your IPA servers are on
RHEL-6 as well.

No, the servers are all running RHEL 7.1, so we're not using winbind at all -- just sssd.

The clients are a mix of RHEL 6 and RHEL 5 machines.

David Guertin

[Sumit Bose]

Post by Guertin, David S.

I would like to just clarify tis a bit. The support to lookup up secondary groups
(the group list the id command shows) for user which never authenticated
was added in 7.1/6.7.

Thanks. This makes sense, and indeed with Client 1 I can indeed log in, and "id 'MIDD\juser'" shows all the groups again.
However, logins to Client 2 (also running RHEL 6.6 and sssd 1.11.6) still fail, and "id 'MIDD\juser'" on that client shows only local IPA groups, not AD groups.

As long as the user has not logged in it is expected that the id command
doe not show the full list of groups. To see why the login fails it
would be good to know how you try to log in (I assume ssh) and which
authentication method is used (password, ssh key, Kerberos ticket).
Additionally the SSSD log files might be needed, most important here are
the logs from the PAM and PAC responders and the domain log.

Post by Guertin, David S.
And logins to Client 3 also fail, and "id 'MIDD\juser'" there shows "No such user". (This is the RHEL 5 box with sssd 1.5.1.) So we're back to my original problem of three clients all behaving differently.

For RHEL5 you need a special configuration for SSSD, call 'ipa-advise
config-redhat-sssd-before-1-9' for more details.

HTH

bye,
Sumit

Post by Guertin, David S.

David, the IPA clients will connect the IPA server to get the user data.
This means if the server cannot resolve the user the clients cannot either. So
the IPA server should be checked first.

All three servers can resolve the user. The user can log in to all the servers and "id 'MIDD\juser'" shows the correct AD groups.

You said that you have three IPA servers (master and replicas). Did you run
ipa-adtrust-install on all server? If not, please do. If you are not sure, running
ipa-adtrust-install multiple times does not so any harm.

Yes, the trust relationship is set up correctly on all three servers, and "ipa trust-find --all" gives identical results on all three servers, correctly showing the trust relationship with our AD domain.

Since you are using RHEL-6 clients I assume your IPA servers are on
RHEL-6 as well.

No, the servers are all running RHEL 7.1, so we're not using winbind at all -- just sssd.
The clients are a mix of RHEL 6 and RHEL 5 machines.
David Guertin

[Guertin, David S.]

To see why the login fails it would be good to
know how you try to log in (I assume ssh) and which authentication method
is used (password, ssh key, Kerberos ticket).
Additionally the SSSD log files might be needed, most important here are the
logs from the PAM and PAC responders and the domain log.

Yes, this is SSH. There are a few hints in the log files on the client:

sssd_ipa.middlebury.edu.log:

(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [ipa_s2n_exop_send] (0x0400): Executing extended operation
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [ipa_s2n_exop_send] (0x2000): ldap_extended_operation sent, msgid = 12
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [sdap_process_result] (0x2000): Trace: sh[0xe7f410], connected[1], ops[0xe80680], ldap[0xe641d0]
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_EXTENDED]
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [ipa_s2n_exop_done] (0x0400): ldap_extended_operation result: Protocol error(2), (null)
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [sdap_id_op_done] (0x4000): releasing operation connection
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [acctinfo_callback] (0x0100): Request processed. Returned 3,1432158221,Account info lookup failed
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [sdap_process_result] (0x2000): Trace: sh[0xe7f410], connected[1], ops[(nil)], ldap[0xe641d0]

Sssd_nss.log:

(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sss_dp_issue_request] (0x0400): Issuing request for [0x418850:1:***@middlebury.edu]
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [middlebury.edu][4097][1][name=juser]
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x6b5a10
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x418850:1:***@middlebury.edu]
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sbus_remove_timeout] (0x2000): 0x6b5a10
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sbus_dispatch] (0x4000): dbus conn: 0x6b0aa0
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 3 errno: 1432158221 error message: Account info lookup failed
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): Unable to get information from Data Provider
Error: 3, 1432158221, Account info lookup failed
Will try to return what we have in cache
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sss_dp_req_destructor] (0x0400): Deleting request: [0x418850:1:***@middlebury.edu]

I don't see any errors in sssd_pam.log, sssd_pac.log, or sssd_ssh.log.

Is this an indication that something is wrong with the trust relationship? If so, why is it happening on this client but not the other one? Any why are the servers working properly?

David Guertin

[Sumit Bose]

Post by Guertin, David S.

To see why the login fails it would be good to
know how you try to log in (I assume ssh) and which authentication method
is used (password, ssh key, Kerberos ticket).
Additionally the SSSD log files might be needed, most important here are the
logs from the PAM and PAC responders and the domain log.

(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [ipa_s2n_exop_send] (0x0400): Executing extended operation
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [ipa_s2n_exop_send] (0x2000): ldap_extended_operation sent, msgid = 12
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [sdap_process_result] (0x2000): Trace: sh[0xe7f410], connected[1], ops[0xe80680], ldap[0xe641d0]
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [sdap_process_message] (0x4000): Message type: [LDAP_RES_EXTENDED]
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [ipa_s2n_exop_done] (0x0400): ldap_extended_operation result: Protocol error(2), (null)

The most likely reason for 'Protocol error' is that the server this
client is connected to does not support the special LDAP extended
operation used by SSSD on IPA clients to get the data for users and
groups from trusted domains. And the most likely reason for this is that
ipa-adtrust-install is not run on that server. Please note that while
'ipa trust-add ...' must be only run once on one of the IPA servers,
ipa-adtrust-install must be run on all, e.g. to enable the LDAP extended
operation mentioned above.

You can check if the exop is enabled on the servers by running

ldapsearch -h localhost -x -b '' -s base|grep 2.16.840.1.113730.3.8.10.4

on each server. YOu should see 1, for RHEL-7.1 even 2 lines of output.

Post by Guertin, David S.
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [ipa_s2n_get_user_done] (0x0040): s2n exop request failed.
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [sdap_id_op_done] (0x4000): releasing operation connection
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [acctinfo_callback] (0x0100): Request processed. Returned 3,1432158221,Account info lookup failed
(Fri Mar 27 09:29:14 2015) [sssd[be[ipa.middlebury.edu]]] [sdap_process_result] (0x2000): Trace: sh[0xe7f410], connected[1], ops[(nil)], ldap[0xe641d0]
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [middlebury.edu][4097][1][name=juser]
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x6b5a10
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sbus_remove_timeout] (0x2000): 0x6b5a10
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sbus_dispatch] (0x4000): dbus conn: 0x6b0aa0
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 3 errno: 1432158221 error message: Account info lookup failed
(Fri Mar 27 09:29:14 2015) [sssd[nss]] [nss_cmd_getby_dp_callback] (0x0040): Unable to get information from Data Provider
Error: 3, 1432158221, Account info lookup failed
Will try to return what we have in cache
I don't see any errors in sssd_pam.log, sssd_pac.log, or sssd_ssh.log.
Is this an indication that something is wrong with the trust relationship? If so, why is it happening on this client but not the other one? Any why are the servers working properly?

Maybe the clients are connected to different servers and only one of
them has the exop enabled? The servers itself lookup the AD users and
groups directly from the AD DC. Since the users are available on the
server and one client is already working I expect that the trust
relationship is fine.

HTH

bye,
Sumit

Post by Guertin, David S.
David Guertin

[Guertin, David S.]

The most likely reason for 'Protocol error' is that the server this client is
connected to does not support the special LDAP extended operation used by
SSSD on IPA clients to get the data for users and groups from trusted
domains. And the most likely reason for this is that ipa-adtrust-install is not
run on that server. Please note that while 'ipa trust-add ...' must be only run
once on one of the IPA servers, ipa-adtrust-install must be run on all, e.g. to
enable the LDAP extended operation mentioned above.
You can check if the exop is enabled on the servers by running
ldapsearch -h localhost -x -b '' -s base|grep 2.16.840.1.113730.3.8.10.4
on each server. YOu should see 1, for RHEL-7.1 even 2 lines of output.

You are correct; I had not run ipa-adtrust-install on the replica servers. I have done that, and now the
ldapsearch command works correctly and the "Protocol error" statement is gone from the logs. But
there was something else going on and users still could not log in to the client.

The log files indicated that there was a permissions problem with /tmp. I changed it to root: root 777, and
now logins are working. Thanks!

David Guertin

[Sumit Bose]

Post by Guertin, David S.

The most likely reason for 'Protocol error' is that the server this client is
connected to does not support the special LDAP extended operation used by
SSSD on IPA clients to get the data for users and groups from trusted
domains. And the most likely reason for this is that ipa-adtrust-install is not
run on that server. Please note that while 'ipa trust-add ...' must be only run
once on one of the IPA servers, ipa-adtrust-install must be run on all, e.g. to
enable the LDAP extended operation mentioned above.
You can check if the exop is enabled on the servers by running
ldapsearch -h localhost -x -b '' -s base|grep 2.16.840.1.113730.3.8.10.4
on each server. YOu should see 1, for RHEL-7.1 even 2 lines of output.

You are correct; I had not run ipa-adtrust-install on the replica servers. I have done that, and now the
ldapsearch command works correctly and the "Protocol error" statement is gone from the logs. But
there was something else going on and users still could not log in to the client.
The log files indicated that there was a permissions problem with /tmp. I changed it to root: root 777, and
now logins are working. Thanks!

Thank you for the feedback. Please note that /tmp/ should be 1777
(sticky bit set) so that only owners can delete files.

bye,
Sumit

Post by Guertin, David S.
David Guertin