[Freeipa-users] sudo (sssd) hangs due to ipa install/uninstall scripts

[Prasun Gera]

Version: idm 4.x on rhel 7.1

Yet again, I've discovered a problem with residual state left behind by ipa
client install and uninstall scripts. I was having some trouble with
autofs+sssd leading to users not being mapped correctly (got nobody users
for everything). So I tried theipa-client-automount --uninstall, followed
by ipa-client-install --uninstall, and then did a fresh install of the
client. The original autofs issue aside, I started getting hangs in sudo.
After spending a better part of the day, the culprit was this line in
nsswitch.conf:

sudoers: files sss sss

It turns out that the extra sss left behind is sufficient to make any sudo
command hang. Easy to reproduce too.

Regarding the original autofs problem, I don't have a conclusive
explanation yet, but explicitly adding nfsvers=3 seems to map users
correctly.

It's always scary to use these install and uninstall scripts. They always
tend to leave bits and pieces behind. Isn't there a cleaner way to achieve
this ?

[Jakub Hrozek]

Post by Prasun Gera
After spending a better part of the day, the culprit was this line in
sudoers: files sss sss

Known ipa-client-install bug, fixed.

Post by Prasun Gera
It turns out that the extra sss left behind is sufficient to make any sudo
command hang. Easy to reproduce too.

Known sudo bug, fixed in upstream and making its way into downstreams:
https://bugzilla.redhat.com/show_bug.cgi?id=1133657
https://bugzilla.redhat.com/show_bug.cgi?id=1147498

[Prasun Gera]

Thanks. It's good to know that it is fixed upstream. For discussion though,
are any enhancements planned for dealing with installation/removal of ipa ?

Post by Jakub Hrozek

Post by Prasun Gera
After spending a better part of the day, the culprit was this line in
sudoers: files sss sss

Known ipa-client-install bug, fixed.

Post by Prasun Gera
It turns out that the extra sss left behind is sufficient to make any

sudo

Post by Prasun Gera
command hang. Easy to reproduce too.

https://bugzilla.redhat.com/show_bug.cgi?id=1133657
https://bugzilla.redhat.com/show_bug.cgi?id=1147498
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Jakub Hrozek]

Post by Prasun Gera
Thanks. It's good to know that it is fixed upstream. For discussion though,
are any enhancements planned for dealing with installation/removal of ipa ?

Not sure, but please file bugs as you see them.

[Dmitri Pal]

Post by Jakub Hrozek

Post by Prasun Gera
Thanks. It's good to know that it is fixed upstream. For discussion though,
are any enhancements planned for dealing with installation/removal of ipa ?

Not sure, but please file bugs as you see them.

Yes, please be more specific . The bugs that were mentioned by Jakub are
making its way into downstream. If there are any other issues you are
concerned about please let us know.
--
Thank you,
Dmitri Pal

Director of Engineering for IdM portfolio
Red Hat, Inc.

[Prasun Gera]

So I've again spent a couple of hours debugging a very similar issue.
Client install would seemingly pass, but with "Unable to find 'admin' user
with 'getent passwd ***@domain'!" at the end. And nobody would be able to
authenticate. The reason was that /etc/nsswitch.conf wasn't updated. sss
wasn't added to it. Wading through his thread
https://www.redhat.com/archives/freeipa-users/2015-March/msg00538.html
provided some hints. I have no idea why it did that, but as I have
experienced before, modifying critical system files this way from python
scripts which don't have proper transnactional support is very dangerous. I
suspect that this has something to do with a prior failed install or
uninstall attempt which left it in an inconsistent state. Is it possible to
move from this backup-modify-restore approach to critical files to
something more robust which has transnational guarantees ?

Post by Dmitri Pal

Post by Jakub Hrozek

Post by Prasun Gera
Thanks. It's good to know that it is fixed upstream. For discussion though,
are any enhancements planned for dealing with installation/removal of ipa ?

Not sure, but please file bugs as you see them.
Yes, please be more specific . The bugs that were mentioned by Jakub are

making its way into downstream. If there are any other issues you are
concerned about please let us know.
--
Thank you,
Dmitri Pal
Director of Engineering for IdM portfolio
Red Hat, Inc.
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Prasun Gera]

FYI, I think the culprit (at least one of) is ipa-client-automount
--uninstall. This removes sss entirely from nssswitch, not just from the
automount section.

Post by Prasun Gera
So I've again spent a couple of hours debugging a very similar issue.
Client install would seemingly pass, but with "Unable to find 'admin' user
to authenticate. The reason was that /etc/nsswitch.conf wasn't updated. sss
wasn't added to it. Wading through his thread
https://www.redhat.com/archives/freeipa-users/2015-March/msg00538.html
provided some hints. I have no idea why it did that, but as I have
experienced before, modifying critical system files this way from python
scripts which don't have proper transnactional support is very dangerous. I
suspect that this has something to do with a prior failed install or
uninstall attempt which left it in an inconsistent state. Is it possible to
move from this backup-modify-restore approach to critical files to
something more robust which has transnational guarantees ?

Post by Dmitri Pal

Post by Jakub Hrozek

Post by Prasun Gera
Thanks. It's good to know that it is fixed upstream. For discussion though,
are any enhancements planned for dealing with installation/removal of ipa ?

Not sure, but please file bugs as you see them.
Yes, please be more specific . The bugs that were mentioned by Jakub are

making its way into downstream. If there are any other issues you are
concerned about please let us know.
--
Thank you,
Dmitri Pal
Director of Engineering for IdM portfolio
Red Hat, Inc.
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Jakub Hrozek]

Post by Prasun Gera
FYI, I think the culprit (at least one of) is ipa-client-automount
--uninstall. This removes sss entirely from nssswitch, not just from the
automount section.

Hmm, I haven't tested that but it sounds like a bug.. I would expect
automount uninstall to touch my passwd or group database..

[Prasun Gera]

I have zero confidence in any of the install and uninstall scripts. And
this is on RHEL systems. On unofficial ones like Ubuntu, things are even
more broken. I really like freeipa, but so far even in a smallish lab
environment, it has been a nightmare. I am really tempted to just go back
to NIS. Does anyone have any ideas or proposals for making things more
robust ? At the very least, I think that these sort of modifications to
system files should only happen with package install/removal. Any changes
that ipa's scripts do should be local to ipa's internal state. Better would
be to have an internal ipa database sort of thing which keeps track of what
the current state is so that even if a script dies, which has happened
often, the next attempt reads the database and figures out what happened
earlier.

Post by Jakub Hrozek

Post by Prasun Gera
FYI, I think the culprit (at least one of) is ipa-client-automount
--uninstall. This removes sss entirely from nssswitch, not just from the
automount section.

Hmm, I haven't tested that but it sounds like a bug.. I would expect
automount uninstall to touch my passwd or group database..
--
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

[Alexander Bokovoy]

Post by Prasun Gera
I have zero confidence in any of the install and uninstall scripts. And
this is on RHEL systems. On unofficial ones like Ubuntu, things are even
more broken. I really like freeipa, but so far even in a smallish lab
environment, it has been a nightmare. I am really tempted to just go back
to NIS. Does anyone have any ideas or proposals for making things more
robust ? At the very least, I think that these sort of modifications to
system files should only happen with package install/removal. Any changes
that ipa's scripts do should be local to ipa's internal state. Better would
be to have an internal ipa database sort of thing which keeps track of what
the current state is so that even if a script dies, which has happened
often, the next attempt reads the database and figures out what happened
earlier.

File bugs with enough details. It is the only reliable way to fix any
issues where environments differ. Install/uninstall scripts work for
fresh installs in RHEL and Fedora because this is what is tested. If you
have repurposed machines from some other setups, things might differ and
only you know what is in your environment.

That's not bad or good, that's just different -- the more different
environments we see, more robust code can be added. People are
infinitely more clever than computers when it comes to configuration
files' format mangling.

I've seen multiple cases where a claim of 'ipa scripts broke my
configuration' was later retracted saying that puppet or other SCM run
afterwards did these changes. That just happen, if there are many
elephants dancing in the room, a careful coordination is always a good
idea.

Coming back to your issues, please file bugs -- either upstream or
downstream, via distributions, whatever way is more suitable to you.
Contributing 'broken' config files would be good too.

[Prasun Gera]

Just writing to say that the automount scripts still seem to be quite
broken in RHEL 7.3. I did a couple of client installs recently, and
ipa-client-automount
--install completed successfully, but didn't add sss to /etc/nsswitch.conf.
By now, I've got used to this pattern. So I look for the presence or
absence of sss in nsswitch.conf after running any of these scripts, since
that seems to be the most common issue.

Post by Alexander Bokovoy

Post by Prasun Gera
I have zero confidence in any of the install and uninstall scripts. And
this is on RHEL systems. On unofficial ones like Ubuntu, things are even
more broken. I really like freeipa, but so far even in a smallish lab
environment, it has been a nightmare. I am really tempted to just go back
to NIS. Does anyone have any ideas or proposals for making things more
robust ? At the very least, I think that these sort of modifications to
system files should only happen with package install/removal. Any changes
that ipa's scripts do should be local to ipa's internal state. Better would
be to have an internal ipa database sort of thing which keeps track of what
the current state is so that even if a script dies, which has happened
often, the next attempt reads the database and figures out what happened
earlier.

File bugs with enough details. It is the only reliable way to fix any
issues where environments differ. Install/uninstall scripts work for
fresh installs in RHEL and Fedora because this is what is tested. If you
have repurposed machines from some other setups, things might differ and
only you know what is in your environment.
That's not bad or good, that's just different -- the more different
environments we see, more robust code can be added. People are
infinitely more clever than computers when it comes to configuration
files' format mangling.
I've seen multiple cases where a claim of 'ipa scripts broke my
configuration' was later retracted saying that puppet or other SCM run
afterwards did these changes. That just happen, if there are many
elephants dancing in the room, a careful coordination is always a good
idea.
Coming back to your issues, please file bugs -- either upstream or
downstream, via distributions, whatever way is more suitable to you.
Contributing 'broken' config files would be good too.
--
/ Alexander Bokovoy

[Rob Crittenden]

Post by Prasun Gera
Just writing to say that the automount scripts still seem to be quite
broken in RHEL 7.3. I did a couple of client installs recently,
and ipa-client-automount --install completed successfully, but didn't
add sss to /etc/nsswitch.conf. By now, I've got used to this pattern. So
I look for the presence or absence of sss in nsswitch.conf after running
any of these scripts, since that seems to be the most common issue.

https://bugzilla.redhat.com/show_bug.cgi?id=1392540

rob

Post by Prasun Gera
I have zero confidence in any of the install and uninstall scripts. And
this is on RHEL systems. On unofficial ones like Ubuntu, things are even
more broken. I really like freeipa, but so far even in a smallish lab
environment, it has been a nightmare. I am really tempted to just go back
to NIS. Does anyone have any ideas or proposals for making things more
robust ? At the very least, I think that these sort of
modifications to
system files should only happen with package install/removal. Any changes
that ipa's scripts do should be local to ipa's internal state. Better would
be to have an internal ipa database sort of thing which keeps track of what
the current state is so that even if a script dies, which has happened
often, the next attempt reads the database and figures out what happened
earlier.
File bugs with enough details. It is the only reliable way to fix any
issues where environments differ. Install/uninstall scripts work for
fresh installs in RHEL and Fedora because this is what is tested. If you
have repurposed machines from some other setups, things might differ and
only you know what is in your environment.
That's not bad or good, that's just different -- the more different
environments we see, more robust code can be added. People are
infinitely more clever than computers when it comes to configuration
files' format mangling.
I've seen multiple cases where a claim of 'ipa scripts broke my
configuration' was later retracted saying that puppet or other SCM run
afterwards did these changes. That just happen, if there are many
elephants dancing in the room, a careful coordination is always a good
idea.
Coming back to your issues, please file bugs -- either upstream or
downstream, via distributions, whatever way is more suitable to you.
Contributing 'broken' config files would be good too.
--
/ Alexander Bokovoy