[Freeipa-users] SSSD setting memcache

Hi,
yesterday I suddenly was unable to use the webinterface of my ipa master. SSH
login (with root user) did not work also.
When I uncommented the setting "memcache_timeout = 600" in the sssd config
file of the master everything seemed to work fine again. (my ipa setup has a
trust to AD)

I doubt it had anything to do memcache_timeout.
I would say that restart of sssd helped. But it difficult to say
without log files. either sssd logs or at least /var/log/secure
(journald for pam).

LS

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Ronald Wimmer

2017-04-04 07:41:38 UTC

Hi,
yesterday I suddenly was unable to use the webinterface of my ipa master. SSH
login (with root user) did not work also.
When I uncommented the setting "memcache_timeout = 600" in the sssd config
file of the master everything seemed to work fine again. (my ipa setup has a
trust to AD)

I doubt it had anything to do memcache_timeout.
I would say that restart of sssd helped. But it difficult to say
without log files. either sssd logs or at least /var/log/secure
(journald for pam).

You were right. I uncommented the setting and the problem ocurred again.

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Lukas Slebodnik

2017-04-08 10:53:39 UTC

Hi,
yesterday I suddenly was unable to use the webinterface of my ipa master. SSH
login (with root user) did not work also.
When I uncommented the setting "memcache_timeout = 600" in the sssd config
file of the master everything seemed to work fine again. (my ipa setup has a
trust to AD)

I doubt it had anything to do memcache_timeout.
I would say that restart of sssd helped. But it difficult to say
without log files. either sssd logs or at least /var/log/secure
(journald for pam).

You were right. I uncommented the setting and the problem ocurred again.

Did you find anything suspicious in journald?
Is sssd_be busy (or any other process)?
high CPU, IO operations ...

It would be good to know more details. Restarting sssd is not a solution.

LS

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Ronald Wimmer

2017-04-08 15:55:00 UTC

Hi,
yesterday I suddenly was unable to use the webinterface of my ipa master. SSH
login (with root user) did not work also.
When I uncommented the setting "memcache_timeout = 600" in the sssd config
file of the master everything seemed to work fine again. (my ipa setup has a
trust to AD)

I doubt it had anything to do memcache_timeout.
I would say that restart of sssd helped. But it difficult to say
without log files. either sssd logs or at least /var/log/secure
(journald for pam).

You were right. I uncommented the setting and the problem ocurred again.

Did you find anything suspicious in journald?
Is sssd_be busy (or any other process)?
high CPU, IO operations ...
It would be good to know more details. Restarting sssd is not a solution.

sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd
cache directory. After following
https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/
the problems did nod reappear.

Regards,
Ronald

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Jakub Hrozek

2017-04-09 19:11:14 UTC

Hi,
yesterday I suddenly was unable to use the webinterface of my ipa master. SSH
login (with root user) did not work also.
When I uncommented the setting "memcache_timeout = 600" in the sssd config
file of the master everything seemed to work fine again. (my ipa setup has a
trust to AD)

I doubt it had anything to do memcache_timeout.
I would say that restart of sssd helped. But it difficult to say
without log files. either sssd logs or at least /var/log/secure
(journald for pam).

You were right. I uncommented the setting and the problem ocurred again.

Did you find anything suspicious in journald?
Is sssd_be busy (or any other process)?
high CPU, IO operations ...
It would be good to know more details. Restarting sssd is not a solution.

sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd cache
directory. After following https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/
the problems did nod reappear.

btw even after the performance improvements we did in 1.14 we an issue
where even parsing the entries takes too long. What we did in 1.14 was
that if the entries didn't change compared to what is already in the
cache, then we skipped saving the full entry again just to bump the
timestamp. Making the parsing faster is planned for the next version.

(btw there was a bug where on upgrade, this new performance improvement
didn't take effect for objects that were already cached. Removing the
cache is a simple workaround and it's something we should fix soon in
the code..)

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Lukas Slebodnik

2017-04-10 10:16:27 UTC

Hi,
yesterday I suddenly was unable to use the webinterface of my ipa master. SSH
login (with root user) did not work also.
When I uncommented the setting "memcache_timeout = 600" in the sssd config
file of the master everything seemed to work fine again. (my ipa setup has a
trust to AD)

I doubt it had anything to do memcache_timeout.
I would say that restart of sssd helped. But it difficult to say
without log files. either sssd logs or at least /var/log/secure
(journald for pam).

You were right. I uncommented the setting and the problem ocurred again.

Did you find anything suspicious in journald?
Is sssd_be busy (or any other process)?
high CPU, IO operations ...
It would be good to know more details. Restarting sssd is not a solution.

sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd cache
directory. After following https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/
the problems did nod reappear.

Did you try all recommended steps or just few?

Do you know which one was the most useful in your case?

LS

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Ronald Wimmer

2017-04-10 11:07:08 UTC

[...]
sssd_be consumed a lot of CPU and produced a lot of I/O in the sssd cache
directory. After following https://jhrozek.wordpress.com/2015/08/19/performance-tuning-sssd-for-large-ipa-ad-trust-deployments/
the problems did nod reappear.
Did you try all recommended steps or just few?
Do you know which one was the most useful in your case?

I think the biggest benefit came from moving the sssd cache into RAM.

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Jakub Hrozek

2017-04-10 11:23:22 UTC

I think the biggest benefit came from moving the sssd cache into RAM.

This shouldn't be the case with 1.14+ and wasn't in my testing. Did you
remove the cache (really remove, not just expire with sss_cache) after
you upgraded from 1.13 to 1.14?

If yes, can you run some simple systemtap scripts?

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Ronald Wimmer

2017-04-10 11:42:19 UTC