Andrew Krause
2017-04-17 16:31:22 UTC
Many hosts in our web ui show a null status for “enrolled”. When you do a search that includes any of these host objects the web UI posts errors, and if you click on one of the problem hosts the same error stops anything from loading on the host page.
I’ve been trying to solve this problem on my own for quite some time and have not been successful. It’s impossible to remove the host through the web UI and using CLI commands seem to remove the entry from IPA (host is not found with ipa host-find), but it is still visible in the UI. One thing that may be common with all of these hosts is that they were enrolled with our IPA system back while we were running version 3.0 and likely have had issues for quite some time. Multiple updates have happened since then, and all of our hosts added within the last year are working fine. I suspect there’s an issue with a path somewhere for a certificate database, but I’m unable to pinpoint what is going wrong.
I’m currently cloning 2 of my IPA servers into a private dmz to test fixes so I can try things without worry...
1. Realized we had many certificates that were expired and not renewing with “getcert list” on primary IPA server
2. Tried every document I could find on renewing the certificates but was never completely successful (on version 4.1 which is our current in production)
3. Upgraded to 4.4 and was actually able to renew all certificates listed on the main IPA server showing current below
4. After having success with #3 I was able to start the CA service without error and everything on the server seems to be working as expected
5. Have attempted many variations of removing a problem host and adding it back, but the errors in the web UI persist.
Output from "getcert list":
Number of certificates and requests being tracked: 8.
Request ID '20160901214852':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=CA Audit,O=DOMAIN.COM
expires: 2018-08-22 22:13:44 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214853':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=OCSP Subsystem,O=DOMAIN.COM
expires: 2018-08-22 21:49:26 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214854':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=CA Subsystem,O=DOMAIN.COM
expires: 2018-08-22 21:49:18 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214855':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=Certificate Authority,O=DOMAIN.COM
expires: 2036-09-01 05:05:00 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214856':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=IPA RA,O=DOMAIN.COM
expires: 2018-08-22 22:15:36 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20160901214857':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=hostname07.domain.com,O=DOMAIN.COM
expires: 2018-07-31 23:31:17 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214858':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DOMAIN-COM/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=hostname07.domain.com,O=DOMAIN.COM
expires: 2018-08-22 23:31:28 UTC
principal name: ldap/***@DOMAIN.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DOMAIN-COM
track: yes
auto-renew: yes
Request ID '20160901214859':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=hostname07.domain.com,O=DOMAIN.COM
expires: 2018-08-22 23:31:19 UTC
principal name: HTTP/***@DOMAIN.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Output for "certutil -L -d /var/lib/pki/pki-tomcat/alias/"
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert cert-pki-ca u,u,u
Certificate Authority - DOMAIN.COM CTu,cu,u
subsystemCert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
caSigningCert cert-pki-ca u,u,u
ocspSigningCert cert-pki-ca u,u,u
Output for latest selftests.log for pki-tomcatd:
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: Initializing self test plugins:
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading all self test plugin instances
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading self test plugins in startup order
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded!
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup:
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] CAPresence: CA is present
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SystemCertsVerification: system certs verification success
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: All CRITICAL self test plugins ran SUCCESSFULLY at startup!
Any assistance would be greatly appreciated.
Andrew Krause
I’ve been trying to solve this problem on my own for quite some time and have not been successful. It’s impossible to remove the host through the web UI and using CLI commands seem to remove the entry from IPA (host is not found with ipa host-find), but it is still visible in the UI. One thing that may be common with all of these hosts is that they were enrolled with our IPA system back while we were running version 3.0 and likely have had issues for quite some time. Multiple updates have happened since then, and all of our hosts added within the last year are working fine. I suspect there’s an issue with a path somewhere for a certificate database, but I’m unable to pinpoint what is going wrong.
I’m currently cloning 2 of my IPA servers into a private dmz to test fixes so I can try things without worry...
1. Realized we had many certificates that were expired and not renewing with “getcert list” on primary IPA server
2. Tried every document I could find on renewing the certificates but was never completely successful (on version 4.1 which is our current in production)
3. Upgraded to 4.4 and was actually able to renew all certificates listed on the main IPA server showing current below
4. After having success with #3 I was able to start the CA service without error and everything on the server seems to be working as expected
5. Have attempted many variations of removing a problem host and adding it back, but the errors in the web UI persist.
Output from "getcert list":
Number of certificates and requests being tracked: 8.
Request ID '20160901214852':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='auditSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=CA Audit,O=DOMAIN.COM
expires: 2018-08-22 22:13:44 UTC
key usage: digitalSignature,nonRepudiation
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "auditSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214853':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='ocspSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=OCSP Subsystem,O=DOMAIN.COM
expires: 2018-08-22 21:49:26 UTC
eku: id-kp-OCSPSigning
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "ocspSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214854':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='subsystemCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=CA Subsystem,O=DOMAIN.COM
expires: 2018-08-22 21:49:18 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "subsystemCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214855':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='caSigningCert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=Certificate Authority,O=DOMAIN.COM
expires: 2036-09-01 05:05:00 UTC
key usage: digitalSignature,nonRepudiation,keyCertSign,cRLSign
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "caSigningCert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214856':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='ipaCert',token='NSS Certificate DB'
CA: dogtag-ipa-ca-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=IPA RA,O=DOMAIN.COM
expires: 2018-08-22 22:15:36 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/renew_ra_cert_pre
post-save command: /usr/libexec/ipa/certmonger/renew_ra_cert
track: yes
auto-renew: yes
Request ID '20160901214857':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB',pin set
certificate: type=NSSDB,location='/etc/pki/pki-tomcat/alias',nickname='Server-Cert cert-pki-ca',token='NSS Certificate DB'
CA: dogtag-ipa-renew-agent
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=hostname07.domain.com,O=DOMAIN.COM
expires: 2018-07-31 23:31:17 UTC
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command: /usr/libexec/ipa/certmonger/stop_pkicad
post-save command: /usr/libexec/ipa/certmonger/renew_ca_cert "Server-Cert cert-pki-ca"
track: yes
auto-renew: yes
Request ID '20160901214858':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/dirsrv/slapd-DOMAIN-COM/pwdfile.txt'
certificate: type=NSSDB,location='/etc/dirsrv/slapd-DOMAIN-COM',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=hostname07.domain.com,O=DOMAIN.COM
expires: 2018-08-22 23:31:28 UTC
principal name: ldap/***@DOMAIN.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_dirsrv DOMAIN-COM
track: yes
auto-renew: yes
Request ID '20160901214859':
status: MONITORING
stuck: no
key pair storage: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB',pinfile='/etc/httpd/alias/pwdfile.txt'
certificate: type=NSSDB,location='/etc/httpd/alias',nickname='Server-Cert',token='NSS Certificate DB'
CA: IPA
issuer: CN=Certificate Authority,O=DOMAIN.COM
subject: CN=hostname07.domain.com,O=DOMAIN.COM
expires: 2018-08-22 23:31:19 UTC
principal name: HTTP/***@DOMAIN.COM
key usage: digitalSignature,nonRepudiation,keyEncipherment,dataEncipherment
eku: id-kp-serverAuth,id-kp-clientAuth
pre-save command:
post-save command: /usr/libexec/ipa/certmonger/restart_httpd
track: yes
auto-renew: yes
Output for "certutil -L -d /var/lib/pki/pki-tomcat/alias/"
Certificate Nickname Trust Attributes
SSL,S/MIME,JAR/XPI
Server-Cert cert-pki-ca u,u,u
Certificate Authority - DOMAIN.COM CTu,cu,u
subsystemCert cert-pki-ca u,u,u
auditSigningCert cert-pki-ca u,u,Pu
caSigningCert cert-pki-ca u,u,u
ocspSigningCert cert-pki-ca u,u,u
Output for latest selftests.log for pki-tomcatd:
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: Initializing self test plugins:
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading all self test plugin logger parameters
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading all self test plugin instances
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading all self test plugin instance parameters
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading self test plugins in on-demand order
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: loading self test plugins in startup order
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: Self test plugins have been successfully loaded!
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: Running self test plugins specified to be executed at startup:
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] CAPresence: CA is present
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SystemCertsVerification: system certs verification success
0.localhost-startStop-1 - [17/Apr/2017:10:11:53 CDT] [20] [1] SelfTestSubsystem: All CRITICAL self test plugins ran SUCCESSFULLY at startup!
Any assistance would be greatly appreciated.
Andrew Krause
--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info