[Freeipa-users] Can't make replica with CA due to LDAP 'replication manager' user not found error

[Chris Dagdigian]

Any guidance for this one?

Summary - this seems to be the fatal error that causes the CA setup on
the replica to fail:

May 03 15:09:09 usaeilidmp002.XXX.org server[3993]: testLDAPConnection:
The specified user cn=Replication Manager
masterAgreement1-usaeilidmp002.XXX.org-pki-tomcat,cn=config does not exist


May 03 15:09:09 usaeilidmp002.XXX.org server[3993]: CMSEngine: init():
password test execution failed for replicationdbwith NO_SUCH_USER. This
may not be a latest instance. Ignoring ..


More details ...


Trying to build a replica with CA duties for the first time.

It hangs here during the replica install process:


ipa : DEBUG stderr=
ipa : DEBUG wait_for_open_ports: localhost [8080, 8443]
timeout 300
ipa : DEBUG Waiting until the CA is running
ipa : DEBUG request POST
http://usaeilidmp002.XXX.org:8080/ca/admin/ca/getStatus
ipa : DEBUG request body ''


However the root cause seems to be that the CA won't start because
something is wrong with an LDAP replication manager user?

When I restart the pki-tomcatd service the replica install STDOUT
refreshes the above status. After the 3rd attempt it triggers the fatal
"CA will not start after 300 seconds" error



From the logs:

# systemctl status pki-***@pki-tomcat.service
● pki-***@pki-tomcat.service - PKI Tomcat Server pki-tomcat
Loaded: loaded (/lib/systemd/system/pki-***@.service; enabled;
vendor preset: disabled)
Active: active (running) since Wed 2017-05-03 15:09:04 UTC; 40s ago
Process: 3843 ExecStop=/usr/libexec/tomcat/server stop (code=exited,
status=1/FAILURE)
Process: 3880 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited,
status=0/SUCCESS)
Main PID: 3993 (java)
CGroup:
/system.slice/system-pki\x2dtomcatd.slice/pki-***@pki-tomcat.service
└─3993 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
-DRESTEASY_LIB=/usr/share/java/resteasy-base
-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/...

May 03 15:09:08 usaeilidmp002.XXX.org server[3993]:
SSLAuthenticatorWithFallback: Setting container
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]:
SSLAuthenticatorWithFallback: Initializing authenticators
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]:
SSLAuthenticatorWithFallback: Starting authenticators
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]:
CMSEngine.initializePasswordStore() begins
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]:
CMSEngine.initializePasswordStore(): tag=internaldb
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]: testLDAPConnection
connecting to usaeilidmp002.XXX.org:389
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]:
CMSEngine.initializePasswordStore(): tag=replicationdb
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]: testLDAPConnection
connecting to usaeilidmp002.XXX.org:389
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]: testLDAPConnection:
The specified user cn=Replication Manager
masterAgreement1-usaeilidmp002.XXX...not exist
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]: CMSEngine: init():
password test execution failed for replicationdbwith NO_SUCH_USER. This
may not...noring ..
Hint: Some lines were ellipsized, use -l to show in full.

[Florence Blanc-Renaud]

Post by Chris Dagdigian
Any guidance for this one?
Summary - this seems to be the fatal error that causes the CA setup on
The specified user cn=Replication Manager
masterAgreement1-usaeilidmp002.XXX.org-pki-tomcat,cn=config does not exist
password test execution failed for replicationdbwith NO_SUCH_USER. This
may not be a latest instance. Ignoring ..
More details ...
Trying to build a replica with CA duties for the first time.
ipa : DEBUG stderr=
ipa : DEBUG wait_for_open_ports: localhost [8080, 8443]
timeout 300
ipa : DEBUG Waiting until the CA is running
ipa : DEBUG request POST
http://usaeilidmp002.XXX.org:8080/ca/admin/ca/getStatus
ipa : DEBUG request body ''
However the root cause seems to be that the CA won't start because
something is wrong with an LDAP replication manager user?
When I restart the pki-tomcatd service the replica install STDOUT
refreshes the above status. After the 3rd attempt it triggers the fatal
"CA will not start after 300 seconds" error
vendor preset: disabled)
Active: active (running) since Wed 2017-05-03 15:09:04 UTC; 40s ago
Process: 3843 ExecStop=/usr/libexec/tomcat/server stop (code=exited,
status=1/FAILURE)
Process: 3880 ExecStartPre=/usr/bin/pkidaemon start %i (code=exited,
status=0/SUCCESS)
Main PID: 3993 (java)
└─3993 /usr/lib/jvm/jre-1.8.0-openjdk/bin/java
-DRESTEASY_LIB=/usr/share/java/resteasy-base
-Djava.library.path=/usr/lib64/nuxwdog-jni -classpath /usr/share/...
SSLAuthenticatorWithFallback: Setting container
SSLAuthenticatorWithFallback: Initializing authenticators
SSLAuthenticatorWithFallback: Starting authenticators
CMSEngine.initializePasswordStore() begins
CMSEngine.initializePasswordStore(): tag=internaldb
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]: testLDAPConnection
connecting to usaeilidmp002.XXX.org:389
CMSEngine.initializePasswordStore(): tag=replicationdb
May 03 15:09:09 usaeilidmp002.XXX.org server[3993]: testLDAPConnection
connecting to usaeilidmp002.XXX.org:389
The specified user cn=Replication Manager
masterAgreement1-usaeilidmp002.XXX...not exist
password test execution failed for replicationdbwith NO_SUCH_USER. This
may not...noring ..
Hint: Some lines were ellipsized, use -l to show in full.

Hi,

the issue looks similar to ticket 6766 [1]
Flo.

[1] https://pagure.io/freeipa/issue/6766

[Chris Dagdigian]

Post by Florence Blanc-Renaud
the issue looks similar to ticket 6766 [1]
Flo.
[1] https://pagure.io/freeipa/issue/6766

Thanks Flo, I agree that this looks like the issue I"m hitting in v4.4
much appreciated!

I'm gonna be watching this closely, it's nerve wracking knowing that I
can't use, update or create *any* replica servers at the moment ...

-Chris

[Standa Laznicka]

Post by Chris Dagdigian

Post by Florence Blanc-Renaud
the issue looks similar to ticket 6766 [1]
Flo.
[1] https://pagure.io/freeipa/issue/6766

Thanks Flo, I agree that this looks like the issue I"m hitting in v4.4
much appreciated!
I'm gonna be watching this closely, it's nerve wracking knowing that I
can't use, update or create *any* replica servers at the moment ...
-Chris

You can, but you probably won't be able to install a CA replica on them
(you have to leave out the --setup-ca option). In the meantime, you can
create replicas without CA replication and when the Dogtag/DS guys solve
the problem, you can run ipa-ca-install on those to setup CA replication
there as well.

[Chris Dagdigian]

Post by Standa Laznicka
You can, but you probably won't be able to install a CA replica on
them (you have to leave out the --setup-ca option). In the meantime,
you can create replicas without CA replication and when the Dogtag/DS
guys solve the problem, you can run ipa-ca-install on those to setup
CA replication there as well.

Appreciate the attention this is getting!

My testing from yesterday shows that all replication is broken for me
due to this 'replication manager' user not existing in LDAP so I may be
hit by something in addition to the dogtag issue

I have two servers that are out of sync with each other

- Manual force update fails
- Manual re-initialization fails
- Installing a new IPA server without CA-service claims to work but no
actual updates transfer

As far as I can tell all of the failures are due to an LDAP access issue
where the logs talk about a replication-agreement-specific LDAP user not
existing.

Example From Replica:

# ipa-replica-manage -v re-initialize --from usaeilidmp001.redactedidm.org
ipa: INFO: Setting agreement
cn=meTousaeilidmp002.redactedidm.org,cn=replica,cn=dc\=redactedidm\,dc\=org,cn=mapping
tree,cn=config schedule to 2358-2359 0 to force synch
ipa: INFO: Deleting schedule 2358-2359 0 from agreement
cn=meTousaeilidmp002.redactedidm.org,cn=replica,cn=dc\=redactedidm\,dc\=org,cn=mapping
tree,cn=config
Update in progress, 14 seconds elapsed

# [usaeilidmp001.redactedidm.org] reports: Update failed! Status: [-2 -
LDAP error: Local error]



dirsirv error logs from Master:

04/May/2017:12:20:08.531621754 +0000] slapi_ldap_bind - Error: could not
bind id [cn=Replication Manager
cloneAgreement1-usaeilidmp002.redactedidm.org-pki-tomcat,ou=csusers,cn=config]
authentication mechanism [SIMPLE]: error 32 (No such object) errno 0
(Success)
[04/May/2017:12:20:10.071619724 +0000] slapi_ldap_bind - Error: could
not bind id [cn=Replication Manager
cloneAgreement1-deawilidmp001.redactedidm.org-pki-tomcat,ou=csusers,cn=config]
authentication mechanism [SIMPLE]: error 32 (No such object) errno 0
(Success)
[04/May/2017:12:20:11.074340742 +0000] set_krb5_creds - Could not get
initial credentials for principal [ldap/usaeilidmp001.redactedidm.org@]
in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not
found)
[04/May/2017:12:20:35.078730934 +0000] set_krb5_creds - Could not get
initial credentials for principal [ldap/usaeilidmp001.redactedidm.org@]
in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not
found)
[04/May/2017:12:21:23.083737475 +0000] set_krb5_creds - Could not get
initial credentials for principal [ldap/usaeilidmp001.redactedidm.org@]
in keytab [FILE:/etc/dirsrv/ds.keytab]: -1765328203 (Key table entry not
found)





Regards,
Chris