[Freeipa-users] Original master lost, cannot create additional CA clones

Discussion:

Brennan, Paul J

2017-03-21 15:42:14 UTC

Hi all,
Some time ago, I encountered issues requiring my first IPA master to be re-initialized, which failed, forcing me to remove it from the domain. While those original issues have since been resolved, I am having difficulty replacing the system. I can create a new replica, but I cannot use the '--setup-ca' option, nor can I run 'ipa-ca-install'. I have been working on this for quite some time, and seem to be going in circles. Any help would be greatly appreciated.

(host and domain names have been modified)
ipasrv001 was the original master, installed using an external CA, no DNS and no NTP. DNS and NTP are already provided in my environment.
ipasrv201 is a replica installed with --setup-ca, which has since been re-configured as the new CA master following this guide:
https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master#Procedure_in_FreeIPA_.3C_4.0
I have also pointed cs-replication to this system from all other replicas. I used ipasrv201 to generate new replica-info for ipasrv001.

These systems are Enterprise Linux 6.8. Current software versions are as follows:
----------------------------------------------------------------------------------------------------
[***@ipasrv001 ~]# rpm -qa --queryformat='%{NAME} %{VERSION}-%{RELEASE}\n' ipa-server pki-ca 389-ds-base java-1.7.0-openjdk certmonger
certmonger 0.77.5-2.el6
pki-ca 9.0.3-50.el6_8
389-ds-base 1.2.11.15-75.el6_8
java-1.7.0-openjdk 1.7.0.111-2.6.7.2.0.1.el6_8
ipa-server 3.0.0-50.el6_8.3
----------------------------------------------------------------------------------------------------

I receive the following error when attempting to create a new replica:
----------------------------------------------------------------------------------------------------
[***@ipasrv001 ~]# ipa-replica-install --no-ntp --setup-ca --skip-conncheck replica-info-ipasrv001.example.com.gpg
Directory Manager (existing master) password:

Configuring directory server for the CA (pkids): Estimated time 30 seconds
[1/3]: creating directory server user
[2/3]: creating directory server instance
[3/3]: restarting directory server
Done configuring directory server for the CA (pkids).
Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds
[1/17]: creating certificate server user
[2/17]: creating pki-ca instance
[3/17]: configuring certificate server instance
ipa : CRITICAL failed to configure ca instance Command '/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname ipasrv001.example.com -cs_port 9445 -client_certdb_dir /tmp/tmp-BDfli8 -client_certdb_pwd XXXXXXXX -preop_pin SoawlFdqmJKt79OSoy1O -domain_name IPA -admin_user admin -admin_email ***@localhost -admin_password XXXXXXXX -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa -agent_cert_subject CN=ipa-ca-agent,O=EXAMPLE.COM,OU=IPA -ldap_host ipasrv001.example.com -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password XXXXXXXX -base_dn o=ipaca -db_name ipaca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd XXXXXXXX -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=EXAMPLE.COM,OU=IPA -ca_subsystem_cert_subject_name CN=CA Subsystem,O=EXAMPLE.COM,OU=IPA -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=EXAMPLE.COM,OU=IPA -ca_server_cert_subject_name CN=ipasrv001.example.com,O=EXAMPLE.COM,OU=IPA -ca_audit_signing_cert_subject_name CN=CA Audit,O=EXAMPLE.COM,OU=IPA -ca_sign_cert_subject_name CN=Certificate Authority,O=EXAMPLE.COM,OU=IPA -external false -clone true -clone_p12_file ca.p12 -clone_p12_password XXXXXXXX -sd_hostname ipasrv201.example.com -sd_admin_port 443 -sd_admin_name admin -sd_admin_password XXXXXXXX -clone_start_tls true -clone_uri https://ipasrv201.example.com:443' returned non-zero exit status 255

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Configuration of CA failed
----------------------------------------------------------------------------------------------------

In /var/log/ipareplica-install.log, I see the following:
----------------------------------------------------------------------------------------------------
#############################################
Attempting to connect to: ipasrv001.example.com:9445
Connected.
Posting Query = https://ipasrv001.example.com:9445//ca/admin/console/config/wizard?p=5&subsystem=CA&session_id=-1815206698136119192&xml=true
RESPONSE STATUS: HTTP/1.1 200 OK
RESPONSE HEADER: Server: Apache-Coyote/1.1
RESPONSE HEADER: Content-Type: text/html;charset=UTF-8
RESPONSE HEADER: Date: Fri, 17 Mar 2017 15:50:34 GMT
RESPONSE HEADER: Connection: close
Exception in SecurityDomainLoginPanel(): java.lang.Exception: Invalid clone_uri
ERROR: ConfigureSubCA: SecurityDomainLoginPanel() failure
ERROR: unable to create CA

#######################################################################

2017-03-17T15:50:35Z DEBUG stderr=java.lang.Exception: Invalid clone_uri
at ConfigureCA.SecurityDomainLoginPanel(ConfigureCA.java:392)
at ConfigureCA.ConfigureCAInstance(ConfigureCA.java:1179)
at ConfigureCA.main(ConfigureCA.java:1663)
----------------------------------------------------------------------------------------------------

In /var/log/pki-ca/debug, I see:
----------------------------------------------------------------------------------------------------
Could not get or build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate
at com.netscape.ca.CertificateAuthority.initSigUnit(CertificateAuthority.java:1285)
at com.netscape.ca.CertificateAuthority.init(CertificateAuthority.java:262)
----------------------------------------------------------------------------------------------------

In /var/log/pki-ca/catalina.out I see:
----------------------------------------------------------------------------------------------------
CMS Warning: FAILURE: Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate|FAILURE: authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value|
----------------------------------------------------------------------------------------------------

I have tried deploying a new replica to freshly installed systems, and the same problem occurs. I have backups from when IPA was first installed, if any config files or certificates need to be brought back. I can provide further log excerpts if needed.

Thank you in advance,
Paul Brennan

David Kupka

2017-03-22 06:09:52 UTC

Permalink

On Tue, Mar 21, 2017 at 03:42:14PM +0000, Brennan, Paul J wrote:
> Hi all,
> Some time ago, I encountered issues requiring my first IPA master to be re-initialized, which failed, forcing me to remove it from the domain. While those original issues have since been resolved, I am having difficulty replacing the system. I can create a new replica, but I cannot use the '--setup-ca' option, nor can I run 'ipa-ca-install'. I have been working on this for quite some time, and seem to be going in circles. Any help would be greatly appreciated.
>
> (host and domain names have been modified)
> ipasrv001 was the original master, installed using an external CA, no DNS and no NTP. DNS and NTP are already provided in my environment.
> ipasrv201 is a replica installed with --setup-ca, which has since been re-configured as the new CA master following this guide:
> https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master#Procedure_in_FreeIPA_.3C_4.0
> I have also pointed cs-replication to this system from all other replicas. I used ipasrv201 to generate new replica-info for ipasrv001.
>
> These systems are Enterprise Linux 6.8. Current software versions are as follows:
> ----------------------------------------------------------------------------------------------------
> [***@ipasrv001 ~]# rpm -qa --queryformat='%{NAME} %{VERSION}-%{RELEASE}\n' ipa-server pki-ca 389-ds-base java-1.7.0-openjdk certmonger
> certmonger 0.77.5-2.el6
> pki-ca 9.0.3-50.el6_8
> 389-ds-base 1.2.11.15-75.el6_8
> java-1.7.0-openjdk 1.7.0.111-2.6.7.2.0.1.el6_8
> ipa-server 3.0.0-50.el6_8.3
> ----------------------------------------------------------------------------------------------------
>
> I receive the following error when attempting to create a new replica:
> ----------------------------------------------------------------------------------------------------
> [***@ipasrv001 ~]# ipa-replica-install --no-ntp --setup-ca --skip-conncheck replica-info-ipasrv001.example.com.gpg
> Directory Manager (existing master) password:
>
> Configuring directory server for the CA (pkids): Estimated time 30 seconds
> [1/3]: creating directory server user
> [2/3]: creating directory server instance
> [3/3]: restarting directory server
> Done configuring directory server for the CA (pkids).
> Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds
> [1/17]: creating certificate server user
> [2/17]: creating pki-ca instance
> [3/17]: configuring certificate server instance
> ipa : CRITICAL failed to configure ca instance Command '/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname ipasrv001.example.com -cs_port 9445 -client_certdb_dir /tmp/tmp-BDfli8 -client_certdb_pwd XXXXXXXX -preop_pin SoawlFdqmJKt79OSoy1O -domain_name IPA -admin_user admin -admin_email ***@localhost -admin_password XXXXXXXX -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa -agent_cert_subject CN=ipa-ca-agent,O=EXAMPLE.COM,OU=IPA -ldap_host ipasrv001.example.com -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password XXXXXXXX -base_dn o=ipaca -db_name ipaca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd XXXXXXXX -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=EXAMPLE.COM,OU=IPA -ca_subsystem_cert_subject_name CN=CA Subsystem,O=EXAMPLE.COM,OU=IPA -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=EXAMPLE.COM,OU=IPA -ca_server_cert_subject_name CN=ipasrv001.example.com,O=EXAMPLE.COM,OU=IPA -ca_audit_signing_cert_subject_name CN=CA Audit,O=EXAMPLE.COM,OU=IPA -ca_sign_cert_subject_name CN=Certificate Authority,O=EXAMPLE.COM,OU=IPA -external false -clone true -clone_p12_file ca.p12 -clone_p12_password XXXXXXXX -sd_hostname ipasrv201.example.com -sd_admin_port 443 -sd_admin_name admin -sd_admin_password XXXXXXXX -clone_start_tls true -clone_uri https://ipasrv201.example.com:443' returned non-zero exit status 255
>
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> Configuration of CA failed
> ----------------------------------------------------------------------------------------------------
>
> In /var/log/ipareplica-install.log, I see the following:
> ----------------------------------------------------------------------------------------------------
> #############################################
> Attempting to connect to: ipasrv001.example.com:9445
> Connected.
> Posting Query = https://ipasrv001.example.com:9445//ca/admin/console/config/wizard?p=5&subsystem=CA&session_id=-1815206698136119192&xml=true
> RESPONSE STATUS: HTTP/1.1 200 OK
> RESPONSE HEADER: Server: Apache-Coyote/1.1
> RESPONSE HEADER: Content-Type: text/html;charset=UTF-8
> RESPONSE HEADER: Date: Fri, 17 Mar 2017 15:50:34 GMT
> RESPONSE HEADER: Connection: close
> Exception in SecurityDomainLoginPanel(): java.lang.Exception: Invalid clone_uri
> ERROR: ConfigureSubCA: SecurityDomainLoginPanel() failure
> ERROR: unable to create CA
>
> #######################################################################
>
> 2017-03-17T15:50:35Z DEBUG stderr=java.lang.Exception: Invalid clone_uri
> at ConfigureCA.SecurityDomainLoginPanel(ConfigureCA.java:392)
> at ConfigureCA.ConfigureCAInstance(ConfigureCA.java:1179)
> at ConfigureCA.main(ConfigureCA.java:1663)
> ----------------------------------------------------------------------------------------------------
>
> In /var/log/pki-ca/debug, I see:
> ----------------------------------------------------------------------------------------------------
> Could not get or build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate
> at com.netscape.ca.CertificateAuthority.initSigUnit(CertificateAuthority.java:1285)
> at com.netscape.ca.CertificateAuthority.init(CertificateAuthority.java:262)
> ----------------------------------------------------------------------------------------------------
>
> In /var/log/pki-ca/catalina.out I see:
> ----------------------------------------------------------------------------------------------------
> CMS Warning: FAILURE: Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate|FAILURE: authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value|
> ----------------------------------------------------------------------------------------------------
>
> I have tried deploying a new replica to freshly installed systems, and the same problem occurs. I have backups from when IPA was first installed, if any config files or certificates need to be brought back. I can provide further log excerpts if needed.
>
> Thank you in advance,
> Paul Brennan
>

> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project

Hello Paul,
is there a reason you run ipa-replica-install with --skip-conncheck option?
Does it fail with the same error when you run with connection check?
There might be some closed ports on ipasrv201's firewall that cause this fail
and connection check would discover this. But it's just my wild guess.

--
David Kupka

Brennan, Paul J

2017-03-22 14:21:35 UTC

Permalink

Hi David,
These two servers and in separate datacenters, and my networking team closed port 80 at the perimeter on me. If I run with the conncheck, the replica install will fail because of this:

----------------------------------------------------------------------------------------------------
[***@ipasrv001 ~]# ipa-replica-install --no-ntp --setup-ca replica-info-ipasrv001.itec.suny.edu.gpg
Directory Manager (existing master) password:

Run connection check to master
Check connection from replica to remote master 'ipasrv201.itec.suny.edu':
Directory Service: Unsecure port (389): OK
Directory Service: Secure port (636): OK
Kerberos KDC: TCP (88): OK
Kerberos Kpasswd: TCP (464): OK
HTTP Server: Unsecure port (80): FAILED
HTTP Server: Secure port (443): OK
PKI-CA: Directory Service port (7389): OK
Port check failed! Inaccessible port(s): 80 (TCP)
Connection check failed!
Please fix your network settings according to error messages above.
If the check results are not valid it can be skipped with --skip-conncheck parameter.
----------------------------------------------------------------------------------------------------

I have manually verified the ipasrv201 can reach ipasrv001 via ports 389, 636, 88, 464, 443, and 7389 tcp.
There is another replica, ipasrv003, in the same datacenter as ipasrv001. These two servers can communicate over port 80/tcp, and I have tried preparing the replica info from ipasrv003 without success. (Same failure with ca-setup.) I have also tried disabling the local firewall an ipasrv001 in case the install was attempting to access a port listening on the external interface (as opposed to loopback). At this point, I believe the network is inconsequential to the problem.

Thank you for the input,
Paul Brennan

________________________________________
From: David Kupka [***@redhat.com]
Sent: Wednesday, March 22, 2017 2:09 AM
To: Brennan, Paul J
Cc: freeipa-***@redhat.com
Subject: Re: [Freeipa-users] Original master lost, cannot create additional CA clones

On Tue, Mar 21, 2017 at 03:42:14PM +0000, Brennan, Paul J wrote:
> Hi all,
> Some time ago, I encountered issues requiring my first IPA master to be re-initialized, which failed, forcing me to remove it from the domain. While those original issues have since been resolved, I am having difficulty replacing the system. I can create a new replica, but I cannot use the '--setup-ca' option, nor can I run 'ipa-ca-install'. I have been working on this for quite some time, and seem to be going in circles. Any help would be greatly appreciated.
>
> (host and domain names have been modified)
> ipasrv001 was the original master, installed using an external CA, no DNS and no NTP. DNS and NTP are already provided in my environment.
> ipasrv201 is a replica installed with --setup-ca, which has since been re-configured as the new CA master following this guide:
> https://www.freeipa.org/page/Howto/Promote_CA_to_Renewal_and_CRL_Master#Procedure_in_FreeIPA_.3C_4.0
> I have also pointed cs-replication to this system from all other replicas. I used ipasrv201 to generate new replica-info for ipasrv001.
>
> These systems are Enterprise Linux 6.8. Current software versions are as follows:
> ----------------------------------------------------------------------------------------------------
> [***@ipasrv001 ~]# rpm -qa --queryformat='%{NAME} %{VERSION}-%{RELEASE}\n' ipa-server pki-ca 389-ds-base java-1.7.0-openjdk certmonger
> certmonger 0.77.5-2.el6
> pki-ca 9.0.3-50.el6_8
> 389-ds-base 1.2.11.15-75.el6_8
> java-1.7.0-openjdk 1.7.0.111-2.6.7.2.0.1.el6_8
> ipa-server 3.0.0-50.el6_8.3
> ----------------------------------------------------------------------------------------------------
>
> I receive the following error when attempting to create a new replica:
> ----------------------------------------------------------------------------------------------------
> [***@ipasrv001 ~]# ipa-replica-install --no-ntp --setup-ca --skip-conncheck replica-info-ipasrv001.example.com.gpg
> Directory Manager (existing master) password:
>
> Configuring directory server for the CA (pkids): Estimated time 30 seconds
> [1/3]: creating directory server user
> [2/3]: creating directory server instance
> [3/3]: restarting directory server
> Done configuring directory server for the CA (pkids).
> Configuring certificate server (pki-cad): Estimated time 3 minutes 30 seconds
> [1/17]: creating certificate server user
> [2/17]: creating pki-ca instance
> [3/17]: configuring certificate server instance
> ipa : CRITICAL failed to configure ca instance Command '/usr/bin/perl /usr/bin/pkisilent ConfigureCA -cs_hostname ipasrv001.example.com -cs_port 9445 -client_certdb_dir /tmp/tmp-BDfli8 -client_certdb_pwd XXXXXXXX -preop_pin SoawlFdqmJKt79OSoy1O -domain_name IPA -admin_user admin -admin_email ***@localhost -admin_password XXXXXXXX -agent_name ipa-ca-agent -agent_key_size 2048 -agent_key_type rsa -agent_cert_subject CN=ipa-ca-agent,O=EXAMPLE.COM,OU=IPA -ldap_host ipasrv001.example.com -ldap_port 7389 -bind_dn cn=Directory Manager -bind_password XXXXXXXX -base_dn o=ipaca -db_name ipaca -key_size 2048 -key_type rsa -key_algorithm SHA256withRSA -save_p12 true -backup_pwd XXXXXXXX -subsystem_name pki-cad -token_name internal -ca_subsystem_cert_subject_name CN=CA Subsystem,O=EXAMPLE.COM,OU=IPA -ca_subsystem_cert_subject_name CN=CA Subsystem,O=EXAMPLE.COM,OU=IPA -ca_ocsp_cert_subject_name CN=OCSP Subsystem,O=EXAMPLE.COM,OU=IPA -ca_server_cert_subject_name CN=ipasrv001.example.com,
O=EXAMPLE.COM,OU=IPA -ca_audit_signing_cert_subject_name CN=CA Audit,O=EXAMPLE.COM,OU=IPA -ca_sign_cert_subject_name CN=Certificate Authority,O=EXAMPLE.COM,OU=IPA -external false -clone true -clone_p12_file ca.p12 -clone_p12_password XXXXXXXX -sd_hostname ipasrv201.example.com -sd_admin_port 443 -sd_admin_name admin -sd_admin_password XXXXXXXX -clone_start_tls true -clone_uri https://ipasrv201.example.com:443' returned non-zero exit status 255
>
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> Configuration of CA failed
> ----------------------------------------------------------------------------------------------------
>
> In /var/log/ipareplica-install.log, I see the following:
> ----------------------------------------------------------------------------------------------------
> #############################################
> Attempting to connect to: ipasrv001.example.com:9445
> Connected.
> Posting Query = https://ipasrv001.example.com:9445//ca/admin/console/config/wizard?p=5&subsystem=CA&session_id=-1815206698136119192&xml=true
> RESPONSE STATUS: HTTP/1.1 200 OK
> RESPONSE HEADER: Server: Apache-Coyote/1.1
> RESPONSE HEADER: Content-Type: text/html;charset=UTF-8
> RESPONSE HEADER: Date: Fri, 17 Mar 2017 15:50:34 GMT
> RESPONSE HEADER: Connection: close
> Exception in SecurityDomainLoginPanel(): java.lang.Exception: Invalid clone_uri
> ERROR: ConfigureSubCA: SecurityDomainLoginPanel() failure
> ERROR: unable to create CA
>
> #######################################################################
>
> 2017-03-17T15:50:35Z DEBUG stderr=java.lang.Exception: Invalid clone_uri
> at ConfigureCA.SecurityDomainLoginPanel(ConfigureCA.java:392)
> at ConfigureCA.ConfigureCAInstance(ConfigureCA.java:1179)
> at ConfigureCA.main(ConfigureCA.java:1663)
> ----------------------------------------------------------------------------------------------------
>
> In /var/log/pki-ca/debug, I see:
> ----------------------------------------------------------------------------------------------------
> Could not get or build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate
> at com.netscape.ca.CertificateAuthority.initSigUnit(CertificateAuthority.java:1285)
> at com.netscape.ca.CertificateAuthority.init(CertificateAuthority.java:262)
> ----------------------------------------------------------------------------------------------------
>
> In /var/log/pki-ca/catalina.out I see:
> ----------------------------------------------------------------------------------------------------
> CMS Warning: FAILURE: Cannot build CA chain. Error java.security.cert.CertificateException: Certificate is not a PKCS #11 certificate|FAILURE: authz instance DirAclAuthz initialization failed and skipped, error=Property internaldb.ldapconn.port missing value|
> ----------------------------------------------------------------------------------------------------
>
> I have tried deploying a new replica to freshly installed systems, and the same problem occurs. I have backups from when IPA was first installed, if any config files or certificates need to be brought back. I can provide further log excerpts if needed.
>
> Thank you in advance,
> Paul Brennan
>

> --
> Manage your subscription for the Freeipa-users mailing list:
> https://www.redhat.com/mailman/listinfo/freeipa-users
> Go to http://freeipa.org for more info on the project

Hello Paul,
is there a reason you run ipa-replica-install with --skip-conncheck option?
Does it fail with the same error when you run with connection check?
There might be some closed ports on ipasrv201's firewall that cause this fail
and connection check would discover this. But it's just my wild guess.

--
David Kupka

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project