[Freeipa-users] FreeIPA 4.2.0 / Replica / Join Issue

Discussion:

d***@pabstatencio.com

2016-03-03 20:12:23 UTC

I am running the latest patched CentOS 7.2, with FreeIPA 4.2.0, and I the Master node in the Data Center, then i created 3 replica's, one in the DC for High Availability, and then 2 Replica's in the AWS Cloud. I'm having major issues with the Replica's in the AWS Cloud. I am trying to have it so it auto-discovers the servers automatically so the failover is dynamic. I created the replica's as well to have a Certificate Authority. When I attempt to join a virtual machine in AWS to the domain it fails half way thru the process. I have attached a full debug of my ipa-client-install, hoping someone can assist me. I know prior to joining the 2 replicas in AWS I had absolutely no issues with joining servers in the DC to IDM. I built all my replica's from the Master server (rspsna-ipa01), so rspsna-ipa02, ipa01-ore, ipa02-ore were built from rspsna-ipa01.

The main part that seems to fail during the (client) join is:

Loading Index file from '/var/lib/ipa-client/sysrestore/sysrestore.index'
Loading StateFile from '/var/lib/ipa-client/sysrestore/sysrestore.state'
Starting external process
args='/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-L' '-n' 'Local IPA host' '-r'
Process finished, return code=255
stdout=
stderr=certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

Starting external process
args='/usr/bin/certutil' '-d' '/etc/pki/nssdb' '-L' '-n' 'IPA Machine Certificate - beanstalk01-ore.prod.cloud.myinc.local' '-r'
Process finished, return code=255
stdout=
stderr=certutil: Could not find cert: IPA Machine Certificate - beanstalk01-ore.prod.cloud.myinc.local
: PR_FILE_NOT_FOUND_ERROR: File not found

Starting external process
args='/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-L'
Process finished, return code=255
stdout=
stderr=certutil: function failed: SEC_ERROR_LEGACY_DATABASE: The certificate/key database is in an old, unsupported format.

Failed to list certificates in /etc/ipa/nssdb: Command ''/usr/bin/certutil' '-d' '/etc/ipa/nssdb' '-L'' returned non-zero exit status 255
Starting external process
args='/bin/systemctl' 'start' 'certmonger.service'
Process finished, return code=0
stdout=
stderr=
Starting external process
args='/bin/systemctl' 'is-active' 'certmonger.service'
Process finished, return code=0
stdout=active

stderr=
Starting external process
args='/bin/systemctl' 'stop' 'certmonger.service'
Process finished, return code=0
stdout=
stderr=
Starting external process
args='/bin/systemctl' 'disable' 'certmonger.service'
Process finished, return code=0
stdout=
stderr=
Unenrolling client from IPA server
Starting external process
args='/usr/sbin/ipa-join' '--unenroll' '-h' 'beanstalk01-ore.prod.cloud.myinc.local' '-d'
Process finished, return code=19
stdout=
stderr=Error obtaining initial credentials: Cannot find KDC for requested realm.

Unenrolling host failed: Error obtaining initial credentials: Cannot find KDC for requested realm.

Removing Kerberos service principals from /etc/krb5.keytab
Starting external process
args='/usr/sbin/ipa-rmkeytab' '-k' '/etc/krb5.keytab' '-r' 'MYINC.LOCAL'
Process finished, return code=0
stdout=
stderr=Removing principal host/beanstalk01-***@MYINC.LOCAL

When I look at the slapd error log on one of the replica's i see this:

[02/Mar/2016:23:40:09 +0000] - Listening on All Interfaces port 636 for LDAPS requests
[02/Mar/2016:23:40:09 +0000] - Listening on /var/run/slapd-MYINC-LOCAL.socket for LDAPI requests
[02/Mar/2016:23:40:09 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available)) errno 0 (Success)
[02/Mar/2016:23:40:09 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -2 (Local error)
[02/Mar/2016:23:40:09 +0000] NSMMReplicationPlugin - agmt="cn=meTorspsna-ipa01.prod.i2x.myinc.local" (rspsna-ipa01:389): Replication bind with GSSAPI auth failed: LDAP error -2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (No Kerberos credentials available))
[02/Mar/2016:23:40:12 +0000] NSMMReplicationPlugin - agmt="cn=meToipa02-ore.prod.cloud.myinc.local" (ipa02-ore:389): Replication bind with GSSAPI auth resumed
[02/Mar/2016:23:40:12 +0000] NSMMReplicationPlugin - agmt="cn=meTorspsna-ipa01.prod.i2x.myinc.local" (rspsna-ipa01:389): Replication bind with GSSAPI auth resumed
[03/Mar/2016:00:07:00 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is not connected)
[03/Mar/2016:00:07:00 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -1 (Can't contact LDAP server)
[03/Mar/2016:00:07:00 +0000] NSMMReplicationPlugin - agmt="cn=meToipa02-ore.prod.cloud.myinc.local" (ipa02-ore:389): Replication bind with GSSAPI auth failed: LDAP error -1 (Can't contact LDAP server) ()
[03/Mar/2016:00:07:03 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is not connected)
[03/Mar/2016:00:07:03 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -1 (Can't contact LDAP server)
[03/Mar/2016:00:07:09 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is not connected)
[03/Mar/2016:00:07:09 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -1 (Can't contact LDAP server)
[03/Mar/2016:00:07:21 +0000] slapd_ldap_sasl_interactive_bind - Error: could not perform interactive bind for id [] mech [GSSAPI]: LDAP error -1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is not connected)
[03/Mar/2016:00:07:21 +0000] slapi_ldap_bind - Error: could not perform interactive bind for id [] authentication mechanism [GSSAPI]: error -1 (Can't contact LDAP server)
[03/Mar/2016:00:07:45 +0000] NSMMReplicationPlugin - agmt="cn=meToipa02-ore.prod.cloud.myinc.local" (ipa02-ore:389): Replication bind with GSSAPI auth resumed
[03/Mar/2016:01:26:53 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:03:24:06 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:05:17:30 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:07:08:29 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:08:59:51 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:10:42:48 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:12:35:51 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:14:28:20 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:16:24:12 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:18:09:51 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
[03/Mar/2016:19:47:07 +0000] NSMMReplicationPlugin - replication keep alive entry already exists
Thanks much.
Devin

Rob Crittenden

2016-03-03 20:33:51 UTC

Permalink

Post by d***@pabstatencio.com
I am running the latest patched CentOS 7.2, with FreeIPA 4.2.0, and I
the Master node in the Data Center, then i created 3 replica's, one in
the DC for High Availability, and then 2 Replica's in the AWS Cloud. I'm
having major issues with the Replica's in the AWS Cloud. I am trying to
have it so it auto-discovers the servers automatically so the failover
is dynamic. I created the replica's as well to have a Certificate
Authority. When I attempt to join a virtual machine in AWS to the domain
it fails half way thru the process. I have attached a full debug of my
ipa-client-install, hoping someone can assist me. I know prior to
joining the 2 replicas in AWS I had absolutely no issues with joining
servers in the DC to IDM. I built all my replica's from the Master
server (rspsna-ipa01), so rspsna-ipa02, ipa01-ore, ipa02-ore were built
from rspsna-ipa01.

The important bits are needed. This part of the log is just trying to
clean things up (so failures are expected and ok). We'd really need to
see a full ipaclient-install.log.

Post by d***@pabstatencio.com
[02/Mar/2016:23:40:09 +0000] - Listening on All Interfaces port 636 for LDAPS requests
[02/Mar/2016:23:40:09 +0000] - Listening on
/var/run/slapd-MYINC-LOCAL.socket for LDAPI requests
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-2 (Local error) (SASL(-1): generic failure: GSSAPI Error: Unspecified
GSS failure. Minor code may provide more information (No Kerberos
credentials available)) errno 0 (Success)
[02/Mar/2016:23:40:09 +0000] slapi_ldap_bind - Error: could not perform
interactive bind for id [] authentication mechanism [GSSAPI]: error -2
(Local error)
[02/Mar/2016:23:40:09 +0000] NSMMReplicationPlugin -
Replication bind with GSSAPI auth failed: LDAP error -2 (Local error)
(SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure.
Minor code may provide more information (No Kerberos credentials available))
[02/Mar/2016:23:40:12 +0000] NSMMReplicationPlugin -
Replication bind with GSSAPI auth resumed
[02/Mar/2016:23:40:12 +0000] NSMMReplicationPlugin -
Replication bind with GSSAPI auth resumed

Up to here is ok and expected, this is just 389-ds realizing it doesn't
have Kerberos credentials yet and obtaining them.

Post by d***@pabstatencio.com
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)

For these I'd run:

$ ipa-replica-manage list -v `hostname` to see the status of the
agreements. It seems that one is unable to connect.

rob

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

d***@pabstatencio.com

2016-03-03 21:05:50 UTC

Permalink

Rob,

Yeah i forgot to attach the file when I initially sent. I also attached the output from all the nodes. I guess what i realized is that my agreements are a little different than i originally thought. What is also strange is on a few hosts that initially did enroll from AWS, when I look at the host via the GUI the host shows:

Kerberos Key Present, Host Provisioned
One-Time-Password Not Present
Host Certificate, No Valid Certificate

So the few that enrolled, they don't show having any Host certificates but they show Kerberos Key present and Host provisioned. Is there a problem with the way I provisioned the Replicas? I'm just using subdomains for human clarification but they all use the same Kerberos domain, etc.

[***@ipa02-ore ~]# ipa-replica-manage list -v `hostname`
Directory Manager password:

ipa01-ore.prod.cloud.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:39:30+00:00

[***@ipa01-ore ~]# ipa-replica-manage list -v `hostname`
ipa02-ore.prod.cloud.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:41:20+00:00
rspsna-ipa01.prod.i2x.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:41:29+00:00

[***@rspsna-ipa01 ~]# ipa-replica-manage list -v `hostname`

ipa01-ore.prod.cloud.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:43:35+00:00
rspsna-ipa02.prod.i2x.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:43:35+00:00

[***@rspsna-ipa02 ~]# ipa-replica-manage list -v `hostname`

rspsna-ipa01.prod.i2x.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:44:14+00:00

See attached file for the initial fail. Thanks very much for your help.

Devin Acosta

Post by Rob Crittenden

The important bits are needed. This part of the log is just trying to
clean things up (so failures are expected and ok). We'd really need to
see a full ipaclient-install.log.

Up to here is ok and expected, this is just 389-ds realizing it doesn't
have Kerberos credentials yet and obtaining them.

Post by d***@pabstatencio.com
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)

$ ipa-replica-manage list -v `hostname` to see the status of the
agreements. It seems that one is unable to connect.
rob

Petr Spacek

2016-03-04 08:43:32 UTC

Permalink

Post by d***@pabstatencio.com
Rob,
Kerberos Key Present, Host Provisioned
One-Time-Password Not Present
Host Certificate, No Valid Certificate
So the few that enrolled, they don't show having any Host certificates but they show Kerberos Key present and Host provisioned. Is there a problem with the way I provisioned the Replicas? I'm just using subdomains for human clarification but they all use the same Kerberos domain, etc.
ipa01-ore.prod.cloud.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:39:30+00:00
ipa02-ore.prod.cloud.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:41:20+00:00
rspsna-ipa01.prod.i2x.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:41:29+00:00
ipa01-ore.prod.cloud.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:43:35+00:00
rspsna-ipa02.prod.i2x.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:43:35+00:00
rspsna-ipa01.prod.i2x.myinc.local: replica
last init status: None
last init ended: 1970-01-01 00:00:00+00:00
last update status: 0 Replica acquired successfully: Incremental update succeeded
last update ended: 2016-03-03 20:44:14+00:00
See attached file for the initial fail. Thanks very much for your help.
Devin Acosta

Post by Rob Crittenden

The important bits are needed. This part of the log is just trying to
clean things up (so failures are expected and ok). We'd really need to
see a full ipaclient-install.log.

Hmm, I'm not sure if realm name "myinc.LOCAL" is a obfuscation artifact or
real configuration. As usual, attempts to obfuscate things make debugging
harder :-)

Generally it is not a good idea to have realm != uppercase primary IPA DNS domain.

It seems that for some reason client is failing to find KDC's addresses.

Try to run kinit in debug mode. Before you try that you might need to replace
krb5.conf on the client with following values (taken form client install log):

[libdefaults]
default_realm = myinc.LOCAL
dns_lookup_realm = true
dns_lookup_kdc = true
rdns = false
ticket_lifetime = 24h
forwardable = yes
udp_preference_limit = 0
default_ccache_name = KEYRING:persistent:%{uid}

[realms]
myinc.LOCAL = {
pkinit_anchors = FILE:/etc/ipa/ca.crt

}

[domain_realm]
.prod.cloud.myinc.local = myinc.LOCAL
prod.cloud.myinc.local = myinc.LOCAL

Then you might try following command on the not-yet-enrolled host:
KRB5_TRACE=/dev/stdout kinit
'host/beanstalk01-***@myinc.LOCAL'

and see what it prints into the stdout.

Very important is to follow recommendations about DNS in the official docs:

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Linux_Domain_Identity_Authentication_and_Policy_Guide/prerequisites.html#dns-reqs

and

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/sec-Securing_DNS_Traffic_with_DNSSEC.html#sec-Recommended_Naming_Practices
(In short: do *not* invent your own names like myinc.LOCAL, ever. Just a DNS
domain you actually own, always.)

Also, section "⁠Verifying the DNS Configuration" might get handy:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Windows_Integration_Guide/trust-requirements.html#dns-realm-settings
(just skip the Active Directory parts and test only IPA)

I hope this will help.

Petr^2 Spacek

Post by d***@pabstatencio.com

Post by Rob Crittenden

Up to here is ok and expected, this is just 389-ds realizing it doesn't
have Kerberos credentials yet and obtaining them.

Post by d***@pabstatencio.com
could not perform interactive bind for id [] mech [GSSAPI]: LDAP error
-1 (Can't contact LDAP server) ((null)) errno 107 (Transport endpoint is
not connected)

$ ipa-replica-manage list -v `hostname` to see the status of the
agreements. It seems that one is unable to connect.
rob

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more i