[Freeipa-users] I think I lost my CA...

Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?

Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the past
month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried to
perform Actions->New Certificate, which did nothing. I tried each of
our 3 servers in turn. All came back with no popup window and no
error, either.
I suspect the problem might be that we no longer have a CA server due
to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm not
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

Bret Wortman

2017-04-26 12:41:54 UTC

Using the firefox debugger, I get these errors when trying to pop up the
New Certificate dialog:

Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined app.js:1:217432

I'm definitely not a web kind of guy so I'm not sure if this is helpful
or not. This is on 4.4.0, API Version 2.213.

Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried each
of our 3 servers in turn. All came back with no popup window and no
error, either.
I suspect the problem might be that we no longer have a CA server due
to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm not
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

Bret Wortman

2017-04-26 13:03:54 UTC

Digging still deeper:

# ipa cert-request f.f --principal=HTTP/`hostname`@DAMASCUSGRP.COM
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)

Looks like this is an HTTP error; so is it possible that my IPA thinks
it has a CA but there's no CMS available?

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried each
of our 3 servers in turn. All came back with no popup window and no
error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm not
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

Rob Crittenden

2017-04-26 14:22:35 UTC

Post by Bret Wortman
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)
Looks like this is an HTTP error; so is it possible that my IPA thinks
it has a CA but there's no CMS available?

Apache proxies requests to the CA so there could be a mismatch I
suppose. I'd ensure that the pki processes are running on the box for
starters and then dig into the CA debug log for more details.

rob

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried each
of our 3 servers in turn. All came back with no popup window and no
error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm not
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Bret Wortman

2017-04-26 14:45:37 UTC

Apache proxies requests to the CA so there could be a mismatch I
suppose. I'd ensure that the pki processes are running on the box for
starters and then dig into the CA debug log for more details.

Is that /var/log/pki/pki-tomcat/ca/debug? If so, then nothing happens in
it during the above operations.

As you noted, apache produces the following when trying to show a valid
cert even though there's nothing in what I think is the pki ca debug
log. ps aux shows pki processes alive, at least, and in ownership of the
8009 port (verified by lsof).

[Wed Apr 26 14:38:48.157961 2017] [:error] [pid 15801] ipa: INFO:
[jsonserver_session] ***@DAMASCUSGRP.COM: ping(): SUCCESS
[Wed Apr 26 14:38:48.247040 2017] [proxy:error] [pid 15804]
(111)Connection refused: AH00957: AJP: attempt to connect to
127.0.0.1:8009 (localhost) failed
[Wed Apr 26 14:38:48.247072 2017] [proxy:error] [pid 15804] AH00959:
ap_proxy_connect_)backend disabling worker for (localhost) for 60s
[Wed Apr 26 14:38:48.247078 2017] [proxy_ajp:error] [pid 15804] [client
192.168.208.54:56618] AH00896: failed to make connection to backend:
localhost
[Wed Apr 26 14:38:48.247531 2017] [:error] [pid 15800] ipa: ERROR:
ra.get_certificate(): Unable to communicate with CMS (503)
[Wed Apr 26 14:38:48.247765 2017] [:error] [pid 15800] ipa: INFO:
[jsonserver_session] ***@DAMASCUSGRP.COM: cert_show/1(u'895',
version=u'2.213'): CertificateOperationError

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried each
of our 3 servers in turn. All came back with no popup window and no
error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm not
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

Bret Wortman

2017-04-26 14:33:43 UTC

So I can see my certs using cert-find, but can't get details using
cert-show or add new ones using cert-request.

# ipa cert-find
:
------------------------------
Number of entries returned 385
------------------------------
# ipa cert-show 895
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)
# ipa cert-show 1 (which does not exist)
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)
# ipa cert-status 895
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)
#

Is this an IPV6 thing? Because ipactl shows everything green and
certmonger is running.

Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

Rob Crittenden

2017-04-26 22:02:06 UTC

Post by Bret Wortman
So I can see my certs using cert-find, but can't get details using
cert-show or add new ones using cert-request.
# ipa cert-find
------------------------------
Number of entries returned 385
------------------------------
# ipa cert-show 895
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)
# ipa cert-show 1 (which does not exist)
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)
# ipa cert-status 895
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)
#
Is this an IPV6 thing? Because ipactl shows everything green and
certmonger is running.

Doubtful.

cert-find and cert-show use different APIs in dogtag. cert-find uses the
newer RESTful API and cert-show uses the older XML-based API (and is
authenticated). I'm guessing that is where the issue lies.

What I'd recommend doing is noting the time, restarting the CA, and then
plow through the debug log looking for failures. It could be that the CA
is only partially up (and I'd check your CA subsystem certs as well).

rob

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Bret Wortman

2017-05-02 14:50:21 UTC

I plowed through /var/log/pki/pki-tomcat/ca/debug, but nothing jumps out
as looking like an error.

The cert-show failure is troubling, but my inability to get CSRs turned
into certs is what's actually driving this.

Bret

Doubtful.
cert-find and cert-show use different APIs in dogtag. cert-find uses the
newer RESTful API and cert-show uses the older XML-based API (and is
authenticated). I'm guessing that is where the issue lies.
What I'd recommend doing is noting the time, restarting the CA, and then
plow through the debug log looking for failures. It could be that the CA
is only partially up (and I'd check your CA subsystem certs as well).
rob

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Bret Wortman

2017-05-02 14:58:10 UTC

The closest I found was this:

[02/May/2017:14:33:57][localhost-startStop-1]: No rule can be found for
publishing: cacert
[02/May/2017:14:33:37][localhost-startStop-1]: published ca cert
[02/May/2017:14:33:37][localhost-startStop-1]: CMSEngine: ca startup done

Post by Bret Wortman
I plowed through /var/log/pki/pki-tomcat/ca/debug, but nothing jumps
out as looking like an error.
The cert-show failure is troubling, but my inability to get CSRs
turned into certs is what's actually driving this.
Bret

Doubtful.
cert-find and cert-show use different APIs in dogtag. cert-find uses the
newer RESTful API and cert-show uses the older XML-based API (and is
authenticated). I'm guessing that is where the issue lies.
What I'd recommend doing is noting the time, restarting the CA, and then
plow through the debug log looking for failures. It could be that the CA
is only partially up (and I'd check your CA subsystem certs as well).
rob

Post by Bret Wortman
# ipa cert-request f.f
ipa: ERROR: Certificate operation cannot be completed: Unable to
communicate with CMS (503)
Looks like this is an HTTP error; so is it possible that my IPA thinks
it has a CA but there's no CMS available?

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

Bret Wortman

2017-05-10 17:59:54 UTC

The log slog continues but isn't turning up anything useful, or I'm
looking in the wrong logs. Now getting twice-daily visits from users who
need new SSL certs wondering when I'm going to be able to create them.

I'm happy to do the work to figure out what went wrong, I just don't
grok these individual components at this level very well. When something
goes wrong, it's not trivial to solve. Well, for me it isn't, anyway. ;-)

Bret

Doubtful.
cert-find and cert-show use different APIs in dogtag. cert-find uses the
newer RESTful API and cert-show uses the older XML-based API (and is
authenticated). I'm guessing that is where the issue lies.
What I'd recommend doing is noting the time, restarting the CA, and then
plow through the debug log looking for failures. It could be that the CA
is only partially up (and I'd check your CA subsystem certs as well).
rob

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Bret Wortman

2017-05-18 12:43:48 UTC

Doubtful.
cert-find and cert-show use different APIs in dogtag. cert-find uses the
newer RESTful API and cert-show uses the older XML-based API (and is
authenticated). I'm guessing that is where the issue lies.
What I'd recommend doing is noting the time, restarting the CA, and then
plow through the debug log looking for failures. It could be that the CA
is only partially up (and I'd check your CA subsystem certs as well).

Which debug log, specifically, do you think will help? I'm also not sure
what you mean by, "check your CA subsystem certs." We still have pending
CSRs that we can't grant until I get this working again.

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Bret Wortman

2017-05-18 12:56:21 UTC

httpd_error seems to give the most information. When i try to use ipa
cert-show:

ipa: INFO: [jsonserver_kerb] ***@DAMASCUSGRP.COM: ping(): SUCCESS
(111)Connection refused: AH00957: AJP: attempt to connect to
127.0.0.1:8009 (localhost) failed
AH00959: ap_proxy_connect_backend disabling worker for (locahost) for 60s
[client 192.168.208.54:52714] AH00896: failed to make connection to
backend: localhost
ipa: ERROR: ra.get_certificate(): Unable to communicate with CMS (503)
ipa: INFO: [jsonserver_kerb] ***@DAMASCUSGRP.COM: cert_show/1(u'895',
version=u'2.213'): CertificateOperationError

/var/log/pki/pki-tomcat/ca/debug just loops through the same set of
messages every 5 minutes or so but doesn't seem to error.

/var/log/pki/localhost_access_log.2017-05-18.txt is basically empty
except for a single entry (for a POST to /ca/admin/ca/getStatus)

Nothing shows up in dirsrv/slapd-DAMASCUSGRP-COM/errors or access when I
issue the request, but periodic messages do appear about every 5 minutes
or so.

Doubtful.
cert-find and cert-show use different APIs in dogtag. cert-find uses the
newer RESTful API and cert-show uses the older XML-based API (and is
authenticated). I'm guessing that is where the issue lies.
What I'd recommend doing is noting the time, restarting the CA, and then
plow through the debug log looking for failures. It could be that the CA
is only partially up (and I'd check your CA subsystem certs as well).

Which debug log, specifically, do you think will help? I'm also not
sure what you mean by, "check your CA subsystem certs." We still have
pending CSRs that we can't grant until I get this working again.

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

Bret Wortman

2017-05-18 12:58:07 UTC

Oops, the slapd messages are arriving every 60s, not 5m.

Post by Bret Wortman
httpd_error seems to give the most information. When i try to use ipa
(111)Connection refused: AH00957: AJP: attempt to connect to
127.0.0.1:8009 (localhost) failed
AH00959: ap_proxy_connect_backend disabling worker for (locahost) for 60s
[client 192.168.208.54:52714] AH00896: failed to make connection to
backend: localhost
ipa: ERROR: ra.get_certificate(): Unable to communicate with CMS (503)
cert_show/1(u'895', version=u'2.213'): CertificateOperationError
/var/log/pki/pki-tomcat/ca/debug just loops through the same set of
messages every 5 minutes or so but doesn't seem to error.
/var/log/pki/localhost_access_log.2017-05-18.txt is basically empty
except for a single entry (for a POST to /ca/admin/ca/getStatus)
Nothing shows up in dirsrv/slapd-DAMASCUSGRP-COM/errors or access when
I issue the request, but periodic messages do appear about every 5
minutes or so.

Doubtful.
cert-find and cert-show use different APIs in dogtag. cert-find uses the
newer RESTful API and cert-show uses the older XML-based API (and is
authenticated). I'm guessing that is where the issue lies.
What I'd recommend doing is noting the time, restarting the CA, and then
plow through the debug log looking for failures. It could be that the CA
is only partially up (and I'd check your CA subsystem certs as well).

Which debug log, specifically, do you think will help? I'm also not
sure what you mean by, "check your CA subsystem certs." We still have
pending CSRs that we can't grant until I get this working again.

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

Florence Blanc-Renaud

2017-04-27 07:42:31 UTC

Hi Bret,

the issue looks similar to https://pagure.io/freeipa/issue/6575 and
https://pagure.io/dogtagpki/issue/2570 which were IPv6 related. Note
that IPv6 must be enabled on the machine but IPA does not require an
IPv6 address to be configured (except for the loopback).

You can check the following:
- is PKI listening to port 8009 on IPv6 or IPv4 interface?
sudo netstat -tunpl | grep 8009
tcp6 0 0 127.0.0.1:8009 :::* LISTEN 10749/java

- /etc/pki/pki-tomcat/server.xml defines a redirection from port 8009 to
8443, and the "address" part is important:
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8443"
address="localhost" />

In the above example, it will be using localhost which can resolve
either to IPv4 or IPv6.

- /etc/hosts must define the loopback addresses with
127.0.0.1 localhost localhost.localdomain localhost4
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6
localhost6.localdomain6

HTH,
Flo.

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Bret Wortman

2017-04-28 12:57:48 UTC

Post by Florence Blanc-Renaud

Flo,

I did find that issue and made those corrections to our /etc/hosts file,
but the problem persists.

Thanks for the idea!

Bret

Hi Bret,
the issue looks similar to https://pagure.io/freeipa/issue/6575 and
https://pagure.io/dogtagpki/issue/2570 which were IPv6 related. Note
that IPv6 must be enabled on the machine but IPA does not require an
IPv6 address to be configured (except for the loopback).
- is PKI listening to port 8009 on IPv6 or IPv4 interface?
sudo netstat -tunpl | grep 8009
tcp6 0 0 127.0.0.1:8009 :::* LISTEN 10749/java
- /etc/pki/pki-tomcat/server.xml defines a redirection from port 8009
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8443"
address="localhost" />
In the above example, it will be using localhost which can resolve
either to IPv4 or IPv6.
- /etc/hosts must define the loopback addresses with
127.0.0.1 localhost localhost.localdomain localhost4
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6
localhost6.localdomain6
HTH,
Flo.

Post by Bret Wortman
Using the firefox debugger, I get these errors when trying to pop up
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: u is undefined
app.js:1:362059
Empty string passed to getElementById(). (5)
jquery.js:4:1060
TypeError: t is undefined
app.js:1:217432
I'm definitely not a web kind of guy so I'm not sure if this is
helpful or not. This is on 4.4.0, API Version 2.213.
Bret

Post by Bret Wortman
Good news. One of my servers _does_ have CA installed. So why does
"Action -> New Certificate" not do anything on this or any other server?
Bret

Post by Bret Wortman
I recently had to upgrade all my Fedora IPA servers to C7. It went
well, and we've been up and running nicely on 4.4.0 on C7 for the
past month or so.
Today, someone came and asked me to generate a new certificate for
their web server. All was good until I went to the IPA UI and tried
to perform Actions->New Certificate, which did nothing. I tried
each of our 3 servers in turn. All came back with no popup window
and no error, either.
I suspect the problem might be that we no longer have a CA server
due to the method I used to upgrade the servers. I likely missed a
"--setup-ca" in there somewhere, so my rolling update rolled over the CA.
What's my best hope of recovery? I never ran this before, so I'm
# ipa ca-find
------------
1 CA matched
------------
Name: ipa
Description IPA CA
Authority ID: 3ce3346[...]
Subject DN: CN=Certificate Authority, O=DAMASCUSGRP.COM
Issuer DN: CN=Certificate Authority,O=DAMASCUSGRP.COM
----------------------------
Number of entries returned 1
----------------------------
# ipa ca-add dg --desc "Damascus Group" --subject "CN=DG CA,
O=DAMASCUSGRP.COM"
ipa: ERROR: Failed to authenticate to CA REST API
# klist
Ticket cache: KEYRING:persistent:0:0
Valid starting Expires Service principal
04/25/2017 18:48:26 04/26/2017 18:48:21
#
What's my best path of recovery?
--
*Bret Wortman*
The Damascus Group

--
Manage your subscription for the Freeipa-users mailing list:
https://www.redhat.com/mailman/listinfo/freeipa-users
Go to http://freeipa.org for more info on the project

Petr Vobornik

2017-05-04 11:31:25 UTC

Post by Bret Wortman
Flo,
I did find that issue and made those corrections to our /etc/hosts file,
but the problem persists.
Thanks for the idea!

after the change did you restart pki?

Post by Florence Blanc-Renaud

Hi Bret,
the issue looks similar to https://pagure.io/freeipa/issue/6575 and
https://pagure.io/dogtagpki/issue/2570 which were IPv6 related. Note
that IPv6 must be enabled on the machine but IPA does not require an
IPv6 address to be configured (except for the loopback).
- is PKI listening to port 8009 on IPv6 or IPv4 interface?
sudo netstat -tunpl | grep 8009
tcp6 0 0 127.0.0.1:8009 :::* LISTEN 10749/java
- /etc/pki/pki-tomcat/server.xml defines a redirection from port 8009
<Connector port="8009"
protocol="AJP/1.3"
redirectPort="8443"
address="localhost" />
In the above example, it will be using localhost which can resolve
either to IPv4 or IPv6.
- /etc/hosts must define the loopback addresses with
127.0.0.1 localhost localhost.localdomain localhost4
localhost4.localdomain4
::1 localhost localhost.localdomain localhost6
localhost6.localdomain6
HTH,
Flo.