Eagle Technologies support team has recently come across an issue which could affect multiple customers.  It seems that the vCenter Server Appliance (version 6.5 is affected) Security Token Service certificate’s expiration isn’t set for as long as it could be, and its expiration causes other service certificates to expire which can cause communications issues with the vCenter Server.  In particular the user is no longer able to log into the vSphere Client/Web Client and backups are also affected.  Below is the most recent version of the resolution document.

vCenter Server Appliance 6.5 Expired STS (Security Token Service) Certificate Fix

Example vSphere Client Errors:

 

 

 

 

 

 

Issue Explanation

The following blog post discusses the issue:

https://blogs.vmware.com/vsphere/2020/05/signing-certificate-is-not-valid-security-token-service-certificate-issue-in-vsphere.html

 

 

How to Check STS Cert Status via vSphere Web Client (Flex)

Notes:

  • Unfortunately, the H5 vSphere Client doesn’t look to have this functionality. (If you are unable to run the Flex-based Web Client, we’ve detailed the option to check the certificate status via CLI below.)
  • To view the certificate status, you will need to log in as the vSphere SSO domain admin.

Log into the vSphere Web Client as the vSphere SSO domain administrator (default is [email protected]).

 

From the home menu, select Administration.

 

 

 

 

 

 

 

 

 

 

Drill down to Single Sign-On > Configuration.  From here click on the Certificates tab and click on the STS Signing button.  Check the Valid To column to see the cert. expiration dates.

 

 

 

 

 

 

 

How to Check STS Cert Status via CLI

The following KB article shows how to verify that the STS (Security Token Service) cert has expired and was used to verify that this was the case.

https://kb.vmware.com/s/article/79248

 

Enable the BASH for the vCSA so that you can copy files via WinSCP:

https://kb.vmware.com/s/article/2107727

 

Uploaded the checksts.py script to the vCSA’s /tmp folder.

 

Results of the cechsts.py script:

[email protected] [ /tmp ]# python checksts.py

 

1 VALID CERTS

================

 

LEAF CERTS:

 

None

 

ROOT CERTS:

 

[] Certificate 44:04:1F:27:54:75:CA:98:3D:CB:3E:A5:06:B5:7F:29:D8:80:A9:7F will expire in 2911 days (7.0 years).

 

1 EXPIRED CERTS

================

 

LEAF CERTS:

 

[] Certificate: 52:80:D8:EE:70:37:7D:EB:D3:9F:31:EE:80:A5:0D:34:07:B7:25:14 expired on 2020-07-10 10:50:17 GMT!

 

ROOT CERTS:

 

None

 

WARNING!

You have expired STS certificates.  Please follow the KB corresponding to your OS:

VCSA:  https://kb.vmware.com/s/article/76719

Windows:  https://kb.vmware.com/s/article/79263

 

[email protected] [ /tmp ]#

 

vCenter Server Appliance Log Error Examples

From the messages log file:

2020-07-16T15:06:53.337906+00:00 VCENTER65 cli: vmware.appliance.vapi.auth Authorization request for service_id: com.vmware.appliance.health.data                                                       basestorage, operation_id: get

2020-07-16T15:06:53.339470+00:00 VCENTER65 cli: root SSO initialization error: [Errno 111] Connection refused

2020-07-16T15:06:53.339805+00:00 VCENTER65 cli: root Authorization module (authorization_sso) failed to initialize {[Errno 111] Connection refuse                                                       d}

 

From the vpxd.log:

2020-07-16T15:28:51.010Z error vpxd[7F62416E0700] [[email protected] sub=LSClient] Caught exception while creating LS client adapter: N7Vmacore3Ssl18SSLVerifyExceptionE(SSL Exception: Verification parameters:

–> PeerThumbprint: C6:58:4F:58:0E:62:E8:EB:78:51:53:47:C1:A4:C5:8A:EB:64:91:7E

–> ExpectedThumbprint:

–> ExpectedPeerName: VCENTER65.domain.com

–> The remote host certificate has these problems:

–>

–> * certificate has expired)

–> [context]zKq7AVECAAAAAPdJxAANdnB4ZAAATHorbGlidm1hY29yZS5zbwAAHiQbAD5yGABe8RsA7XAiAPg9IgAvQiIAn/kjAAvFIwDyxyMAA9MrAdRzAGxpYnB0aHJlYWQuc28uMAACvY4ObGliYy5zby42AA==[/context]

 

2020-07-16T15:28:51.013Z warning vpxd[7F62416E0700] [[email protected] sub=LSClient] Endpoint not found for Product: com.vmware.cis, Type: cs.identity, EndPointType:  com.vmware.cis.cs.identity.admin

2020-07-16T15:28:51.013Z info vpxd[7F62416E0700] [[email protected] sub=HostGateway] stsUrlFromLs:  ssoAdminUrlFromLs:

2020-07-16T15:28:51.026Z info vpxd[7F62416E0700] [[email protected] sub=[SSO][SsoCertificateManagerImpl]] Try to connect to SSO VMOMI endpoint

2020-07-16T15:28:51.075Z error vpxd[7F62416E0700] [[email protected] sub=HostGateway] [CisConnection]: Error getting trusted STS certificates: vmodl.fault.SystemError

2020-07-16T15:28:51.075Z warning vpxd[7F62416E0700] [[email protected] sub=HostGateway] State(ST_INIT) failed with: vmodl.fault.SystemError

 

Issue Solution

VMware’s STS Cert Issue Fix:

https://kb.vmware.com/s/article/76719

 

Note: before making any changes to the vCenter Server Appliance, we strongly recommend taking a VM snapshot of the VM.

 

In the below examples, the fixsts.sh script was uploaded to the vCSA’s /tmp folder.  (Note: you will not need to later manually delete the script files as the next reboot of the vCSA should do this automatically.)

 

Allow the fixsts.sh script to execute:

[email protected] [ /tmp ]# chmod +x fixsts.sh

[email protected] [ /tmp ]#

 

Run the fixsts.sh script:

[email protected] [ /tmp ]# ./fixsts.sh

NOTE: This works on external and embedded PSCs

This script will do the following

1: Regenerate STS certificate

What is needed?

1: Offline snapshots of VCs/PSCs

2: SSO Admin Password

IMPORTANT: This script should only be run on a single PSC per SSO domain

==================================

Resetting STS certificate for VCENTER65.domain.com started on Thu Jul 16 11:03:23 CDT 2020

 

 

Detected DN: cn=VCENTER65.domain.com,ou=Domain Controllers,dc=vsphere,dc=local

Detected PNID: VCENTER65.domain.com

Detected PSC: VCENTER65.domain.com

Detected SSO domain name: vsphere.local

Detected Machine ID: 31d5a9f9-0258-4281-88b2-ddbbc90a59e3

Detected IP Address: 192.168.1.75

Domain CN: dc=vsphere,dc=local

==================================

==================================

 

Detected Root’s certificate expiration date: 2028 Jul 5

Detected today’s date: 2020 Jul 16

==================================

 

Exporting and generating STS certificate

 

Status : Success

Using config file : /tmp/vmware-fixsts/certool.cfg

Status : Success

 

Enter password for [email protected]:

Amount of tenant credentials: 1

Exporting tenant and trustedcertchain 1 to /tmp/vmware-fixsts

 

Deleting tenant and trustedcertchain 1

 

Applying newly generated STS certificate to SSO domain

adding new entry “cn=TenantCredential-1,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local”

 

adding new entry “cn=TrustedCertChain-1,cn=TrustedCertificateChains,cn=vsphere.local,cn=Tenants,cn=IdentityManager,cn=Services,dc=vsphere,dc=local”

 

 

Replacement finished – Please restart services on all vCenters and PSCs in your SSO domain

==================================

IMPORTANT: In case you’re using HLM (Hybrid Linked Mode) without a gateway, you would need to re-sync the certs from Cloud to On-Prem after following this procedure

==================================

==================================

[email protected] [ /tmp ]#

 

Stop and then start all vCSA services:

service-control –stop –all
service-control –start –all

 

You can re-run the checksts.py script to verify that all is well with the STS cert:

[email protected] [ /tmp ]# python checksts.py

 

2 VALID CERTS

================

 

LEAF CERTS:

 

[] Certificate 62:48:1D:99:86:83:A3:54:90:15:67:D1:D4:81:0C:FD:A4:6E:F3:C0 will expire in 730 days (2.0 years).

 

ROOT CERTS:

 

[] Certificate 44:04:1F:27:54:75:CA:98:3D:CB:3E:A5:06:B5:7F:29:D8:80:A9:7F will expire in 2911 days (7.0 years).

 

0 EXPIRED CERTS

================

 

LEAF CERTS:

 

None

 

ROOT CERTS:

 

None

[email protected] [ /tmp ]#

 

Note: in some cases, the service load can hang after the start of the vmware-psc-client service and then show the following:

Service-control failed. Error Failed to start vmon services.vmon-cli RC=1, stderr=Failed to start sca, cm, vpxd-svcs, statsmonitor, vapi-endpoint services. Error: Operation timed out

 

If the STS certificate has expired, you will need to also check the other vCSA service certificates.

 

Per KB76719, run the following command to check for other expired or soon-to-expire certs:

for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list –store $i –text | egrep “Alias|Not After”; done

 

[email protected] [ /tmp ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list –store $i –text | egrep “Alias|Not After”; done

STORE MACHINE_SSL_CERT

Alias : __MACHINE_CERT

Not After : Jul 10 22:59:58 2020 GMT

STORE TRUSTED_ROOTS

Alias : 44041f275475ca983dcb3ea506b57f29d880a97f

Not After : Jul  5 10:59:57 2028 GMT

Alias : 6a8c55c1e5eb02734be202eba6b1f20f486ba91a

Not After : Jul 11 15:40:20 2030 GMT

Alias : f9a3fd4684cd4dd098d69304d38dc0d58bc918ed

Not After : Jul 11 15:43:59 2030 GMT

STORE TRUSTED_ROOT_CRLS

Alias : c60d3978014dc591d029c8e44197acb6f01922d6

Alias : 19c903b62b71f6b5be847fdd87902f09ed9ccd8d

Alias : 9fcf3c287f56b4a7cfc1695ffa0217fb204597d4

STORE machine

Alias : machine

Not After : Jul 10 10:51:14 2020 GMT

STORE vsphere-webclient

Alias : vsphere-webclient

Not After : Jul 10 10:51:14 2020 GMT

STORE vpxd

Alias : vpxd

Not After : Jul 10 10:51:15 2020 GMT

STORE vpxd-extension

Alias : vpxd-extension

Not After : Jul 10 10:51:15 2020 GMT

STORE SMS

Alias : sms_self_signed

Not After : Jul 11 16:09:10 2028 GMT

STORE BACKUP_STORE

Alias : bkp___MACHINE_CERT

Not After : Jul 10 22:59:58 2020 GMT

Alias : bkp_machine

Not After : Jul 10 10:51:14 2020 GMT

Alias : bkp_vsphere-webclient

Not After : Jul 10 10:51:14 2020 GMT

Alias : bkp_vpxd

Not After : Jul 10 10:51:15 2020 GMT

Alias : bkp_vpxd-extension

Not After : Jul 10 10:51:15 2020 GMT

[email protected] [ /tmp ]#

 

In the above example, the vCSA’s certs needed to be regenerated.  The following VMware KB article was used to regenerate the certs:

https://kb.vmware.com/s/article/2112283

 

Option 4 (“Regenerate a new VMCA Root Certificate and replace all certificates”) was selected.  With the exception of the vCenter SSO admin password and the vCenter Server-specific information (IPAddress, Hostname, and VMCA Name), all of the default values were selected.  If you would like, please feel free to set the other values as desired.

 

Note: for the ‘Hostname’ and VMCA ‘Name’ fields, use the vCenter Server Appliances FQDN.

 

After the certificate re-generation and replacement was completed, I check for expired certs again to verify that they wouldn’t be expiring anytime soon.  Note: the certs below with the expired dates are the backup (“bkp_” prefix) certificates.

[email protected] [ /tmp ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list –store $i –text | egrep “Alias|Not After”; done

STORE MACHINE_SSL_CERT

Alias : __MACHINE_CERT

Not After : Jul 16 16:11:23 2022 GMT

STORE TRUSTED_ROOTS

Alias : 44041f275475ca983dcb3ea506b57f29d880a97f

Not After : Jul  5 10:59:57 2028 GMT

Alias : 6a8c55c1e5eb02734be202eba6b1f20f486ba91a

Not After : Jul 11 15:40:20 2030 GMT

Alias : f9a3fd4684cd4dd098d69304d38dc0d58bc918ed

Not After : Jul 11 15:43:59 2030 GMT

Alias : d37eed403326433f5d867ec6b0ace030c5b8dffe

Not After : Jul 11 16:21:22 2030 GMT

STORE TRUSTED_ROOT_CRLS

Alias : c60d3978014dc591d029c8e44197acb6f01922d6

Alias : 19c903b62b71f6b5be847fdd87902f09ed9ccd8d

Alias : 9fcf3c287f56b4a7cfc1695ffa0217fb204597d4

Alias : 4afba1f60d1f43f5afec99574ea02444eb3a11cf

STORE machine

Alias : machine

Not After : Jul 16 16:13:26 2022 GMT

STORE vsphere-webclient

Alias : vsphere-webclient

Not After : Jul 16 16:13:27 2022 GMT

STORE vpxd

Alias : vpxd

Not After : Jul 16 16:13:27 2022 GMT

STORE vpxd-extension

Alias : vpxd-extension

Not After : Jul 16 16:13:28 2022 GMT

STORE SMS

Alias : sms_self_signed

Not After : Jul 11 16:09:10 2028 GMT

STORE BACKUP_STORE

Alias : bkp___MACHINE_CERT

Not After : Jul 10 22:59:58 2020 GMT

Alias : bkp_machine

Not After : Jul 10 10:51:14 2020 GMT

Alias : bkp_vsphere-webclient

Not After : Jul 10 10:51:14 2020 GMT

Alias : bkp_vpxd

Not After : Jul 10 10:51:15 2020 GMT

Alias : bkp_vpxd-extension

Not After : Jul 10 10:51:15 2020 GMT

[email protected] [ /tmp ]#

 

If all is now well with your vCenter Server Appliance, delete the snapshot that was taken before the above changes were made.

If you have additional questions, please reach out to Eagle Technologies support at [email protected] or 800.477.5432.