VCSA多服务起不来，后台报证书错误 PKIX path validation failed

突然发现veeam的备份发了个备份失败的邮件，检查发现是vcsa挂掉了，登陆失败，通过网页也没法打开，各种重启服务重启vcsa系统之后都没效果，看了下日志，发现主要提示如下：

vcsa Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: validity check failed

大概是证书过期了，忘了更新了，平时都是在web端更新的，可以执行如下命令查看：

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

找了一通，发现vcsa自带了一个证书处理工具，/usr/lib/vmware-vmca/bin/certificate-manager
跑之前先查看一下VCSA的FQDN：

/usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost

使用以下命令检查当前证书的Alternative name：

/usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | grep -A1 Alternative

直接/usr/lib/vmware-vmca/bin/certificate-manager跑起：

   *** Welcome to the vSphere 6.8 Certificate Manager  ***
                   -- Select Operation --
      1. Replace Machine SSL certificate with Custom Certificate
      2. Replace VMCA Root certificate with Custom Signing
         Certificate and replace all Certificates
      3. Replace Machine SSL certificate with VMCA Certificate
      4. Regenerate a new VMCA Root Certificate and
         replace all certificates
      5. Replace Solution user certificates with
         Custom Certificate
         NOTE: Solution user certs will be deprecated in a future
         release of vCenter. Refer to release notes for more details.
      6. Replace Solution user certificates with VMCA certificates
      7. Revert last performed operation by re-publishing old
         certificates
      8. Reset all Certificates

选第8个，Y/N的地方都选Y，有几点需要注意的：

• IPAddress 如果VCSA用的是ip，这里要填，否则为空
• Hostname 这个要填vcsa的地址，如果没配置，那就是ip地址

开始处理后会有一个进度展示

Status : 60% Completed [Reset vpxd-extension Cert...]

会把各种服务的证书全部更新一遍，跑完就可以正常访问了，他会自动重启服务的，登陆后在web端查看证书情况如下：

补充

证书更新可能会失败，尤其是wcp服务会起不来，如下现象：

Error while starting services, please see service-control log for more details
Status : 0% Completed [Reset operation failed]
                 
please see /var/log/vmware/vmcad/certificate-manager.log for more information.

查看详情可以看出来是wcp启动失败了

Service-control failed. Error: Failed to start services in profile ALL. RC=1, stderr=Failed to start wcp services. Error: Operation timed out

这时候需要手动给wcp生成证书，这是vmware的一个bug，是eam导致的wcp起不来，官方说后续版本会修复。

mkdir /certificate
/usr/lib/vmware-vmafd/bin/vecs-cli entry getcert --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.crt
/usr/lib/vmware-vmafd/bin/vecs-cli entry getkey --store vpxd-extension --alias vpxd-extension --output /certificate/vpxd-extension.key
python /usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py -e com.vmware.vim.eam -c /certificate/vpxd-extension.crt -k /certificate/vpxd-extension.key -s <PNID/FQDN of vCenter Server> -u Administrator@vsphere.local

这里有个坑，有时候会报个错：

2024-08-10T02:27:19.388Z  Updating certificate for "com.vmware.vim.eam" extension
2024-08-10T02:27:19.668Z  Successfully updated certificate for "com.vmware.vim.eam" extension
Traceback (most recent call last):
  File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 175, in <module>
    update_extension_cert_in_VC()
  File "/usr/lib/vmware-vpx/scripts/updateExtensionCertInVC.py", line 163, in update_extension_cert_in_VC
    sessionMgr = si.content.sessionManager
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 577, in __call__
    return self.f(*args, **kwargs)
  File "/usr/lib/vmware/site-packages/pyVmomi/VmomiSupport.py", line 382, in _InvokeAccessor
    return self._stub.InvokeAccessor(self, info)
  File "/usr/lib/vmware/site-packages/pyVmomi/StubAdapterAccessorImpl.py", line 42, in InvokeAccessor
    return self.InvokeMethod(mo, info, (prop, ))
  File "/usr/lib/vmware/site-packages/pyVmomi/SoapAdapter.py", line 1525, in InvokeMethod
    conn.request('POST', self.path, req, headers)
  File "/usr/lib/python3.7/http/client.py", line 1277, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1323, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1272, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1032, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 972, in send
    self.connect()
  File "/usr/lib/python3.7/http/client.py", line 1447, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'sdkTunn

其实这个错可以忽略，如果看到Successfully updated certificate for "com.vmware.vim.eam" extension输出，这个错误就可以忽略不计了，因为证书已经正常处理了，这时候重启EAM服务

service-control --stop vmware-eam && service-control --stop -all

一切恢复正常！