How to correctly change database connection password?

[mwaag]

Hi,

i tried to mail to [email protected], but unfortunately my mails don't seem to show up. So i try it that way.

Iam currently trying to change database-passwords for cloudstack-usage and cloudstack-management-service (hint: they are the same).

Cloudstack-Version is: 4.18.2.4
MySQL-Versioni is: 5.7.44
Hypervisor: VMWare 7.0.3

According to https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235837182 (Paragraph 6 - EncryptionCLI in cloudstack-utils.jar)

• I stop all services
  systemctl stop cloudstack-management
  systemctl stop cloudstack-usage

• create a new secret with:
  java -classpath /usr/share/cloudstack-common/lib/cloudstack-utils.jar com.cloud.utils.crypt.EncryptionCLI -p <my mgmt key from /etc/cloudstack/management/key> -i newpassword
  Output:
  log4j:WARN No appenders could be found for logger (com.cloud.utils.crypt.CloudStackEncryptor).
  log4j:WARN Please initialize the log4j system properly.
  log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
  WARNING: An illegal reflective access operation has occurred
  WARNING: Illegal reflective access by com.google.protobuf.UnsafeUtil (file:/usr/share/cloudstack-common/lib/cloudstack-utils.jar) to field java.nio.Buffer.address
  WARNING: Please consider reporting this to the maintainers of com.google.protobuf.UnsafeUtil
  WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  WARNING: All illegal access operations will be denied in a future release
  3ZH7CjJqxpH7ZFvPJRBBOKvO/sS/3ByeE55YURu1COX9y5qM3Hun

• update my db.properties file:
  db.usage.password=ENC(3ZH7CjJqxpH7ZFvPJRBBOKvO/sS/3ByeE55YURu1COX9y5qM3Hun)
  db.cloud.password=ENC(3ZH7CjJqxpH7ZFvPJRBBOKvO/sS/3ByeE55YURu1COX9y5qM3Hun)

• change the password of the according user in the database (via MySQL-Workbench) + flush privileges

• manually test connection with
  mysql -u -p -h
  ==> Connection successful!

• start cloudstack-usage-service
  systemctl start cloudstack-usage

• Result in cloudstack-usage-logs:
  journalctl -u cloudstack-usage

Apr 08 10:49:17 cloudstack0-b systemd[1]: Started CloudStack Usage Server.
Apr 08 10:49:17 cloudstack0-b sh[10223]: 08:49:17,995  INFO LogUtils:57 - log4j configuration found at /etc/cloudstack/usage/log4j-cloud.xml
Apr 08 10:49:18 cloudstack0-b sh[10223]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Apr 08 10:49:18 cloudstack0-b sh[10223]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Apr 08 10:49:18 cloudstack0-b sh[10223]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Apr 08 10:49:18 cloudstack0-b sh[10223]: WARNING: An illegal reflective access operation has occurred
Apr 08 10:49:18 cloudstack0-b sh[10223]: WARNING: Illegal reflective access by net.sf.cglib.core.ReflectUtils$1 (file:/usr/share/cloudstack-usage/lib/cglib-nodep-3.3.0.jar) to method java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int,java.security.ProtectionDomain)
Apr 08 10:49:18 cloudstack0-b sh[10223]: WARNING: Please consider reporting this to the maintainers of net.sf.cglib.core.ReflectUtils$1
Apr 08 10:49:18 cloudstack0-b sh[10223]: WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
Apr 08 10:49:18 cloudstack0-b sh[10223]: WARNING: All illegal access operations will be denied in a future release
Apr 08 10:49:19 cloudstack0-b sh[10223]: 08:49:19,078  INFO Transaction:1027 - Is Data Base High Availiability enabled? Ans : false
Apr 08 10:49:19 cloudstack0-b sh[10223]: Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary.
Apr 08 10:49:20 cloudstack0-b sh[10223]: 08:49:20,757  INFO ComponentContext:60 - Setup Spring Application context
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,600  INFO ComponentContext:115 - Configuring org.apache.cloudstack.framework.config.dao.ConfigurationDaoImpl_EnhancerByCloudStack_6b3e2359
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,601  INFO ComponentContext:115 - Configuring com.cloud.storage.dao.SnapshotDaoImpl_EnhancerByCloudStack_bf7884b3
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,601  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_7573a7bf
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,602  INFO ComponentContext:115 - Configuring com.cloud.storage.dao.VolumeDaoImpl_EnhancerByCloudStack_15c1582d
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,602  INFO ComponentContext:115 - Configuring com.cloud.user.dao.SSHKeyPairDaoImpl_EnhancerByCloudStack_4ef1487a
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,602  INFO ComponentContext:115 - Configuring com.cloud.dc.dao.DataCenterDaoImpl_EnhancerByCloudStack_465ac8b0
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,607  INFO ComponentContext:115 - Configuring com.cloud.dc.dao.ClusterDaoImpl_EnhancerByCloudStack_47b3697
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,607  INFO ComponentContext:115 - Configuring org.apache.cloudstack.storage.datastore.db.ImageStoreDaoImpl_EnhancerByCloudStack_3550535
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,609  INFO ComponentContext:115 - Configuring com.cloud.vm.dao.NicDaoImpl_EnhancerByCloudStack_e3106961
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,609  INFO ComponentContext:115 - Configuring com.cloud.event.dao.EventDaoImpl_EnhancerByCloudStack_1a861650
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,610  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageLoadBalancerPolicyDaoImpl_EnhancerByCloudStack_8b734d94
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,610  INFO ComponentContext:115 - Configuring com.cloud.user.dao.UserAccountDaoImpl_EnhancerByCloudStack_a5587143
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,610  INFO ComponentContext:115 - Configuring com.cloud.host.dao.HostTagsDaoImpl_EnhancerByCloudStack_a3ab328f
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,611  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageStorageDaoImpl_EnhancerByCloudStack_ef630ac3
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,613  INFO ComponentContext:115 - Configuring com.cloud.configuration.dao.ResourceCountDaoImpl_EnhancerByCloudStack_f8484535
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,614  INFO ComponentContext:115 - Configuring org.apache.cloudstack.quota.dao.QuotaBalanceDaoImpl_EnhancerByCloudStack_f3cb37e1
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,614  INFO ComponentContext:115 - Configuring org.apache.cloudstack.storage.datastore.db.PrimaryDataStoreDaoImpl_EnhancerByCloudStack_ea40352e
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,616  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageVMInstanceDaoImpl_EnhancerByCloudStack_5a097486
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,617  INFO ComponentContext:115 - Configuring com.cloud.service.dao.ServiceOfferingDetailsDaoImpl_EnhancerByCloudStack_73fdd9a2
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,617  INFO ComponentContext:115 - Configuring com.cloud.user.dao.AccountDaoImpl_EnhancerByCloudStack_11ca5a80
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,617  INFO ComponentContext:115 - Configuring com.cloud.dc.dao.DataCenterIpAddressDaoImpl_EnhancerByCloudStack_68d85503
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,617  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageJobDaoImpl_EnhancerByCloudStack_9cabe0b1
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,617  INFO ComponentContext:115 - Configuring org.apache.cloudstack.quota.dao.QuotaTariffDaoImpl_EnhancerByCloudStack_24533da7
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,618  INFO ComponentContext:115 - Configuring com.cloud.storage.dao.StoragePoolHostDaoImpl_EnhancerByCloudStack_182ad59a
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,618  INFO ComponentContext:115 - Configuring com.cloud.network.dao.AccountGuestVlanMapDaoImpl_EnhancerByCloudStack_cf52cd48
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,618  INFO ComponentContext:115 - Configuring com.cloud.event.dao.UsageEventDetailsDaoImpl_EnhancerByCloudStack_bf4c37c9
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,618  INFO ComponentContext:115 - Configuring com.cloud.offerings.dao.NetworkOfferingDaoImpl_EnhancerByCloudStack_4a7a7a3b
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,618  INFO ComponentContext:115 - Configuring com.cloud.dc.ClusterDetailsDaoImpl_EnhancerByCloudStack_6233dc7b
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,618  INFO ComponentContext:115 - Configuring org.apache.cloudstack.framework.config.dao.ConfigurationSubGroupDaoImpl_EnhancerByCloudStack_ed937802
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,619  INFO ComponentContext:115 - Configuring com.cloud.service.dao.ServiceOfferingDaoImpl_EnhancerByCloudStack_cbe5e214
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,619  INFO ComponentContext:115 - Configuring org.apache.cloudstack.acl.dao.RoleDaoImpl_EnhancerByCloudStack_a64c7141
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,619  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_3a35c008
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,619  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageNetworkDaoImpl_EnhancerByCloudStack_f91523d8
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,619  INFO ComponentContext:115 - Configuring com.cloud.user.dao.UserDaoImpl_EnhancerByCloudStack_b2849510
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,619  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageNetworkOfferingDaoImpl_EnhancerByCloudStack_eb5fbf42
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,620  INFO ComponentContext:115 - Configuring com.cloud.storage.dao.GuestOSDaoImpl_EnhancerByCloudStack_10f2ae15
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,620  INFO ComponentContext:115 - Configuring com.cloud.vm.dao.VMInstanceDaoImpl_EnhancerByCloudStack_b746a533
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,620  INFO ComponentContext:115 - Configuring com.cloud.dc.dao.DataCenterVnetDaoImpl_EnhancerByCloudStack_c78d35d1
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,620  INFO ComponentContext:115 - Configuring org.apache.cloudstack.quota.dao.QuotaCreditsDaoImpl_EnhancerByCloudStack_eea2bf3
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,620  INFO ComponentContext:115 - Configuring org.apache.cloudstack.resourcedetail.dao.DiskOfferingDetailsDaoImpl_EnhancerByCloudStack_91d54433
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,621  INFO ComponentContext:115 - Configuring com.cloud.cluster.agentlb.dao.HostTransferMapDaoImpl_EnhancerByCloudStack_4513e0b4
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,621  INFO ComponentContext:115 - Configuring org.apache.cloudstack.backup.dao.BackupOfferingDaoImpl_EnhancerByCloudStack_abcad5f1
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,621  INFO ComponentContext:115 - Configuring com.cloud.user.dao.UserStatsLogDaoImpl_EnhancerByCloudStack_f9080543
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,621  INFO ComponentContext:115 - Configuring com.cloud.dc.dao.PodVlanDaoImpl_EnhancerByCloudStack_70cf2699
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,621  INFO ComponentContext:115 - Configuring com.cloud.event.dao.UsageEventDaoImpl_EnhancerByCloudStack_5c8e11cd
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,621  INFO ComponentContext:115 - Configuring com.cloud.domain.dao.DomainDaoImpl_EnhancerByCloudStack_ec0c3270
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,622  INFO ComponentContext:115 - Configuring com.cloud.vm.dao.UserVmDetailsDaoImpl_EnhancerByCloudStack_35cf2469
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,622  INFO ComponentContext:115 - Configuring com.cloud.tags.dao.ResourceTagsDaoImpl_EnhancerByCloudStack_af8c1572
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,622  INFO ComponentContext:115 - Configuring com.cloud.host.dao.HostDetailsDaoImpl_EnhancerByCloudStack_c6e20c6
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,622  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageVmDiskDaoImpl_EnhancerByCloudStack_5a07669e
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,622  INFO ComponentContext:115 - Configuring com.cloud.user.dao.UserStatisticsDaoImpl_EnhancerByCloudStack_76994315
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,622  INFO ComponentContext:115 - Configuring org.apache.cloudstack.storage.datastore.db.SnapshotDataStoreDaoImpl_EnhancerByCloudStack_4fc7b0b2
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,628  INFO ComponentContext:115 - Configuring org.apache.cloudstack.quota.dao.QuotaUsageDaoImpl_EnhancerByCloudStack_3918f1d4
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,628  INFO ComponentContext:115 - Configuring com.cloud.dc.dao.HostPodDaoImpl_EnhancerByCloudStack_61de15c
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,628  INFO ComponentContext:115 - Configuring com.cloud.usage.dao.UsageVPNUserDaoImpl_EnhancerByCloudStack_4bdf2bbf
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,628  INFO ComponentContext:115 - Configuring org.apache.cloudstack.framework.config.dao.ConfigurationGroupDaoImpl_EnhancerByCloudStack_6ddc1cf8
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,629  INFO ComponentContext:115 - Configuring com.cloud.offerings.dao.NetworkOfferingServiceMapDaoImpl_EnhancerByCloudStack_ec760f9c
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,629  INFO ComponentContext:115 - Configuring com.cloud.storage.dao.StoragePoolTagsDaoImpl_EnhancerByCloudStack_ae0c19c1
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,629  INFO ComponentContext:115 - Configuring org.apache.cloudstack.quota.dao.QuotaEmailTemplatesDaoImpl_EnhancerByCloudStack_5580fd26
Apr 08 10:49:21 cloudstack0-b sh[10223]: 08:49:21,629  INFO ComponentContext:115 - Configuring com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4
Apr 08 10:49:25 cloudstack0-b sh[10223]: Exception in thread "main" com.cloud.utils.exception.CloudRuntimeException: DB Exception on: null
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1228)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4.CGLIB$expunge$44(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4_FastClassByCloudStack_d2d37d15.invoke(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4.expunge(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.db.GenericDaoBase.remove(GenericDaoBase.java:1893)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4.CGLIB$remove$9(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4_FastClassByCloudStack_d2d37d15.invoke(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4.remove(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl.configure(DataCenterLinkLocalIpAddressDaoImpl.java:189)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4.CGLIB$configure$0(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4_FastClassByCloudStack_d2d37d15.invoke(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:122)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.dc.dao.DataCenterLinkLocalIpAddressDaoImpl_EnhancerByCloudStack_236811f4.configure(<generated>)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.component.ComponentContext.initComponentsLifeCycle(ComponentContext.java:123)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.usage.UsageServer.start(UsageServer.java:54)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.usage.UsageServer.main(UsageServer.java:41)
Apr 08 10:49:25 cloudstack0-b sh[10223]: Caused by: java.sql.SQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:89)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:63)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:73)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:906)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:831)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at org.apache.commons.dbcp2.DriverManagerConnectionFactory.createConnection(DriverManagerConnectionFactory.java:121)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at org.apache.commons.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:374)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:918)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:431)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:356)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at org.apache.commons.dbcp2.PoolingDataSource.getConnection(PoolingDataSource.java:141)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.db.TransactionLegacy.getConnection(TransactionLegacy.java:555)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.db.TransactionLegacy.prepareStatement(TransactionLegacy.java:466)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.db.TransactionLegacy.prepareAutoCloseStatement(TransactionLegacy.java:459)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1221)
Apr 08 10:49:25 cloudstack0-b sh[10223]: ... 20 more
Apr 08 10:49:25 cloudstack0-b sh[10223]: Caused by: com.mysql.cj.exceptions.CJException: Access denied for user 'cloud'@'192.168.0.90' (using password: YES)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:61)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:105)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:151)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.exceptions.ExceptionFactory.createException(ExceptionFactory.java:129)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:787)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:712)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:680)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.protocol.a.NativeProtocol.checkErrorMessage(NativeProtocol.java:128)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.java:540)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:202)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1340)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.NativeSession.connect(NativeSession.java:157)
Apr 08 10:49:25 cloudstack0-b sh[10223]: at com.mysql.cj.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:850)
Apr 08 10:49:25 cloudstack0-b sh[10223]: ... 36 more
Apr 08 10:49:26 cloudstack0-b systemd[1]: cloudstack-usage.service: main process exited, code=exited, status=1/FAILURE
Apr 08 10:49:26 cloudstack0-b systemd[1]: Unit cloudstack-usage.service entered failed state.
Apr 08 10:49:26 cloudstack0-b systemd[1]: cloudstack-usage.service failed.

And this goes on and on until i stop cloudstack-usage service again.

Restoring the old values in db.properties and changing back the database password in the database will restore functionality.

Am I doing something wrong? Does anybody else have experience with this? How can i investigate this further?

Thanks in advance
Markus waag

[weizhouapache]

@mwaag
did you face the same issue with cloudstack-management service ?

[mwaag]

To be honest, I hadn't tested that far because I assumed I would get the same result there. But I just caught up on it, and indeed, I don't have the problem there. Even the usage service starts again without issues after starting the management service. Is that intended behaviour?

[weizhouapache]

To be honest, I hadn't tested that far because I assumed I would get the same result there. But I just caught up on it, and indeed, I don't have the problem there. Even the usage service starts again without issues after starting the management service. Is that intended behaviour?

@mwaag
I think usage server does not rely on management server, although they uses the same db configuration
/etc/cloudstack/usage/db.properties -> /etc/cloudstack/management/db.properties

everything should work. I could not think out a possible issue, except that /etc/cloudstack/usage/db.properties is not linked to /etc/cloudstack/management/db.properties

[mwaag]

@weizhouapache
I wasn't aware of /etc/cloudstack/usage before, but db.properties and key are linked to /etc/cloudstack/management as you suspect.

"Magically", it works now. Human error cannot be ruled out ;) Thanks for your help. Issue can be closed.