Pivotal Knowledge Base

Follow

Errand push-pivotal-account fails during update

Environment

Pivotal Cloud Foundry® (PCF) 1.10.x

Symptom

When attempting an upgrade from Elastic Runtime 1.9.x to 1.10.x, errand push-pivotal-account fails.

Running errand manually also fails with error message: 

FAILED
Start app timeout

TIP: Application must be listening on the right port. Instead of hard coding the port, use the $PORT environment variable.

Use 'cf logs pivotal-account-green --recent' for more information

Check `cf logs pivotal-account-green` output and see the error message as below: 

2017-04-12T01:31:15.17-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.177 INFO 15 --- [ost-startStop-1] o.f.core.internal.util.VersionPrinter : Flyway 3.2.1 by Boxfuse
2017-04-12T01:31:15.37-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.378 INFO 15 --- [ost-startStop-1] o.f.c.i.dbsupport.DbSupportFactory : Database: jdbc:mysql://172.20.250.18:3306/account?verifyServerCertificate=false&useSSL=false&requireSSL=false (MySQL 5.5)
2017-04-12T01:31:15.41-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.418 INFO 15 --- [ost-startStop-1] o.f.core.internal.command.DbValidate : Validated 2 migrations (execution time 00:00.023s)
2017-04-12T01:31:15.45-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.456 INFO 15 --- [ost-startStop-1] o.f.core.internal.command.DbMigrate : Current version of schema `account`: 1
2017-04-12T01:31:15.45-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.459 ERROR 15 --- [ost-startStop-1] o.s.b.c.embedded.tomcat.TomcatStarter : Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'org.springframework.session.jdbc.config.annotation.web.http.JdbcHttpSessionConfiguration': Unsatisfied dependency expressed through method 'setCookieSerializer' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'application': Invocation of init method failed; nested exception is org.flywaydb.core.api.FlywayException: Schema `account` contains a failed migration to version 1 !
2017-04-12T01:31:15.49-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.494 WARN 15 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT 2017-04-12 05:31:15.529 ERROR 15 --- [ main] o.s.boot.SpringApplication : Application startup failed
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT org.springframework.context.ApplicationContextException: Unable to start embedded container; nested exception is org.springframework.boot.context.embedded.EmbeddedServletContainerException: Unable to start embedded Tomcat
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.onRefresh(EmbeddedWebApplicationContext.java:137) ~[spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:536) ~[spring-context-4.3.4.RELEASE.jar!/:4.3.4.RELEASE]
2017-04-12T01:31:15.53-0400 [APP/PROC/WEB/0]OUT at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:122) ~[spring-boot-1.4.2.RELEASE.jar!/:1.4.2.RELEASE]

Cause

The account database is somewhat corrupted. 

Resolution

The current workaround is to recreate the account database.

Here are the list of steps: 

1.) In Ops Manager, open the Elastic Runtime tile, go to credentials tab, and get the credentials for the MySQL Admin, the username should be root and the password will be random. Save these as you'll need them in step 5.

2.) Log into your Ops Manager VM, or whichever VM you use to execute bosh commands. Make sure you're logged in as director.

3.) Verify you're targeting your cf-*.yml deployment by running bosh deployment, if you're not targeting it, you can run bosh deployment /var/tempest/workspaces/default/deployments/cf-*.yml

4.) bosh ssh into one of of your mysql VM nodes, not the proxy or replication canary

5.) Run mysql -u root -p account to connect to the account DB. If the mysql command doesn't work, you might have to use the -h localhost flag to make sure it picks up the host is the mysql VM. After you execute the command, you'll need the root password you obtained in step 1.).

6.) After you login successfully, you can run show tables; To list all tables in the account DB. You will see something similar to this:

MariaDB [account]> show tables;
+---------------------------+
| Tables_in_account |
+---------------------------+
| SPRING_SESSION |
| SPRING_SESSION_ATTRIBUTES |
| schema_version |
| zone_client |
+---------------------------+

7.) Now that you have the list, you now need to drop them by running drop table if exists <table1>,<table2>,<table3>;, place the table names in step 6 into the command. Example:

MariaDB [account]> drop table if exists SPRING_SESSION, SPRING_SESSION_ATTRIBUTES, schema_version, zone_client;

8.) Now that you have dropped all the tabled, exit mysql, exit the mysql VM, then you're back on the Ops Manager, run bosh run errand push-pivotal-account again. This time it should be successful.

Additional Information

Please be noted that this operation won't cause any user data loss. 

 

Comments

Powered by Zendesk