Pivotal Knowledge Base

Follow

ERT upgrade results in an error: "Failed Jobs: cloud_controller_ng"

Environment 

Pivotal Elastic Runtime: 1.10.0 to 1.10.12

Symptom

Upgrading Elastic Runtime to versions ranging from 1.10.0 to 1.10.12 throws an error:

Started updating instance cloud_controller > cloud_controller/1d84c557-6820-411b-a53e-ece92293e6f6 (0) (canary). 
Failed: Action Failed get_task: Task 27952c11-4624-4b20-7253-52f0f66f7ed3 result: 1 of 6 pre-start scripts failed. Failed Jobs: cloud_controller_ng.

To get more information on the failed task, execute bosh task <taskid> --debug. It will return the following errors:

[2017-06-25 03:38:31+0000] Sequel::Migrator::Error: Applied migration files not in file system: 20170502171127_remove_orphaned_apps.rb, 20170502181209_add_fk_apps_space_guid.rb 
[2017-06-25 03:38:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/sequel-4.41.0/lib/sequel/extensions/migration.rb:716:in `get_applied_migrations'
[2017-06-25 03:38:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/sequel-4.41.0/lib/sequel/extensions/migration.rb:672:in `initialize'
[2017-06-25 03:38:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/sequel-4.41.0/lib/sequel/extensions/migration.rb:401:in `new'
[2017-06-25 03:38:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/sequel-4.41.0/lib/sequel/extensions/migration.rb:401:in `run'
[2017-06-25 03:38:31+0000] /var/vcap/data/packages/cloud_controller_ng/0a1ab6a548e55b64c2d66a0f8207db3bfc780022/cloud_controller_ng/lib/cloud_controller/db_migrator.rb:18:in `apply_migrations'
[2017-06-25 03:38:31+0000] /var/vcap/data/packages/cloud_controller_ng/0a1ab6a548e55b64c2d66a0f8207db3bfc780022/cloud_controller_ng/lib/tasks/db.rake:47:in `migrate'
[2017-06-25 03:38:31+0000] /var/vcap/data/packages/cloud_controller_ng/0a1ab6a548e55b64c2d66a0f8207db3bfc780022/cloud_controller_ng/lib/tasks/db.rake:52:in `block (2 levels) in <top (required)>'
[2017-06-25 03:38:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.3.0/gems/rake-11.3.0/exe/rake:27:in `<top (required)>'
[2017-06-25 03:38:31+0000] /var/vcap/packages/ruby-2.3/bin/bundle:22:in `load'
[2017-06-25 03:38:31+0000] /var/vcap/packages/ruby-2.3/bin/bundle:22:in `<main>'
[2017-06-25 03:38:31+0000] Tasks: TOP => db:migrate
[2017-06-25 03:38:31+0000] (See full trace by running task with --trace)
[2017-06-25 03:38:31+0000] + result=1
[2017-06-25 03:38:31+0000] + [[ 1 -eq 0 ]]
[2017-06-25 03:38:31+0000] + sleep 5
[2017-06-25 03:38:36+0000] + (( counter++ ))
[2017-06-25 03:38:36+0000] + [[ 3 -le 3 ]]
[2017-06-25 03:38:36+0000] + echo 'Running migration try number 3 of 3'
[2017-06-25 03:38:36+0000] + bundle exec rake db:migrate
[2017-06-25 03:38:39+0000] rake aborted!

Cause

This is an existing bug that was resolved in 1.10.13

Resolution

Upgrade to Elastic Runtime version 1.10.13 or higher to fix this issue.

 

 

Comments

Powered by Zendesk