Pivotal Knowledge Base


Ops Manager Director is Down Following an Upgrade Where Migration Scripts fail


Pivotal Cloud Foundry Version 1.10


You see the following issues in Ops Manager:

  • Apply changes will fail with dial tcp <director-ip>:22: getsockopt: connection timed out
  • The Status page for the Ops Manager Director Tile will not show any data
  • bosh login will fail saying it cannot access the director

Also, in the Bosh Director logs, you will see the following error like:

E, [2017-05-26 18:27:59 #6554] [task:5616] ERROR -- DirectorJobRunner: 419: unexpected token at '"consul.a'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/json_pure-1.8.1/lib/json/common.rb:155:in `parse'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/json_pure-1.8.1/lib/json/common.rb:155:in `parse'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/lib/bosh/director/models/template.rb:61:in `object_or_nil'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/lib/bosh/director/models/template.rb:52:in `provides'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/lib/bosh/director/deployment_plan/instance_group_spec_parser.rb:185:in `block in parse_jobs'


The Director VM is down because a migration script failed to run. The migration script's job was to increase the column size of two columns in the database. When the migration script does not run the columns will have a smaller size and the JSON data stored in the columns may be truncated causing JSON parsing errors in the Director.


To resolve this upgrade issue in ERT 1.9.x:

1. Make sure the DB migration schema file is in place


2. The contents of this file should be the following (For MySQL only)

Sequel.migration do 
change do
if [:mysql2, :mysql].include?(adapter_scheme)
alter_table(:templates) do
set_column_type :provides_json, 'longtext'
set_column_type :consumes_json, 'longtext'

3. Restart the director service

monit restart director

4. Upload the release file one more time

bosh upload release /var/tempest/releases/consul-165.xx.tgz --fix

5. Delete the DB schema migration file from the database

/var/vcap/jobs/director/bin/director_ctl console

6. Remove the DB schema migration file from the hard disk

rm /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/db/migrations/director/20170303175054_expand_template_json_column_lengths.rb

7. Apply the change

The issue should be resolved now.


Powered by Zendesk