Pivotal Knowledge Base

Follow

Ops Manager Director is down following an upgrade where migration scripts fail

Environment

Pivotal Cloud Foundry: 1.10

Symptom

You see the following issues in Ops Manager:

  • Apply changes will fail with dial tcp <director-ip>:22: getsockopt: connection timed out
  • The Status page for the Ops Manager Director Tile will not show any data
  • bosh login will fail saying it cannot access the director

Also, in the Bosh Director logs, you will see an error like this:

E, [2017-05-26 18:27:59 #6554] [task:5616] ERROR -- DirectorJobRunner: 419: unexpected token at '"consul.a'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/json_pure-1.8.1/lib/json/common.rb:155:in `parse'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/json_pure-1.8.1/lib/json/common.rb:155:in `parse'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/lib/bosh/director/models/template.rb:61:in `object_or_nil'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/lib/bosh/director/models/template.rb:52:in `provides'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/lib/bosh/director/deployment_plan/instance_group_spec_parser.rb:185:in `block in parse_jobs'

Cause

The Director VM is down because a migration script failed to run. The migration script's job was to increase the column size of two columns in the database. When the migration script does not run the columns will have a smaller size and the JSON data stored in the columns may be truncated causing JSON parsing errors in the Director.

Resolution 

Below is the workaround to resolve this upgrade issue in ERT 1.9.x

1. Make sure the DB migration schema file is in place

/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/db/migrations/director/20170303175054_expand_template_json_column_lengths.rb

2. The contents of this file should be the following (For MySQL only)

Sequel.migration do 
change do
if [:mysql2, :mysql].include?(adapter_scheme)
alter_table(:templates) do
set_column_type :provides_json, 'longtext'
set_column_type :consumes_json, 'longtext'
end
end
end
end

3. Restart the director service 

monit restart director

4. Upload the release file one more time

bosh upload release /var/tempest/releases/consul-165.xx.tgz --fix

5. Delete the DB schema migration file from the database

/var/vcap/jobs/director/bin/director_ctl console
Bosh::Director::Models::Instance.db[:schema_migrations].where(filename:"20170303175054_expand_template_json_column_lengths.rb").delete

6. Remove the DB schema migration file from the hard disk

rm /var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/db/migrations/director/20170303175054_expand_template_json_column_lengths.rb

7. Apply the change

The issue should be resolved now. 

Comments

Powered by Zendesk