Pivotal Cloud Foundry: 1.10
You see the following issues in Ops Manager:
- Apply changes will fail with
dial tcp <director-ip>:22: getsockopt: connection timed out
- The Status page for the Ops Manager Director Tile will not show any data
bosh loginwill fail saying it cannot access the director
Also, in the Bosh Director logs, you will see an error like this:
E, [2017-05-26 18:27:59 #6554] [task:5616] ERROR -- DirectorJobRunner: 419: unexpected token at '"consul.a'
/var/vcap/packages/director/gem_home/ruby/2.3.0/gems/bosh-director-260.6.0/lib/bosh/director/deployment_plan/instance_group_spec_parser.rb:185:in `block in parse_jobs'
The Director VM is down because a migration script failed to run. The migration script's job was to increase the column size of two columns in the database. When the migration script does not run the columns will have a smaller size and the JSON data stored in the columns may be truncated causing JSON parsing errors in the Director.
Below is the workaround to resolve this upgrade issue in ERT 1.9.x
1. Make sure the DB migration schema file is in place
2. The contents of this file should be the following (For MySQL only)
if [:mysql2, :mysql].include?(adapter_scheme)
set_column_type :provides_json, 'longtext'
set_column_type :consumes_json, 'longtext'
3. Restart the director service
monit restart director
4. Upload the release file one more time
bosh upload release /var/tempest/releases/consul-165.xx.tgz --fix
5. Delete the DB schema migration file from the database
6. Remove the DB schema migration file from the hard disk
7. Apply the change
The issue should be resolved now.