Pivotal Knowledge Base

Follow

Elastic Runtime 1.7.x Upgrade results in Error 100: undefined method `[]' for nil:NilClass

Environment

Product Version
Pivotal Cloud Foundry® (PCF) Ops Manager Pre 1.7.8+
Pivotal Cloud Foundry® (PCF) Elastic Runtime 1.7.x

Symptom

In an environment running Ops Manager versions prior to 1.7.8+ when attempting to upgrade Elastic Runtime v1.7.x, you see an error such as the following:

D, [2016-07-12 21:42:19 #9068] [task:3237] DEBUG -- DirectorJobRunner: SENT: hm.director.alert {"id":"169c0739-5ca5-442d-8f98-a0c697f0435b","severity":3,"source":"director","title":"director - error during update deployment","summary":"Error during update deployment for 'cf-7d12b2492efb81d8fba9' against Director '11bc0a8f-d4e2-468a-8e0f-58194f6964e9': #<NoMethodError: undefined method `[]' for nil:NilClass>","created_at":1468359739}
 

When attempting to run `bosh vms` you see that the UAA VM has no VM Type:

+-----------------------------------------------------------------------------+---------+-----+--------------+--------------+
| VM | State | AZ | VM Type | IPs |
+-----------------------------------------------------------------------------+---------+-----+--------------+--------------+
| uaa-partition-9ed9effa594aa3c4753c/1 (fc2774d0-0ae5-401b-a957-b1758a887ad5) | running | n/a | | xx.xx.xx.xx |
+-----------------------------------------------------------------------------+---------+-----+--------------+--------------+

 Cause

The Director's mysql db was truncating data when it didn't fit, types on mysql needed to be large text. This was determined to be the issue because the Ops Manager Director was configured to use mysql and paired with running `bosh vms` showed the uaa-partition with an empty VM type.

Resolution

To solve the Error 100: undefined method `[]' for nil:NilClass error, the following steps were taken:

  1. Upgraded Ops Manager to 1.7.8+
  2. Ran `bosh vms` to verify the UAA VM has no VM Type (see the example above in the Symptom section)
  3. Ran `bosh resurrection disable uaa-partition-xxx <index>` to prevent BOSH from interrupting this process
  4. Located the UAA VM in the IaaS and deleted the VM. You can get the VM CID by running `bosh vms --details` and looking under the CID column, which will help you find the VM in your IaaS.
  5. Ran `bosh cck` and when prompted, selected option 3 to delete the disk reference
  6. Ran `bosh deploy` which should then recreate and update the UAA VM from the stemcell
  7. Ran `bosh vms` to verify the UAA VM has a VM Type again
  8. Ran `bosh resurrection enable uaa-partition-xxx <index>` to re-enable the BOSH resurrector for UAA again

 

Comments

Powered by Zendesk