Pivotal Knowledge Base

Follow

Autoscaler failing on SQL query: "panic: gorp: multiple rows returned"

Environment

Pivotal Cloud Foundry 1.11

Symptom

Autoscaler is failing on SQL query, and you see the following go panic error:

2017-06-12T22:24:05.38+0000 [APP/PROC/WEB/0]ERR 
panic: gorp: multiple rows returned for: SELECT * FROM autoscale_instances WHERE is_leader=true - []

2017-06-12T22:24:05.38+0000
[APP/PROC/WEB/0]ERR goroutine 55 [running]:
2017-06-12T22:24:05.38+0000
[APP/PROC/WEB/0]ERR cf-autoscaling/db/repositories.(*AutoscaleInstanceRepository)
.FindLeader(0xc420115020, 0x0, 0x0, 0x0, 0x0, 0x0, 0x5b1417b02bc1e944, 0xcc86830f30ecaa08)

Cause

This error occurs because Autoscaler gets confused and can’t determine it’s leader. This is due to a race condition which has been fixed in the latest version.

Resolution

An operator can clear this by going into the ERT MySQL instance to clear out the leader and then reestablish the quorum by scaling down and restarting the Autoscaler app one instance at a time. Below are the steps for fixing the leader status in the ERT MySQL Database:

1. Update autoscale_instances SET is_leader=0 WHERE is_leader=1;

Screen_Shot_2017-06-13_at_12.17.49_PM.png

Screen_Shot_2017-06-13_at_12.18.00_PM.png

2. Restart App

3. Once started, scale up to 2 instances. Start app and wait until finished  

Screen_Shot_2017-06-13_at_12.18.08_PM.png

4. Then scale up to 3 instances and start app

Screen_Shot_2017-06-13_at_12.18.16_PM.png

5. You can see the events summary indicating that the query has passed

Screen_Shot_2017-06-13_at_12.18.25_PM.png

Screen_Shot_2017-06-13_at_12.18.36_PM.png

 

Comments

Powered by Zendesk