Pivotal Knowledge Base

Follow

Spring Cloud Services error "Failure provisioning service instance: Timed out after 8 minutes waiting for service instance to provision"

Environment 

Spring Cloud Services for Cloud Foundry: 1.8.x

Pivotal Cloud Foundry: 1.9.x, 1.10.x

Problem

Unable to create Spring Cloud Services config-server service instance.

Symptom

You are unable to provision a Spring Cloud Services service instance. Further, the creation of registry service instance fails with error:

Failure provisioning service instance test-service-registry-1495017703: 
Timed out after 8 minutes waiting for service instance to provision.

For example,

cf service test-service-registry

Service instance: test-service-registry
Service: p-service-registry
Bound apps:
Tags:
Plan: standard
Description: Service Registry for Spring Cloud Applications
Documentation url: http://docs.pivotal.io/spring-cloud-services/
Dashboard: https://spring-cloud-broker.cfapps-03.haas-59.pez.pivotal.io/dashboard/p-service-registry/ef94a651-6e2d-44ff-95ac-3577b01fe938

Last Operation
Status: Failure provisioning service instance test-service-registry-1495017703: Timed out after 8 minutes waiting for service instance to provision
Message:
Started: 2017-06-14T14:13:45Z
Updated: 2017-06-14T14:13:50Z

Analysis

(A) Check for the Spring Cloud Services broker and broker-worker logs.

  • Login to CF CLI and target the Org/Space system/p-spring-cloud-services
    • cf login
    • cf target -o system -s p-spring-cloud-services
  • Get the recent logs of the Spring Cloud broker-worker
    • cf logs spring-cloud-broker-worker --recent
    • The ideal output should have the logs ending with this line:
Successfully created service instance: ServiceRequest(serviceName=p-service-registry, instanceId=ef94a651-6e2d-44ff-95ac-3577b01fe938
  • However, in this scenario, as the service instance failed to provision after 8 minutes, we will not get any logs from the broker-worker

(B) Next, check the Cloud Controller logs for recent events when this service instance creation was attempted.

  • Log into the Ops Manager dashboard
  • Collected the Cloud Controller logs from the Status tab
  • The logs can also be obtained using the article: How to Obtain Logs from Pivotal Cloud Foundry
  • After reviewing, Request failed: 500 errors were found in the cloud_controller_ng logs
    • Extract all the logs to a local directory for all Cloud Controller nodes
    • Under the cloud_controller.?.job directories, search for Request failed: 500 in all the cloud_controller_ng/cloud_controller_ng.log in all nodes of cloud controller (? is node index number):
    • grep "Request failed: 500" cloud_controller_ng.log
{"timestamp":1488486081.527088,"message":"Request failed: 500: {"code"=>10001, "description"=>"undefined method `name' for nil:NilClass", 
"error_code"=>"CF-NoMethodError", "backtrace"=>["/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/app.rb:408

Cause

This is a race condition with cloud controller caused due to querying of the app when the app is being deleted.

The smoke-tests of Spring Cloud Services creates and deletes config-server instance apps while queried under Org/Space p-spring-cloud-services/instances. These events on the Spring Cloud Services smoke-tests lead to the race condition which causes the failures listed above.

Resolution

As a workaround, restart the Spring Cloud Services Broker and Broker-worker apps

cf stop spring-cloud-broker
cf start spring-cloud-broker
cf stop spring-cloud-broker-worker
cf start spring-cloud-broker-worker

A permanent fix is being tracked under as a story in the CF CAPI Public Tracker.

 

Comments

Powered by Zendesk