Pivotal Knowledge Base

Follow

Pivotal Cloud Foundry Push Notification Service: Recreate Service Instance Fails with "Duplicate Tenant"

Environment 

Push Notification Service for Pivotal Cloud Foundry (PCF) all versions

Symptom

PCF Push Notification Service: Recreate service instance fails with "Duplicate Tenant"

When a pcf-push-service is removed, an attempt to re-create a new pcf-push-service fails with Error:

Server error, status code: 502, error code: 10001, message: Service broker error: {"errors"=>[{"code"=>1018, 
"label"=>"DUPLICATE_OBJECT", "description"=>"Duplicate Tenant", "httpStatus"=>"CONFLICT"}]}

Cause

The old service-instance was not deleted correctly from the Push Notification Service broker database.

The command used to remove the pcf-push-service used was cf purge-service-instance.

This command ignores the pcf-push-service broker database, but removes the service instance from cloud controller database. This leaves an orphan service-instance entry in the pcf-push-service broker database. This service-instancee is actually pointing to one the spaces it was created in, however the service-instance actually doesn't exist as it was already purged from cloud controller database.

Please confirm this by following these steps:

  • Target the Org/Space where the pcf-push-service is created.
    • Run cf services to confirm there is no pivotal-push-service instance in the given space.
    • Run cf space <space-name> --guid and make a note of this GUID.
  • Login to the MySQL database:
    • cf target -o system -s push-notifications
    • cf env push
    • Get the mysql server/proxy IP, username, password and cf-database (name) from jdbcURL in VCAP_SERVICES.
    • mysql -h <mysql_IP> -u <username> -p<password> -D <database_name>
  • Run this command:
mysql> select provisions.tenant_id, provisions.space_guid, 
provisions.service_guid,tenants.uuid from provisions inner join tenants on provisions.tenant_id=tenants.id;

Check if the space_guid from 1 is present in the database.

Add a note of the service_guid in the same row.

cf curl /v2/service_instances/<service_guid>
cf curl /v2/spaces/<space_guid>

If the cf curl commands are unable to get the listed service_guid, this confirms that we have an orphan entry of a service-instance in the broker database.

Resolution

If there are no other production apps or services in the corresponding space, we can delete the space. Recreate a new space and create a new service instance.

Please contact your PCF Administrator to ensure this issue does not affect any other apps or permissions.

If there are other production apps or services active in the same space and space deletion affects production, please contact Pivotal Support.

Comments

Powered by Zendesk