Pivotal Knowledge Base

Follow

Push Notification Service is Unable to Reconnect Queues after RabbitMQ Restarts

Environment 

Push Notification Service for Pivotal Cloud Foundry (PCF) 1.9.3 or below

Symptom

After RabbitMQ outages like restart/upgrade or some other issues, push-analytics and push-api cannot connect back to RabbitMQ instance used for queuing push notifications.

push-analytics app panics with an error:

org.springframework.amqp.rabbit.listener.QueuesNotAvailableException:
Cannot prepare queue for listener. Either the queue doesn't exist or the broker will not allow us to use it.

push-api app panics with an error: 

[OUT] [WARN] 1196346734 23 Jan 2018 16:12:19.196 +0000 [push-worker-pool-ios-3]
io.pivotal.push.jobs.workers.RabbitMQAbstractWorker - [SYSTEM] Rabbit seems to be down,
waiting for reconnect
[push-worker-pool-android-baidu-0] io.pivotal.push.jobs.workers.RabbitMQAbstractWorker
- [SYSTEM] Rabbit seems to be down, waiting for reconnect
[dispatch-worker-pool-6] io.pivotal.push.jobs.util.FixedWorkerRestartableThreadPoolExecutor
- [SYSTEM] Restarting worker thread: RabbitMQDispatchWorker, 1792240021
[dispatch-worker-pool-6] io.pivotal.push.jobs.workers.RabbitMQAbstractWorker
- [SYSTEM] Rabbit seems to be down, waiting for reconnect
[APP/PROC/WEB/0] [OUT] com.rabbitmq.client.AlreadyClosedException:
connection is already closed due to connection error;
cause: com.rabbitmq.client.MissedHeartbeatException: Heartbeat missing with heartbeat = 60 seconds

Impact

This affects the functionality of push messages and can cause loss of small to a large number of push notifications.

Cause

This is a known issue with Push Notification service with the push notifications service tile.

Resolution

Workaround

Restart the push-api and push-analytics app to reconnect to the RabbitMQ instance.

  1. Login to cf cli
  2. cf target -o system -s push-notifications
  3. cf restart push-api
  4. cf restart push-analytics

Permanent Fix

Upgrade the Push Notification Service tile to version 1.9.4.

Comments

Powered by Zendesk