Pivotal Knowledge Base


Consul Failing "x/y Nodes Reported Success"


Pivotal Cloud Foundry®  versions prior to 1.9.42, 1.10.31, 1.12.5, and 1.11.17


In some cases (sometimes during a platform upgrade), Consul will not reach a quorum and it will fail with the following message:

{"timestamp":"1507476742.803297520","source":"confab","message":"confab.agent-client.set-keys.install-key.request.failed","log_level":2,"data":{"error":"Unexpected response code: 500 (1 error(s) occurred:\n\n* 1/89 nodes reported failure)","key":"jm3apuzmw868klczs2xv
2017/10/08 15:32:23 [INFO] serf: Received list-keys query
2017/10/08 15:32:23 [INFO] serf: Received list-keys query
2017/10/08 15:32:24 [INFO] serf: Received install-key query
2017/10/08 15:32:24 [INFO] serf: Received install-key query
2017/10/08 15:32:25 [ERR] http: Request POST /v1/operator/keyring, error: 1 error(s) occurred:017/05/04 12:31:39 [INFO] serf: EventMemberFailed: dedicated-node-8 {"timestamp":"1493901099.471777678","source":"confab","message":"confab.agent-client.set-keys.list-keys.request.failed","log_level":2,"data":{"error":"155/179 nodes reported success"}}

The important part to note is at the end, where 1 and 89 can vary depending on the size of your environment.

{"error":"1/89 nodes reported success"}

This will also result in Consul/0 instance being reported as down or failing.  This is the canary instance and its failure will halt the deployment so that the other Consul nodes remain up and working.


This error is indicating that Consul cannot communicate with some subset of the Consul agents.  This has typically happened when Consul is propagating new keys or rotating the existing keys that it uses to communicate securely and there is a problem during this process.  One common problem is that port 8301 is blocked and prevents Consul from distributing the keys.


Upgrade to versions 1.9.42, 1.10.31, 1.11.17, or 1.12.5, which avoids this issue.

This is not typically a problem with the Consul server, but with the specific Consul Agents. To resolve this issue, you need to resolve the issue with the Consul Agent. Exactly how you can resolve this depends on the problem affecting the agent.

Pivotal recommends that you check the problem Consul Agent VMs for the following:

  • Make sure that there is adequate disk space (runbosh vms --vitals to see disk usage on the virtual machines)on the node. If ephemeral or persistent disks are more than 90% full, increase their size or delete something to free up space.
  • Confirm that Consul Server is able to communicate on port 8301 with the VMs in question (both TCP & UDP).

In some cases, these steps might not be sufficient to resolve the issue.  In that case, please contact Pivotal Support for additional assistance troubleshooting and resolving the problem.


This situation is not the same as when Consul goes split brain. In fact, following the instructions to repair a split-brain Consul will make this situation worse and can cause application downtime.  If you are seeing the error message indicated in the Symptoms section above, make sure you resolve this before restarting or recreating any of the Consul server nodes.

Additional Information

Related article:

Consul fails to start during upgrade in Cloud Foundry

How to enable debug mode for Consul

Consul-release: X/Y nodes reported success


  • Avatar
    Jim Worrell

    * As an addendum:

    It is a good idea to check processes across all vms by using `bosh vms --details` to determine if there is evidence of high disk usage.

    Also it is recommended to check all deployments and not just the CF deployment.

  • Avatar
    Ryan Hall

    Log entry that you'll see on the Consul node that is failing the key rotation:

    /var/vcap/sys/log/consul_agent/consul_agent.stdout.log: 2018/03/28 17:09:08 [ERR] http: Request PUT /v1/agent/check/pass/service:cell?note=, error: failed persisting state for check "service:cell": failed writing temp file "/var/vcap/data/consul_agent/checks/state/e894e9560119905ddb39e01a0d321cd0.tmp": write /var/vcap/data/consul_agent/checks/state/e894e9560119905ddb39e01a0d321cd0.tmp: no space left on device from= 

    This error is indicating that the failing node has a full disk, thus it can't rotate the key, even though it's the same. You'll need to clear some space off the disk of the node with this error so that you can move past the following error you'll see in consul_agent.stderr.log whenever you perform an Ops Manager Apply Changes or Bosh operation:

    error during start: timeout exceeded: "Unexpected response code: 500 (1 error(s) occurred:\n\n* 1/n nodes reported failure

    After you clear that error, you can upgrade your environment to the patched ERT/PAS version without Consul failing the operation.

    Edited by Ryan Hall
Powered by Zendesk