Pivotal Knowledge Base

Follow

How to Enable Debug Mode for Consul

Environment

Pivotal Cloud Foundry® (PCF) 1.6.x and 1.7.x

Purpose

This article discusses how to enable debug mode for Consul, including Consul server and Consul agent.

Procedure

  • Connect via 'bosh ssh' to the VM (either Consul VM acting as Consul server, or other VMs as Consul agent, like Cloud Controller, Diego Brain, etc).
  • 'sudo su -' to root account. 
  • Edit the configuration file /var/vcap/jobs/consul_agent/config/config.json. 

The default value is "info" for "log_level". You can change it to "trace", "debug", "warn", or "err". 

This is a sample file from a testing environment. "log_level" is changed to debug now. 

{"server":true,"domain":"cf.internal","datacenter":"dc1","data_dir":"/var/vcap/store/consul_agent","log_level":"debug","node_name":"consul-server-partition-1506967bb2eea78b2b46-0","ports":{"dns":53},"rejoin_after_leave":true,"retry_join":["192.168.27.72"],"bind_addr":"192.168.27.72","disable_remote_exec":true,"disable_update_check":true,"protocol":2,"verify_outgoing":true,"verify_incoming":true,"verify_server_hostname":true,"ca_file":"/var/vcap/jobs/consul_agent/config/certs/ca.crt","key_file":"/var/vcap/jobs/consul_agent/config/certs/server.key","cert_file":"/var/vcap/jobs/consul_agent/config/certs/server.crt","encrypt":"Lfs9s5z/2qAqmjQAoWs+DA==","bootstrap_expect":1}
  • Run 'consul reload' command to reload the configuration file. The consul command is always in /var/vcap/jobs/consul_agent/packages/consul/bin/ directory. 

This is the output from a testing environment. 

root@ce241448-d9cd-4106-8a81-b87c86e0c577:/var/vcap/sys/log/consul_agent# /var/vcap/jobs/consul_agent/packages/consul/bin/consul reload
Configuration reload triggered
  • You should see log debug entries that look like this: 
root@ce241448-d9cd-4106-8a81-b87c86e0c577:/var/vcap/sys/log/consul_agent# tail -f consul_agent.stdout.log 
2016/08/02 02:37:30 [INFO] memberlist: Suspect diego-cell-partition-1506967bb2eea78b2b46-0 has failed, no acks received
2016/08/02 02:37:36 [INFO] memberlist: Suspect diego-cell-partition-1506967bb2eea78b2b46-0 has failed, no acks received
2016/08/02 05:28:52 [INFO] agent.rpc: Accepted client: 127.0.0.1:53139
==> Caught signal: hangup
==> Reloading configuration...
==> WARNING: LAN keyring exists but -encrypt given, using keyring
==> WARNING: WAN keyring exists but -encrypt given, using keyring
==> WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
2016/08/02 05:28:52 [DEBUG] agent: Service 'consul' in sync
2016/08/02 05:29:15 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.75:8301
2016/08/02 05:29:45 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.80:8301
2016/08/02 05:30:11 [DEBUG] agent: Service 'consul' in sync
2016/08/02 05:30:15 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.82:8301
2016/08/02 05:30:25 [DEBUG] memberlist: TCP connection from: 192.168.27.80:41909
2016/08/02 05:30:45 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.82:8301
2016/08/02 05:31:10 [DEBUG] memberlist: TCP connection from: 192.168.27.83:52494
2016/08/02 05:31:15 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.83:8301
2016/08/02 05:31:33 [DEBUG] memberlist: TCP connection from: 192.168.27.91:34780
2016/08/02 05:31:41 [DEBUG] memberlist: TCP connection from: 192.168.27.84:55011
2016/08/02 05:31:45 [DEBUG] memberlist: Initiating push/pull sync with: 192.168.27.91:8301
2016/08/02 05:31:58 [DEBUG] agent: Service 'consul' in sync

Impact

Enabling debug mode will impact the system performance and may also fill up the ephemeral disk, so we strongly recommend to use it only for development/troubleshooting purpose.

Additional Information

Related article:

Consul fails to start during upgrade in Cloud Foundry

Consul failing "x/y nodes reported success"

 

Comments

Powered by Zendesk