Pivotal Knowledge Base

Follow

How to monitor NATS traffic on the Gorouter

Environment

Pivotal Cloud Foundry Diego Cell

NATS traffic on a Gorouter VM

Purpose

The purpose of this KB is to confirm a Gorouter is receiving announcements from a Diego Cell's rep process. To do this you can packet capture the NATS traffic on a Gorouter VM to confirm the announcements are being received. One might do this to debug routes not found by a Gorouter for incoming app URLs, or to confirm a Diego Cell's rep process is sending the <IP Address:Port> of one of the Cell's hosted apps.

Procedure

1. Log in to the Operations Manager console and SSH to a Router VM.

bosh deployments
bosh -d cf-dd5556db39fdde2c5c84 vms
bosh -d cf-dd5556db39fdde2c5c84 ssh router/04751932-18ba-4242-ba1a-f6327fd24712

2. Listen for the NATS announcements for an app with a hostname spring-music. Give it a minute or two to show up.  

sudo -i tcpdump -A -nn -i eth0 'port 4222' | grep spring-music

3. Network packets can be fragmented, so if you do not get readable data it would be better to save the tcpdump to a file

tcpdump -i eth0 'port 4222' -w /var/tmp/nats.tcpdump

4. Cancel it after a minute or two and read the file back with

tcpdump -A -nn  -r /var/tmp/nats.tcpdump 'port 4222' | grep spring-music

5. Otherwise, download the file to view in a network capture utility like Wireshark.

chmod +r /var/tmp/nats.tcpdump

6. Then back on the Operations Manager, copy the file from the router.

bosh -d cf-dd5556db39fdde2c5c84 scp router/04751932-18ba-4242-ba1a-f6327fd24712:/var/tmp/nats.tcpdump nats.tcpdump

7. This is an example NATS message advertising the spring-music app...

{"host":"10.193.84.34","port":61028,"tls_port":61030,"uris":["spring-music-unashamed-range.cfapps-19.haas-59.pez.pivotal.io"],"app":"fdb25a52-a203-415a-a3fd-3bd0b9180c7f","private_instance_id":"03c12d80-30f5-48a1-5a17-477e","private_instance_index":"0","server_cert_domain_san":"03c12d80-30f5-p48a1-5a17-477e","tags":{"component":"route-emitter"}}

8. From the above step, you see the app is in the Diego cell with the address 10.193.84.34:61028. This confirms that the Diego cell with IP 10.193.84.34 is sending app routes.

If you have multiple NATS servers, it may be helpful to view the messages for each server one at a time. You can get the IP addresses from the Operations Manager /PAS/Status page, or from the routers config file /var/vcap/jobs/gorouter/config/gorouter.yml.

tcpdump -A -nn  -r /var/tmp/nats.tcpdump 'port 4222 and src 10.193.84.19' | grep spring-music
tcpdump -A -nn  -r /var/tmp/nats.tcpdump 'port 4222 and src 10.193.84.55' | grep spring-music

  

Comments

Powered by Zendesk