Pivotal Knowledge Base

Follow

How to use Application Security Group (ASG) logging

Environment

Pivotal Cloud Foundry 1.8, 1.9, 1.10, 1.11

Purpose

As an operator, you might want to correlate the emitted logs back to an application.

Background

Example 1

In this example, we are adding a sample ASG  with logging enabled to allow traffic from a container to the Gorouter IP address. The ASG is bound to org “o” and space “s” which has one app “proxy” running.

The "proxy" app is merely used for demonstration here. It will forward HTTP requests to the URL/IP provided in the curl URI. For, e.g.,

curl proxy.<cf-domain>/proxy/foo.com will forward requests to foo.com

Follow the instructions here to setup application security groups. The ASG configured in this example is:

[ 
  { 
    "protocol": "tcp", 
    "destination": "10.87.35.105", 
    "ports":"80", 
    "log": true, 
    "description": "Allow apps to reach gorouter" 
  } 
] 

In the above example, all requests to IP 10.87.35.105 (Gorouter IP) will be logged in Syslog. To generate the log entry, we will use the proxy app to forward the request to the Gorouter. For e.g., 

curl proxy.cfapps-14.haas-59.pez.pivotal.io/proxy/10.193.79.30:80

This will generate a Syslog entry as follows:

Jun 16 23:18:38 10.193.79.37 kern kernel [job=diego_cell index=1] [2681198.826158] 1f97ff6d-4517-43a0-5aef-
f5915IN=wbrdg-0afe0008 OUT=eth0 MAC=1e:4d:39:af:e6:04:72:87:1b:3b:15:01:08:00 SRC=10.254.0.10 DST=10.193.79.30 LEN=60
TOS=0x00 PREC=0x00 TTL=63 ID=40628 DF PROTO=TCP SPT=52232 DPT=80 WINDOW=28280 RES=0x00 SYN URGP=0

Note the SRC IP and DST IP - SRC IP is the IP of the container trying to reach DST Ggorouter IP 

Example 2

In the 2nd example, we are adding an ASG with logging enabled to log all traffic egressing from a container inside Cloud Foundry

[
{
"description": "Allow apps to reach router",
"destination": "0.0.0.0-0.0.0.255",
"log": true,
"ports": "80",
"protocol": "tcp"
}
]  

NOTE: For this procedure to work, it is necessary to have aggregated logging for Syslog messages to a Syslog server. See the instructions to set up Syslog forwarding here

Instructions

Here we will illustrate an example of how a Syslog entry from an ASG can be mapped back to the app container that is generating the log

Step 1. Run the curl command to the "proxy" app from the example above to generate a log entry

Step 2. From logging aggregator server, to get log entries to the Gorouter IP, grep or search for the string "DST=<gorouter_ip>" to get a sample log from ASG logging. In your case, it could be any of the search string of interest to you based on the sample ASG log entry

$ grep "DST=10.193.79.30" *

10.193.79.42.log:Jun 20 23:05:03 10.193.79.42 kern kernel [job=diego_cell index=2] [1122561.945562] d08bb414-cdba-4381-412a-e9eccIN=wbrdg-0afe0008 OUT=eth0 MAC=4e:b7:a4:74:e2:45:0a:5a:66:81:1f:1d:08:00 SRC=10.254.0.10 DST=10.193.79.30 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=60879 DF PROTO=TCP SPT=42630 DPT=80 WINDOW=28280 RES=0x00 SYN URGP=0

Step 3. Note the container-guid highlighted in red above

Step 4. From your aggregated logging server, run the following search command to get the app guid

$  grep d08bb414-cdba-4381-412a-e9ecc 10.193.79.42.log| grep "vcap.rep" | grep process_guid

Jun 22 13:07:52 10.193.79.42 user vcap.rep [job=diego_cell index=2] {"timestamp":"1498136872.889604807",
"source":"rep","message":"rep.executing-container-operation.ordinary-lrp-processor.process-reserved-
container.run-container.containerstore-run.node-run.action.run-step.process-exit","log_level":1,
"data":{"cancelled":true,"container-guid":"d08bb414-cdba-4381-412a-e9eccdcedb40","container-state":
"reserved","exitStatus":143,"guid":"d08bb414-cdba-4381-412a-e9eccdcedb40","lrp-instance-key"
:{"instance_guid":"d08bb414-cdba-4381-412a-e9eccdcedb40","cell_id":"diego_cell-2"},"lrp-key"
:{"process_guid":"4ccbb24c-268e-421a-9d1b-3678b6eb40d0-4a2fe5af-5d55-4c0e-b720-5e2a8ee686a3","
index":0,"domain":"cf-apps"},"process":"56c9d

Note the process_guid from the above log entry. Grab the the first 32 octets from the 64 octet entry.  

Step 5. Find the App name by calling CC API if the App is still running

$ cf curl /v2/apps/4ccbb24c-268e-421a-9d1b-3678b6eb40d0
{
"metadata":
"guid": "4ccbb24c-268e-421a-9d1b-3678b6eb40d0",
"url": "/v2/apps/4ccbb24c-268e-421a-9d1b-3678b6eb40d0", "created_at": "2017-06-16T23:13:55Z", "updated_at": "2017-06-20T22:56:40Z" }, "entity": { "name": "proxy", "production": false, "space_guid": "c65add41-d85b-4823-9e2a-545a5e0e5812", "stack_guid": "9dad41b8-3043-48da-acf4-e1d4a700ea88", " "ports": [ 8080
--- snip ---
], "space_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812", "stack_url": "/v2/stacks/9dad41b8-3043-48da-acf4-e1d4a700ea88", "routes_url": "/v2/apps/4ccbb24c-268e-421a-9d1b-3678b6eb40d0/routes", "events_url": "/v2/apps/4ccbb24c-268e-421a-9d1b-3678b6eb40d0/events", "service_bindings_url": "/v2/apps/4ccbb24c-268e-421a-9d1b-3678b6eb40d0/service_bindings", "route_mappings_url": "/v2/apps/4ccbb24c-268e-421a-9d1b-3678b6eb40d0/route_mappings" }
 

Step 6. Find the space name by calling Cloud Controller API

$ cf curl /v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812 
{
   "metadata": {
      "guid": "c65add41-d85b-4823-9e2a-545a5e0e5812",
      "url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812",
      "created_at": "2017-06-16T18:14:40Z",
      "updated_at": null
   },
   "entity": {
      "name": "s",
      "organization_guid": "2c279d6c-6f15-4cc0-99ed-72ef76ec50eb",
      "space_quota_definition_guid": null,
      "allow_ssh": true,
      "organization_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb",
      "developers_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/developers",
      "managers_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/managers",
      "auditors_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/auditors",
      "apps_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/apps",
      "routes_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/routes",
      "domains_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/domains",
      "service_instances_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/service_instances",
      "app_events_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/app_events",
      "events_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/events",
      "security_groups_url": "/v2/spaces/c65add41-d85b-4823-9e2a-545a5e0e5812/security_groups"  

Step 7. Find the org name by calling Cloud Controller (CC) API

$ cf curl /v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb 
{
   "metadata": {
      "guid": "2c279d6c-6f15-4cc0-99ed-72ef76ec50eb",
      "url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb",
      "created_at": "2017-06-16T18:14:19Z",
      "updated_at": null
   },
   "entity": {
      "name": "o",
      "billing_enabled": false,
      "quota_definition_guid": "6002a805-f285-4e7b-8ff6-cd4ac607d26e",
      "status": "active",
      "quota_definition_url": "/v2/quota_definitions/6002a805-f285-4e7b-8ff6-cd4ac607d26e",
      "spaces_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/spaces",
      "domains_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/domains",
      "private_domains_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/private_domains",
      "users_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/users",
      "managers_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/managers",
      "billing_managers_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/billing_managers",
      "auditors_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/auditors",
      "app_events_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/app_events",
      "space_quota_definitions_url": "/v2/organizations/2c279d6c-6f15-4cc0-99ed-72ef76ec50eb/space_quota_definitions"
   }
}

If the App has been deleted, "curl" will return an "App not found" message

$ cf curl /v2/apps/4ccbb24c-268e-421a-9d1b-3678b6eb40d0
{
"description": "The app could not be found: 4ccbb24c-268e-421a-9d1b-3678b6eb40d0",
"error_code": "CF-AppNotFound",
"code": 100004
}

Call the events endpoint to find the app information and follow the steps above to get the space and org information

$ cf curl /v2/events?q=actee:4ccbb24c-268e-421a-9d1b-3678b6eb40d0
{
   "total_results": 6,
   "total_pages": 1,
   "prev_url": null,
   "next_url": null,
   "resources": [
      {
         "metadata": {
            "guid": "d343a7dc-8882-4529-b3b9-83677ee2017e",
            "url": "/v2/events/d343a7dc-8882-4529-b3b9-83677ee2017e",
            "created_at": "2017-06-16T23:13:55Z",
            "updated_at": null
         },
         "entity": {
            "type": "audit.app.create",
            "actor": "7b2d6421-db00-4a1d-86d3-dc07edbe8afd",
            "actor_type": "user",
            "actor_name": "admin",
            "actee": "4ccbb24c-268e-421a-9d1b-3678b6eb40d0",
            "actee_type": "app",
            "actee_name": "proxy",
            "timestamp": "2017-06-16T23:13:55Z",
            "metadata": {
               "request": {
                  "name": "proxy",
                  "space_guid": "c65add41-d85b-4823-9e2a-545a5e0e5812",
                  "memory": 32,
                  "disk_quota": 32,
                  "buildpack": "go_buildpack",
                  "environment_json": "PRIVATE DATA HIDDEN",
                  "console": false,
                  "docker_credentials_json": "PRIVATE DATA HIDDEN",
                  "health_check_type": "port",
                  "instances": 1,
                  "production": false,
                  "state": "STOPPED"
               }
            },
            "space_guid": "c65add41-d85b-4823-9e2a-545a5e0e5812",
            "organization_guid": "2c279d6c-6f15-4cc0-99ed-72ef76ec50eb"
         }
      --snip--

Comments

Powered by Zendesk