Pivotal Knowledge Base

Follow

How to get into an App Container Manually with Garden-RunC Backend

Environment

Elastic Runtime versions 

  • 1.6.45 and above
  • 1.7.28 and above
  • 1.8.8 and above
  • 1.8.40 and above
  • Below 1.9.18
  • Below 1.10.5

Purpose

Due to security reasons, some customers do not allow SSH access to app containers via the `cf ssh` command.  At the same time, it can be useful sometimes to enter into app containers for troubleshooting purposes. This procedure outlines the steps to enter into an app container manually where Garden-runC backend is running.

Note that Garden-runC backend is available only on ERT versions >1.65, >1.7.28, >1.8.9, >1.9.0. Previous versions of ERT are shipped with the Garden-Linux backend.  This produced for the Garden-Linux backend is described here.

Procedure

1) Find the app guid for the app container that we want to enter into using cf app <app-name> --guid, where <app-name> is the name of the app.

Example:

$ cf app spring-music --guid
e0afa9f2-4020-4da1-919c-59d1c66d7d3c 

2) Find the IP of the Diego Cell where the container app is hosted:

cf curl /v2/apps/<app-guid>/stats | grep -w host

where <app-guid> is the GUID of the app from the command in step 1)

Example:

cf curl /v2/apps/e0afa9f2-4020-4da1-919c-59d1c66d7d3c/stats | grep -w host
"host": "10.193.76.23", 

Note: There may be multiple IPs returned from the above command, like when there are multiple app instances.  In most cases, app instances will be split across multiple Cells so each one will have its own IP address.  Simply pick one of the IP addresses to use in the next step, and we'll enter the app instance container running on that Cell.

3) From the Ops Manager VM, locate the name of your deployment using `bosh deployments`.  It will be in the form `cf-<uuid>` for Elastic Runtime or Pivotal Application Service.  Run `export BOSH_DEPLOYMENT=<deployment-name>`.

4) Now find the name of the Diego Cell from the IP using the `bosh instances | grep <IP>` command, where IP is the IP address from step 2).

$ bosh instances | grep 10.193.82.13
diego_cell/4e404849-1479-4b1b-9ebc-0a5fe5ba0c6f running main 10.193.82.13

The VM name is denoted by "<job_name>/<uuid>". From the above command, "diego_cell" is the job name and index is "0", you would see `compute/<uuid>` if you're using the Small Footprint variant of PCF.

5) SSH to Diego cell using `bosh ssh <job_name>/<uuid>` command. Replace <name> and <uuid> with the the values from step 3).  You can then run `sudo -i` to become root.

Example:

$ bosh ssh diego_cell/4e404849-1479-4b1b-9ebc-0a5fe5ba0c6f
Using environment '10.193.82.11' as user 'director' (bosh.*.read, openid, bosh.*.admin, bosh.read, bosh.admin)

Using deployment 'cf-435c6aa7c377367d89f3'

Task 7625. Done
Unauthorized use is strictly prohibited. All access and activity
...

diego_cell/4e404849-1479-4b1b-9ebc-0a5fe5ba0c6f:~$ sudo -i
diego_cell/4e404849-1479-4b1b-9ebc-0a5fe5ba0c6f:~# 

6) Once SSH'ed into the Diego Cell, we can obtain the <container-guid> of the app by running the following command:

grep rep.executing-container-operation.ordinary-lrp-processor.process-reserved-container.run-container.containerstore-run.node-run.monitor-run.run-step.running \
/var/vcap/sys/log/rep/rep.stdout.log | grep <app-guid> | head -1 | python -m json.tool | grep 'container-guid'

Replace <app-guid> in the above command with the app-guid of the app from step 1)

Example:

# grep rep.executing-container-operation.ordinary-lrp-processor.process-reserved-container.run-container.containerstore-run.node-run.monitor-run.run-step.running \
/var/vcap/sys/log/rep/rep.stdout.log | grep e0afa9f2-4020-4da1-919c-59d1c66d7d3c | head -1 | python -m json.tool | grep 'container-guid'
"container-guid": "891090cc-259f-4648-4d45-43ea0d8d2af9"

7) If you're using one of the following versions, you can skip steps 7 & 8 and proceed with step #9.

Elastic Runtime

  • >= 1.8.40
  • >= 1.9.18
  • >= 1.10.5
  • 1.11 (all)
  • 1.12 (all)

Using the value of container-guid from output of the command in step 6), change directory to /var/vcap/data/garden/depot/<container-guid> directory. From the output in step 6), the container-GUID is 891090cc-259f-4648-4d45-43ea0d8d2af9. 

# cd  /var/vcap/data/garden/depot/891090cc-259f-4648-4d45-43ea0d8d2af9

8) Enter the guardian mount namespace:

# /var/vcap/packages/guardian/bin/inspector-garden -pid $(pidof guardian) /bin/bash

9) Drop into the container shell as root by running the following command:

# /var/vcap/packages/runc/bin/runc exec -t <container-guid> /bin/bash

Alternatively, to enter the container as a non-root user such as vcap, run the following command:

# /var/vcap/packages/runc/bin/runc exec -u 2000:2000 -t <container-guid> /bin/bash

10) To confirm you're in the container you can check the app processes by running `ps -ef`.

# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 2016 ? 00:00:00 /proc/self/exe init
vcap 13 0 0 2016 ? 00:00:00 /tmp/lifecycle/diego-sshd --allowedKeyExchanges= --address=0.0.0.0:2222 --allowUnauthenticatedClients=false --inheritDaemonEnv=true --allowedCiphers= --allo
vcap 19 0 0 2016 ? 00:22:36 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.util.logging.config.file=/home/vcap/app/.java-buildpack/tomcat/conf/logging.properties -Djava.ut
root 8581 0 0 14:15 ? 00:00:00 /bin/bash
root 8606 8581 0 14:15 ? 00:00:00 ps -ef
root 20334 1 0 Jan07 ? 00:00:00 [sudo] <defunct>
root@891090cc-259f-4648-4d45-43ea0d8d2af9:/#  

Additional Information

The commands above assume that you're using v2 of the bosh cli.  In Ops Manager 1.11 and 1.12, the name of the binary is `bosh2`.  It switches to `bosh` in Ops Manager 2.0.

For SSH'ing into a container with garden-Linux backend, please follow the procedure here. 

Comments

  • Avatar
    Theo Cushion

    We found that using: `/var/vcap/packages/runc/bin/runc exec -u 2000:2000 -t /bin/bash`
    This worked better for us as it sets the uid and gid to 2000, as opposed to setting the uid to 2000 and gid to 0. This was critical for us when we were trying to dump out a Java heap using jmap. I think it is a safe option to update the documentation above to explicitly set the gid to 2000 that will get around this kind of thing.

Powered by Zendesk