Pivotal Knowledge Base

Follow

How to run errands step-by-step

Environment

Product Version
Pivotal Cloud Foundry® (PCF) 1.7.x

Purpose

It is common to see errands fail to run when deploying PCF. However, it is sometimes very hard to figure out how they fail. If there was a way to run the errand step-by-step, we can reproduce the issue which it is a great help in finding the root cause.

For example, the following error was seen when running the smoke test errand. In the next section, I will show how to rerun this errand step-by-step.

[stderr]
+ which go
+ local_gopath=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace
+ mkdir -p /var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin
+ export GOPATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests
+ GOPATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests
+ export PATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/bin:/var/vcap/packages/cli/bin:/var/vcap/data/packages/golang1.6/85a489b7c0c2584aa9e0a6dd83666db31c6fc8e8.1-efe60220544a82382eba1e42113f25920146eb7e/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ PATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/bin:/var/vcap/packages/cli/bin:/var/vcap/data/packages/golang1.6/85a489b7c0c2584aa9e0a6dd83666db31c6fc8e8.1-efe60220544a82382eba1e42113f25920146eb7e/bin:/usr/sbin:/usr/bin:/sbin:/bin
+ go install -v github.com/onsi/ginkgo/ginkgo
github.com/onsi/ginkgo/config
github.com/onsi/ginkgo/ginkgo/convert
github.com/onsi/ginkgo/ginkgo/interrupthandler
github.com/onsi/ginkgo/ginkgo/nodot
github.com/onsi/ginkgo/ginkgo/testsuite
github.com/onsi/ginkgo/types
github.com/onsi/ginkgo/reporters/stenographer
github.com/onsi/ginkgo/reporters
github.com/onsi/ginkgo/internal/remote
github.com/onsi/ginkgo/ginkgo/testrunner
github.com/onsi/ginkgo/ginkgo/watch
github.com/onsi/ginkgo/ginkgo
+ ginkgo --noColor -r --succinct -slowSpecThreshold=300


===== 2016-06-20 06:12:11 UTC Finished "bundle exec bosh -n run errand smoke-tests"; Duration: 593s; Exit Status: 1
{"type": "step_finished", "id": "errands.running.cf-8c094c43c58cba45c3f5.smoke-tests"}
Exited with 1.

Resolution

In order to run the errand step-by-step, several steps need to be completed to set up an environment. Here I am attaching an example describing "how to run the smoke test errand?"

1. Set the current deployment. 

ubuntu@pivotal-ops-manager:/var/tempest/workspaces/default/deployments$ bosh deployment cf-2c3ea6cf6f11d78615c3.yml
Deployment set to '/var/tempest/workspaces/default/deployments/cf-2c3ea6cf6f11d78615c3.yml'

2. Run the specified errand.

Do "run errand" with the "--keep-alive" option, which prevents deletions of the VM after running the errand. This is important as we need this VM to rerun commands in this errand.

ubuntu@pivotal-ops-manager:~$ bosh run errand smoke-tests --keep-alive
Acting as user 'director' on 'p-bosh-05f96aa4a0bc7e91b12f'

RSA 1024 bit CA certificates are loaded due to old openssl compatibility
Director task 2302
  Started preparing deployment > Preparing deployment. Done (00:00:02)

  Started preparing package compilation > Finding packages to compile. Done (00:00:01)

  Started running errand > smoke-tests/0. Done (00:03:11)

  Started fetching logs for smoke-tests/f5e94f33-7e65-4890-956a-3906ae602321 (0) > Finding and packing log files. Done (00:00:01)

Task 2302 done

3. Log on to the smoke test VM.

ubuntu@pivotal-ops-manager:~$ bosh ssh
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
1. diego_brain-partition-1506967bb2eea78b2b46/0 (35cd43ae-1f87-41cb-8207-66e72372736e)
2. cloud_controller-partition-1506967bb2eea78b2b46/0 (122690f3-88b7-401b-b519-f055decec276)
3. diego_cell-partition-1506967bb2eea78b2b46/1 (1334adf4-c115-44c5-93b7-e5b04e88d031)
4. diego_database-partition-1506967bb2eea78b2b46/0 (e1f20540-52d6-4041-9148-647d15f882b1)
5. diego_cell-partition-1506967bb2eea78b2b46/0 (8a8a7548-d6f1-4f5c-bcc3-9f914df15094)
6. mysql_proxy-partition-1506967bb2eea78b2b46/0 (89b08fe8-6cf5-4ff3-8d57-23821d8d83f0)
7. nats-partition-1506967bb2eea78b2b46/0 (ebce5bb1-aabf-419e-9946-1b78181c6cdb)
8. etcd_server-partition-1506967bb2eea78b2b46/0 (07009616-5aaa-4555-aa83-42c9061f6186)
9. smoke-tests/0 (f5e94f33-7e65-4890-956a-3906ae602321)
10. clock_global-partition-1506967bb2eea78b2b46/0 (04a8d03a-ec51-47a1-abac-ed970f77a998)
11. uaa-partition-1506967bb2eea78b2b46/0 (08582c1b-7891-4c6e-a413-0411f699d9a6)
12. loggregator_trafficcontroller-partition-1506967bb2eea78b2b46/0 (01e2efb3-d49d-4935-ab02-402bc67d49e4)
13. nfs_server-partition-1506967bb2eea78b2b46/0 (f5ad04d2-aec6-41a3-a535-75d04356de14)
14. mysql-partition-1506967bb2eea78b2b46/0 (36ed0315-22a1-4530-b568-ad6e7b823018)
15. cloud_controller_worker-partition-1506967bb2eea78b2b46/0 (9aeed684-c884-4a1a-861b-912c2b2de9e5)
16. ha_proxy-partition-1506967bb2eea78b2b46/0 (ffd4bfaa-f060-4230-807a-10a9992212fb)
17. doppler-partition-1506967bb2eea78b2b46/0 (74a9a8b7-fc5d-4351-9e63-a38a67b0f028)
18. consul_server-partition-1506967bb2eea78b2b46/0 (5de99499-4045-48b8-9226-e5937c84f93e)
19. router-partition-1506967bb2eea78b2b46/0 (c66d0314-86c9-43da-a606-100d49bd80ee)
Choose an instance: 9
Acting as user 'director' on deployment 'cf-2c3ea6cf6f11d78615c3' on 'p-bosh-05f96aa4a0bc7e91b12f'
Target deployment is 'cf-2c3ea6cf6f11d78615c3'

Setting up ssh artifacts

bosh_ihhyo6to1@3c7ecde6-6c46-4044-8362-169571083f6a:~$ 

4. Switch to user root and mount /tmp as exec. 

bosh_z3qsk6wkk@3c7ecde6-6c46-4044-8362-169571083f6a:~$ sudo su
root@3c7ecde6-6c46-4044-8362-169571083f6a:/var/vcap/bosh_ssh/bosh_z3qsk6wkk$ mount -o remount,exec /tmp

5.  Run the errand scripts.

The errand script can be found at "/var/vcap/jobs/<errand>/bin/run" (where "<errand>" is the name like smoke-tests).  This is the script that BOSH runs and it would allow you to re-run the whole errand. 

root@95b474c6-5532-441c-8f4e-0f4626e8482b:/var/vcap/jobs/smoke-tests/bin# /bin/bash run
################################################################################################################
go version go1.6.1 linux/amd64
CONFIG=/var/vcap/jobs/smoke-tests/bin/config.json
CONFIG=/var/vcap/jobs/smoke-tests/bin/config.json
GOPATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests
GOROOT=/var/vcap/data/packages/golang1.6/85a489b7c0c2584aa9e0a6dd83666db31c6fc8e8.1-fdfacf5e19519cb806ec88b57e3b693b4363e941
HOME=/root
LANG=en_US.UTF-8
LESSCLOSE=/usr/bin/lesspipe %s %s
LESSOPEN=| /usr/bin/lesspipe %s
LOGNAME=root
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.axa=00;36:*.oga=00;36:*.spx=00;36:*.xspf=00;36:
MAIL=/var/mail/root
OLDPWD=/var/vcap/jobs/smoke-tests/bin
PATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/bin:/var/vcap/packages/cli/bin:/var/vcap/data/packages/golang1.6/85a489b7c0c2584aa9e0a6dd83666db31c6fc8e8.1-fdfacf5e19519cb806ec88b57e3b693b4363e941/bin:/var/vcap/bosh/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
PWD=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests
SHELL=/bin/bash
SHLVL=2
SUDO_COMMAND=/bin/su
SUDO_GID=1002
SUDO_UID=1001
SUDO_USER=bosh_3xhsscuqp
TERM=xterm
USERNAME=root
USER=root
_=/usr/bin/env
################################################################################################################
Running smoke tests...
+ which go
/var/vcap/data/packages/golang1.6/85a489b7c0c2584aa9e0a6dd83666db31c6fc8e8.1-fdfacf5e19519cb806ec88b57e3b693b4363e941/bin/go
+ local_gopath=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace
+ mkdir -p /var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin
+ export GOPATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests
+ GOPATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace:/var/vcap/packages/smoke-tests
+ export PATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/bin:/var/vcap/packages/cli/bin:/var/vcap/data/packages/golang1.6/85a489b7c0c2584aa9e0a6dd83666db31c6fc8e8.1-fdfacf5e19519cb806ec88b57e3b693b4363e941/bin:/var/vcap/bosh/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+ PATH=/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/src/github.com/cloudfoundry/cf-smoke-tests/Godeps/_workspace/bin:/var/vcap/packages/smoke-tests/bin:/var/vcap/packages/cli/bin:/var/vcap/data/packages/golang1.6/85a489b7c0c2584aa9e0a6dd83666db31c6fc8e8.1-fdfacf5e19519cb806ec88b57e3b693b4363e941/bin:/var/vcap/bosh/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
+ go install -v github.com/onsi/ginkgo/ginkgo
+ ginkgo --noColor -r --succinct -slowSpecThreshold=300
[1473060148] CF-Logging-Smoke-Tests - 2/2 specs •S SUCCESS! 1m33.378018168s PASS
[1473060148] CF-Runtime-Smoke-Tests - 2/2 specs •S SUCCESS! 1m36.902579885s PASS

Ginkgo ran 2 suites in 3m13.374180355s
Test Suite Passed
Smoke Tests Complete; exit status: 0
root@95b474c6-5532-441c-8f4e-0f4626e8482b:/var/vcap/jobs/smoke-tests/bin#

 6. Destroy the errand VM.

Run the errand again without --keep-alive. BOSH will then destroy the errand VM after it runs. 

Impact/Risks 

Some errands assume that they'll be run in a fresh VM and when you use --keep-alive, the VM does not get recycled. Because of this reason, the errand may fail for unrelated reasons when it is run for the second time. Under these circumstances, please contact Pivotal support and provide details of the specific failure that's occurring in your environment.

 

Comments

  • Avatar
    David Zhou
    1. If you hit error "fork/exec /tmp/ginkgo848850644/logging.test: permission denied", please execute 'sudo mount -o remount,exec /tmp' after step 3. 
    2. Sometimes the errand requires root permission. So please run as root in step 4, for example "sudo su".          
    Edited by David Zhou
Powered by Zendesk