Pivotal Knowledge Base

Follow

BOSH deploy reports "Fetching package blob: Getting blob from inner blobstore: SHA1 mismatch"

Environment

Product Version
Pivotal Cloud Foundry® (PCF) 1.6.x, 1.7.x

Symptom

When a customer was trying to update one of the etcd partition jobs through BOSH, the job fails with the following error message:

Started updating job etcd_server-partition > etcd_server-partition/1 (b9799760-5b4f-4fdb-baee-2016c0380a02) (canary). Failed: Action Failed get_task: Task f5c7c48b-ae7c-4883-5032-7f77005ba5cd result: Applying: Applying job etcd: Applying package etcd-common for job etcd: Fetching package blob: Getting blob from inner blobstore: SHA1 mismatch. Expected 594e2aa92ccf2188475b5e417de29e19d1e98d38, got 112ed609bab4fa5b45b108cc848ae7858084ca02 for blob /var/vcap/data/tmp/bosh-blobstore-externalBlobstore-Get489845863 (00:00:01)

Error 450001: Action Failed get_task: Task f5c7c48b-ae7c-4883-5032-7f77005ba5cd result: Applying: Applying job etcd: Applying package etcd-common for job etcd: Fetching package blob: Getting blob from inner blobstore: SHA1 mismatch. Expected 594e2aa92ccf2188475b5e417de29e19d1e98d38, got 112ed609bab4fa5b45b108cc848ae7858084ca02 for blob /var/vcap/data/tmp/bosh-blobstore-externalBlobstore-Get489845863

Cause

One possible cause for this issue is a bug in BOSH that allowed agents to consider packages as successfully compiled even though the package upload failed. The Director ends up saving that SHA1 and considers that package successfully compiled. This is not the only cause, though. Anything that would cause the SHA1 that is saved in the director's database to not match the actual SHA1 of the file in the blob store would cause the same error.

Resolution

One way to fix this is to remove compiled packages and kick off deploy again.

These are the detailed steps:

1. Connect via Single SHell (SSH) into the director VM. 
2. sudo su.
3. /var/vcap/jobs/director/bin/director_ctl console.
4. Bosh::Director::Models::CompiledPackage.all.map(&:blobstore_id).
5. Bosh::Director::Models::CompiledPackage.all.each(&:delete).
6. Kick off OpsMgr deploy.

Step 4 shows which blobs will be deleted. Record the result, if in future we want to clean up blobs that will be deleted from the database.

Step 5 deletes all compiled packages in the system which means that the Ops Manager deploy will try to recompile every single package.

Additional Information

It is also possible to look up the package with specific sha1 value: Bosh::Director::Models::CompiledPackage[:sha1 => "xxx"].

This is output from my environment.

irb(main):107:0* Bosh::Director::Models::CompiledPackage[:sha1 => "65764b981509abc66961f98d2445a6e56ad373b6"]
=> #<Bosh::Director::Models::CompiledPackage @values={:id=>919, :blobstore_id=>"41d899e0-6c4e-47f2-7a52-33fbd79dd3f7", :sha1=>"65764b981509abc66961f98d2445a6e56ad373b6", :dependency_key=>"[[\"golang\",\"aa5f90f06ada376085414bfc0c56c8cd67abba9c\"]]", :build=>1, :package_id=>447, :dependency_key_sha1=>"305b0176f81c385325fcb056f4a9b5cddfd76fc5", :stemcell_os=>"ubuntu-trusty", :stemcell_version=>"3232.8"}>

 Then you can delete this specific package using Bosh::Director::Models::CompiledPackage[:sha1 => "xxx"].delete. 

irb(main):047:0* Bosh::Director::Models::CompiledPackage[:shal => "65764b981509abc66961f98d2445a6e56ad373b6"].delete
=>  #<Bosh::Director::Models::CompiledPackage @values={:id=>919, :blobstore_id=>"41d899e0-6c4e-47f2-7a52-33fbd79dd3f7", :sha1=>"65764b981509abc66961f98d2445a6e56ad373b6", :dependency_key=>"[[\"golang\",\"aa5f90f06ada376085414bfc0c56c8cd67abba9c\"]]", :build=>1, :package_id=>447, :dependency_key_sha1=>"305b0176f81c385325fcb056f4a9b5cddfd76fc5", :stemcell_os=>"ubuntu-trusty", :stemcell_version=>"3232.8"}>

On checking again, we find that this package doesn't exist anymore. 

irb(main):102:0* Bosh::Director::Models::CompiledPackage[:shal => "65764b981509abc66961f98d2445a6e56ad373b6"]
=> nil 

Kicking off the deploy again will recompile the package that you deleted; however this may not be the only bad package. This is why we recommend deleting all of the compiled packages above. Rather than fixing one bad package at a time, you can fix them all at the same time."

 

Comments

Powered by Zendesk