Pivotal Knowledge Base


Stop of all services via Ambari fail with "Error: rpmdb open failed"


Product Version
Pivotal HD 3.0.x
Ambari 1.7.x / 2.x


When trying to stop all services via Ambari, the operation may fail with the following errors:

Error Message:

Ambari will show the following errors on the nodes marked in red:

2016-09-08 09:48:50,965 - Package['unzip'] {}
2016-09-08 09:48:51,003 - Installing package unzip ('/usr/bin/yum -d 0 -e 0 -y install unzip')
2016-09-08 09:48:51,110 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 123, in execute
File "/var/lib/ambari-agent/cache/stacks/PHD/2.0.6/hooks/before-INSTALL/scripts/hook.py", line 34, in hook
File "/var/lib/ambari-agent/cache/stacks/PHD/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py", line 63, in install_packages
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 107, in __new__
cls(name.pop(0), env, provider, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 40, in action_install
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 36, in install_package
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 36, in checked_call
return _call(command, logoutput, True, cwd, env, preexec_fn, user, wait_for_finish, timeout, path)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in _call
raise Fail(err_msg)
Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install unzip' returned 1. rpmdb: Thread/process 2335/139851783018240 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 - (-30974)
error: cannot open Packages database in /var/lib/rpm

Error: rpmdb open failed


The rpm database is corrupt and needs to be rebuilt. This was likely caused by a file system crash or file system corruption, /var/log/messages should be checked to look for hardware/os configuration errors.


1. Log on via SSH, to the node that is showing up in red in the Ambari shutdown operation (dn02 and dn05 above).

2. Confirm that the Operating system/hardware is healthy by checking /var/log/messages and other system monitoring tools.

3. Backup and remove the current rpm database files:

mv /var/lib/rpm/__db* /tmp/

4. Rebuild the rpm database:

rpm -rebuilddb

5. Clean the caches in yum and confirm that yum works correctly:

yum clean all




Powered by Zendesk