Pivotal Knowledge Base

Follow

Stop of all services via Ambari fail with "Error: rpmdb open failed"

Environment

Product Version
Pivotal HD 3.0.x
Ambari 1.7.x / 2.x

Symptom

When trying to stop all services via Ambari, the operation may fail with the following errors:

Error Message:

Ambari will show the following errors on the nodes marked in red:

2016-09-08 09:48:50,965 - Package['unzip'] {}
2016-09-08 09:48:51,003 - Installing package unzip ('/usr/bin/yum -d 0 -e 0 -y install unzip')
2016-09-08 09:48:51,110 - Error while executing command 'install':
Traceback (most recent call last):
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 123, in execute
method(env)
File "/var/lib/ambari-agent/cache/stacks/PHD/2.0.6/hooks/before-INSTALL/scripts/hook.py", line 34, in hook
install_packages()
File "/var/lib/ambari-agent/cache/stacks/PHD/2.0.6/hooks/before-INSTALL/scripts/shared_initialization.py", line 63, in install_packages
Package(packages)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 107, in __new__
cls(name.pop(0), env, provider, **kwargs)
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 148, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 149, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 115, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/__init__.py", line 40, in action_install
self.install_package(package_name)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/package/yumrpm.py", line 36, in install_package
shell.checked_call(cmd)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 36, in checked_call
return _call(command, logoutput, True, cwd, env, preexec_fn, user, wait_for_finish, timeout, path)
File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py", line 102, in _call
raise Fail(err_msg)
Fail: Execution of '/usr/bin/yum -d 0 -e 0 -y install unzip' returned 1. rpmdb: Thread/process 2335/139851783018240 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 - (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed

Cause

The rpm database is corrupt and needs to be rebuilt. This was likely caused by a file system crash or file system corruption, /var/log/messages should be checked to look for hardware/os configuration errors.

Resolution

1. Log on via SSH, to the node that is showing up in red in the Ambari shutdown operation (dn02 and dn05 above).

2. Confirm that the Operating system/hardware is healthy by checking /var/log/messages and other system monitoring tools.

3. Backup and remove the current rpm database files:

mv /var/lib/rpm/__db* /tmp/

4. Rebuild the rpm database:

rpm -rebuilddb

5. Clean the caches in yum and confirm that yum works correctly:

yum clean all

 

 

Comments

Powered by Zendesk