Pivotal Knowledge Base

Follow

Maintenance procedures and running the HDFS balancer in a HAWQ/HDB 1.x Cluster

Environment

Product Version
Pivotal HDB

1.x

Pivotal HD 2.x

Purpose

This article is to provide some general guidance and answers around HAWQ 1.x maintenance operations. It discusses certain cluster maintenance procedures, including balancing HDFS storage across DataNodes in HAWQ 1.x.

Overview

In HAWQ 1.x, DataNode replacement, DataNode Decommissioning or a disk loss can affect data locality and distribution. Performance may be affected by some level. After a replacement or recommission, there can be a negative impact if segment servers have to then fetch certain data blocks over the network during an HDFS read. Regardless of whether data blocks from the original node are still in place or lost.

Refer to the section, If the hosts are HAWQ Segment Servers, within Shutting Down The Slave Node.

As discussed in the maintenance procedures covered below, running the HDFS balancer to balance out storage across DataNodes further exacerbates this impact in the entire HAWQ cluster.  If you are going to run the balancer, a HAWQ table redistribution plan should follow rebalancing.

Depending upon cluster usage, workload and resource availability, you also should consider the impacts of table redistribution outside of a maintenance window.

The decision to rebalance might be made as a result of various cluster operations, such as those listed below, or over time. 

Maintenance Procedures and Considerations

HAWQ 1.x and the Decommissioning of Slave Nodes:

If you have to decommission a Slave/Worker Node where the host is also running HAWQ segments:

  1. First, decommission the Slave from HDFS, YARN, etc, by following their relevant procedures outlined in Decommissioning Slave Nodes.
  2. Shut the relevant Slave Node processes down, as outlined in Shutting Down the Slave Node.

HAWQ 1.x and Slave Node Replacement:

In the situation where an entire Slave/Worker Node running HAWQ 1.x segments must be replaced (and could not be decommissioned), follow the procedures outlined in Replacing the Slave Node.

HAWQ 1.x and Disk Replacement:

When a disk on Slave/Worker Node fails, follow the steps for Replacing the Slave Node Disk.  

Balancer considerations:

  • Complete all maintenance operations prior to running the balancer.
  • Do not run the HDFS balancer unless you are prepared to redistribute tables again. 
  • Balancer should run on a machine that can support the network socket connections from the DataNodes.
  • Balancer must be located on the same network as DataNodes and NameNode. 
  • Do not run the balancer on a NameNode if you do not have to.  The increased socket connections could impact NameNode operations.
  • Spend time to determine network throughput available for the balancer operation.
  • Set balancer bandwidth appropriately:
    • Via the hdfs utility. No server restart needed.  But is not persistent across DataNode restarts:
sudo -u hdfs hdfs dfsadmin -help setBalancerBandwidth <BytesPerSecondPerDataNode>
    •  Or update dfs.datanode.balance.bandwidthPerSec in hdfs-site.xml.  (Requires DataNodes to be restarted).

Redistributing HAWQ data:

  • Redistributing data can be resource intensive, so plan accordingly. All HAWQ segments should be online and balancing completed.
  • Upon the completion of the balancer, individual tables can be redistributed, while maintaining the existing distribution policy, by running the following in psql by the gpadmin user:

Note: Due to a known bug, you must turn off the optimizer in HAWQ 1.2.x and 1.3.x for table redistribution to work

set optimizer=off;
ALTER TABLE <table_name> SET WITH (REORGANIZE=TRUE);

Comments

Powered by Zendesk