Note: This article applies to PHD 1.1.1 / HDB 1.1.3. With Feb release of PHD 2.x, we may see different options to perform expansion.
How to modify the number of segments in an existing HDB cluster. Example: Increasing the number from 2 segment per node to 3 segment per node.
Current Installation Approach :
HDB could be installed in 2 ways:
- Using icm_client to deploy PHD cluster & HDB binaries and then initialize HDB
- Manually install required PHD HDFS binaries & then initialize HDB
With an install performed using icm_client, the number of segment couldn't be modified. If hdb segment or master directories are modified, and an attempt to reconfigure HDB is made using icm_client, it would result in an error indicating that these parameters couldn't be changed.
In such scenarios, as a workaround, since as of now HDB does not support online expansion, HDB cluster needs to be reinitialized again. However, icm_client would not be aware of the change made but would not cause any operational impact, although while fetching the current configuration of the cluster using icm_client fetch-configuration, the retrieved cluserconfig.xml will still show the number of directories as deployed initially.
Flexibility to allow such a change and identify current state of cluster is under consideration & we would see future versions to provide relevant options for the same.
Below is typical list of steps which must be performed to modify the number of segments. The same steps applies even if manual install of PHD binaries is performed.
- Backup HDB database using pg_dump utility
- Stop HDB database: gpstop -af
- Stop icm_cluster
- Delete directories from HDB segment, master & HDFS. (Change the below directories as per your cluster configuration, these are mentioned here as an example)
On segments: rm -rf /data1/hawq/primary/gpseg*
On master & standby rm -rf /data/master/gpseg*
On HDFS: sudo -u hdfs hdfs dfs -rm -r /hawq_data/gpseg*
- Edit gpinitsystem_config and modify DATA_DIRECTORY parameter at 2 places with the required number of segment / each server. 1 directory value represents 1 segment on a server, so modify it accordingly.
- Initialize HDB database
gpinitsystem -c /etc/gphd/hawq/conf/gpinitsystem_config -s <standbyhostname>
- Verify the database
- Restore HDB database using pg_restore utility from the backup taken while starting this activity