Pivotal Knowledge Base

Follow

Backup disk store bloat with GemFire 7 when killing the GemFire processes

Applies to

GemFire 7.0.2.x and earlier

Purpose

The purpose of this article is to describe why backup bloat can happen and how to avoid it.

Description

When doing a full online backup the backup diskstore is bigger than the cache in memory plus the persisted data.

Gemfire creates 1 GB oplog files, but the filesystem does not necessarily allocate the space for the oplog. Every time the GemFire member is bounced it starts with a new oplog. So the old oplog would be left reporting it's size as 1 GB, even though GemFire maybe only wrote the first few hundred bytes and the rest of the file is unallocated space.

Because the filesystem didn't actually allocate that space, those 1 GB files wouldn't actually take up 1 GB as reported by the df command, but when copied all of the zeros will be included and the resulting file actually takes up 1 GB.

Check to see if this is the issue by doing an ls -l on each one of the copies of the disk store: online, backup. If the online disk store has a lot of 1 GB files that actually add up to a lot more that 6 GB when you look at what is reported by ls -l then you are probably experiencing this issue.

Solution

This issue was fixed in GemFire 8.0 with the fix of bug #44113.

Work Around

You only end up with an oplog with empty, unallocated space if you kill -9 the GemFire process. If shutting down the GemFire members using gfsh shutdown or Java commands this issue doesn't happen.

Comments

Powered by Zendesk