Pivotal Knowledge Base


Deleting GemFire Service Instance Returns Error "unexpected end of JSON input"


Pivotal Cloud Foundry Version 1.7.x
Pivotal Operations Manager Version 1.6.13
Pivotal Elastic Runtime Version 1.6.23-build.2
GemFire Tile Version


When a user attempts to delete a GemFire instance, the following error is observed:

Status: delete failed 
Message: An unexpected error occurred: "unexpected end of JSON input" 

Also, the following stack trace is observed in log file /var/vcap/sys/log/broker/broker.stderr found on the Gemfire service broker VM:

#135124 2016-08-15 10:22:40 ▶ CRI Failure: Preparing service instance: unexpected end of JSON input #135125 2016-08-15 10:22:40 ▶ CRI Stack of 1559 bytes: goroutine 275064 [running]: github.com/pivotal-cf/gemfire-broker/broker_logging.createStackTraceString(0x0, 0x0) /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/broker_logging/broker_logging.go:43 +0x89 github.com/pivotal-cf/gemfire-broker/broker_logging.(*BrokerLogger).Critical(0xc8200b67e0, 0xc8210fefa0, 0x41) /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/broker_logging/broker_logging.go:51 +0x3e github.com/pivotal-cf/gemfire-broker/management/instance.ConcreteServiceInstanceManager.deallocateInstance(0xc82000ff20, 0x24, 0xd7b5, 0x1f90, 0x2, 0xc8200c6a00, 0x13e4, 0xc820056600, 0x5a, 0xc8200e4c80, ...) /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/management/instance/service_instance_manager.go:485 +0x1c6 github.com/pivotal-cf/gemfire-broker/management/instance.ConcreteServiceInstanceManager.DeallocateInstance(0xc82000ff20, 0x24, 0xd7b5, 0x1f90, 0x2, 0xc8200c6a00, 0x13e4, 0xc820056600, 0x5a, 0xc8200e4c80, ...) /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/management/instance/service_instance_manager.go:453 +0x2ac github.com/pivotal-cf/gemfire-broker/management/instance.(*ConcreteServiceInstanceManager).DeallocateInstance(0xc820023400, 0xc8213dc011, 0x24, 0x0, 0x0) :13 +0xdc created by github.com/pivotal-cf/gemfire-broker/handlers.DeleteServiceInstance.ServeHTTP /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/handlers/delete_service_instance.go:70 +0xdba

GemFire locator vm at index 0 logs contain the following errors. Also, you can verify if this is the same error by running a curl command "curl" against the locator VM /v1/regions API endpoint.

[warning 2016/09/13 08:49:28.819 UTC locator6  tid=0x4f] Error for /pulse/
java.lang.OutOfMemoryError: Metaspace
    at javax.crypto.JarVerifier.verify(JarVerifier.java:250)
    at javax.crypto.JceSecurity.verifyProviderJar(JceSecurity.java:160)
    at javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:186)
    at javax.crypto.JceSecurity.canUseProvider(JceSecurity.java:200)
    at javax.crypto.KeyAgreement.getInstance(KeyAgreement.java:179)
    at sun.security.ssl.JsseJce.getKeyAgreement(JsseJce.java:290)


GemFire instances are created with 2 VMs that host the GemFire locator service which acts as the load balancers for GemFire clients. During a delete operation, the GemFire service broker will attempt to connect to the first locator using its rest API port. From the stack trace above, we can infer that the rest of the API call "Gemfire/v1/regions" is made to the designated locator VM. The locator is returning an error, java.lang.OutOfMemoryError: Metaspace but the CF client only sees the end of the JSON error.

$ curl
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /gemfire/v1/regions. Reason:
<pre>    Server Error</pre></p><h3>Caused by:</h3><pre>java.lang.OutOfMemoryError: Metaspace
<hr><i><small>Powered by Jetty://</small></i><hr/>



Collect logs for analysis

The root cause for JVM out of memory due to metaspace exhaustion can vary. We recommend following these steps to gather information for further investigation:

  1. Export the GemFire job and agent logs for the instance in question:
    cf gemfire export-gemfire SERVICE_INSTANCE
  2. Follow this article How to collect basic information for gemfire issues to collect GemFire related artifacts.
  3. Download the Gemfire Service Broker logs using Ops Manager. If Ops Manager is not available, logs can be found on broker VM under /var/vcap/sys/log/broker/*
  4. SSH into the Locator VM reporting the out of memory error and collect information about the JVM process:
    1. Run JPS to get a list of the running Java processes on the VM. There should only be one PID running, and that is the locator JVM:
    2. Collect jstack information. Multiple samples are helpful to ensure enough information is collected:
      sudo /var/vcap/packages/jdk7/bin/jstack -F <PID>
    3. Get multiple heap dump samples:
      sudo /var/vcap/packages/jdk7/bin/jmap -heap <PID>
    4. Generate a JVM binary heap dump:
      sudo /var/vcap/packages/jdk7/bin/jmap -dump:format=b,file=dump.hprof <PID>


Restart the locator instance VM to clear the memory issues and retry the delete operation. Make sure to collect all the logs before restarting the instance. 



Powered by Zendesk