Pivotal Knowledge Base

Follow

Deleting GemFire Service Instance Returns Error "unexpected end of JSON input"

Environment

 Product  Version
 Pivotal Cloud Foundry  1.7.x
 Pivotal Operations Manager  1.6.13
 Pivotal Elastic Runtime  1.6.23-build.2
 GemFire Tile   1.6.0.0

Symptom

When a user attempts to delete a GemFire instance, the following error is observed:

Status: delete failed 
Message: An unexpected error occurred: "unexpected end of JSON input" 

Also, the following stack trace is observed in log file /var/vcap/sys/log/broker/broker.stderr found on the Gemfire service broker VM:

#135124 2016-08-15 10:22:40 ▶ CRI Failure: Preparing service instance: unexpected end of JSON input
#135125 2016-08-15 10:22:40 ▶ CRI Stack of 1559 bytes:
goroutine 275064 [running]:
github.com/pivotal-cf/gemfire-broker/broker_logging.createStackTraceString(0x0, 0x0)
        /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/broker_logging/broker_logging.go:43 +0x89
github.com/pivotal-cf/gemfire-broker/broker_logging.(*BrokerLogger).Critical(0xc8200b67e0, 0xc8210fefa0, 0x41)
        /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/broker_logging/broker_logging.go:51 +0x3e
github.com/pivotal-cf/gemfire-broker/management/instance.ConcreteServiceInstanceManager.deallocateInstance(0xc82000ff20, 0x24, 0xd7b5, 0x1f90, 0x2, 0xc8200c6a00, 0x13e4, 0xc820056600, 0x5a, 0xc8200e4c80, ...)
        /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/management/instance/service_instance_manager.go:485 +0x1c6
github.com/pivotal-cf/gemfire-broker/management/instance.ConcreteServiceInstanceManager.DeallocateInstance(0xc82000ff20, 0x24, 0xd7b5, 0x1f90, 0x2, 0xc8200c6a00, 0x13e4, 0xc820056600, 0x5a, 0xc8200e4c80, ...)
        /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/management/instance/service_instance_manager.go:453 +0x2ac
github.com/pivotal-cf/gemfire-broker/management/instance.(*ConcreteServiceInstanceManager).DeallocateInstance(0xc820023400, 0xc8213dc011, 0x24, 0x0, 0x0)
        :13 +0xdc
created by github.com/pivotal-cf/gemfire-broker/handlers.DeleteServiceInstance.ServeHTTP
        /var/vcap/data/compile/broker/gospace/src/github.com/pivotal-cf/gemfire-broker/handlers/delete_service_instance.go:70 +0xdba

GemFire locator vm at index 0 logs contains the following errors. Also, you can verify if this is the same error by running a curl command "curl http://10.1.1.80:8080/gemfire/v1/regions" against the locator VM /v1/regions API endpoint.

[warning 2016/09/13 08:49:28.819 UTC locator6  tid=0x4f] Error for /pulse/
java.lang.OutOfMemoryError: Metaspace
    at javax.crypto.JarVerifier.verify(JarVerifier.java:250)
    at javax.crypto.JceSecurity.verifyProviderJar(JceSecurity.java:160)
    at javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:186)
    at javax.crypto.JceSecurity.canUseProvider(JceSecurity.java:200)
    at javax.crypto.KeyAgreement.getInstance(KeyAgreement.java:179)
    at sun.security.ssl.JsseJce.getKeyAgreement(JsseJce.java:290)

Cause 

GemFire instances are created with 2 VMs that host theGemFire locator service which acts as the load balancers for GemFire clients. During a delete operation, the GemFire service broker will attempt to connect to the first locator using its rest API port. From the stack trace above, we can infer that the rest of the API call "Gemfire/v1/regions" is made to the designated locator VM. The locator is returning an error, java.lang.OutOfMemoryError: Metaspace but the CF client only sees the end of the JSON error.

$ curl http://10.1.1.80:8080/gemfire/v1/regions
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
<title>Error 500 Server Error</title>
</head>
<body><h2>HTTP ERROR 500</h2>
<p>Problem accessing /gemfire/v1/regions. Reason:
<pre>    Server Error</pre></p><h3>Caused by:</h3><pre>java.lang.OutOfMemoryError: Metaspace
</pre>
<hr><i><small>Powered by Jetty://</small></i><hr/>

</body>
</html>

Resolution

Collect logs for analysis

The root cause for JVM out of memory due to metaspace exhaustion can vary. We recommend following these steps to gather information for further investigation:

  1. Export the GemFire job and agent logs for the instance in question:
    cf gemfire export-gemfire SERVICE_INSTANCE
  2. Follow this article How to collect basic information for gemfire issues to collect GemFire related artifacts.
  3. Download the Gemfire Service Broker logs using Ops Manager. If Ops Manager is not available, logs can be found on broker VM under /var/vcap/sys/log/broker/*
  4. SSH into the Locator VM reporting the out of memory error and collect information about the JVM process:
    1. Run JPS to get a list of the running Java processes on the VM. There should only be one PID running, and that is the locator JVM:
      /var/vcap/packages/jdk7/bin/jps
    2. Collect jstack information. Multiple samples are helpful to ensure enough information is collected:
      sudo /var/vcap/packages/jdk7/bin/jstack -F <PID>
    3. Get multiple heap dump samples:
      sudo /var/vcap/packages/jdk7/bin/jmap -heap <PID>
    4. Generate a JVM binary heap dump:
      sudo /var/vcap/packages/jdk7/bin/jmap -dump:format=b,file=dump.hprof <PID>

Workaround

Restart the locator instance VM to clear the memory issues and retry the delete operation. Make sure to collect all the logs before restarting the instance. 

 

Comments

Powered by Zendesk