Reason for out of memory error metaspace:

  1. Too many classes
  2. Too big classes being loaded into the Metaspace.

Observation from heap dump:

Below are 2 heap dumps from a microservice that has no metaspace out-of-memory-error.
In day 1 we have 24,453 objects, in day 2 we have 24,516 objects.

Below are 2 heap dumps from a microservice with metaspace out-of-memory-error.
In day 1 we have 37,855 objects, in day 2 we have 38,988 objects. We have 1,133 objects added into the heap in 1 day.

The new objects are sun.reflect.GeneratedMethodAccessor and sun.reflect.GeneratedSerializationConstructorAccessor.
These new objects are generated when we trigger API calls but not getting clear during garbage collection.

Solution suggested in google:

  1. Increase -XX:MetaspaceSize and -XX:MaxMetaspaceSize.
    Since in Java 8, meterSpace is part of native space, we can use as much memory as we want with out using the JVM memory. However, this approach will only postpone the problem until it reach the new -XX:MaxMetaspaceSize.

Solutions that we tried:

  1. Cache: Cache bigger API in Gateway, Service and API level
    Result: This approach did not work in out case, because (1) many of the request are unique, new API request will send anyway. (2) Even with cache, objects are still increasing.

  2. Introduce a life cycle mechanism which nullify bigger objects after the API call.
    Result: We did not implement this mechanism, because we found out solution #3 resolved the issue.

  3. Removing-Xnoclassgc and -XX:+DisableExplicitGCfrom JVM Startup arguments

    -Xnoclassgc: This will prevent unload classes during GC.
    "if an application needs to load a large number of classes or during runtime some set of classes become unreachable and it loads another set of new classes, and the application happens to be running with –Xnoclassgc, then it runs the risk of hitting the maximum capacity of the PermGen/Metaspace and so failing with an OutOfMemoryError."

    -XX:+DisableExplicitGC: "there are potential unintended consequences: For example, in some JVM implementations, core Java functionality such as DirectByteBuffer cleanup may be affected in some situations, leading to unnecessary OutOfMemoryErrors and crashes since the self-healing calls to System.gc to cleanup iceberg native objects have no effect."

    Result: Below graph show metaspace increase between Aug 03 and Aug 05. After we remove -Xnoclassgc and -XX:+DisableExplicitGC from JVM startup arguments at Aug 05, the metaspace is getting properly garbage collected.

results matching ""

    No results matching ""