Skip to content
This repository was archived by the owner on Sep 18, 2023. It is now read-only.
This repository was archived by the owner on Sep 18, 2023. It is now read-only.

Unexpected heap memory and off-heap memory allocation #1208

Description

@ziyangRen

Describe the bug
Running Spark tasks using gazelle should use more off-heap memory. However, when we run 2TB tpc-ds, we find that for most applications, the onheap memory usage is much greater than that in the off-heap memory, and some SQL statements do not even use off-heap memory, which troubles our parameter configuration.
notes:Traverse all SparkListenerTaskEnd events in the eventlog, obtain the maximum values of JVMHeapMemory and OffHeapExecutionMemory in all Task Executor Metrics.Used as the basis for determining the memory usage onheap and offheap.The test results are as follows(The memory unit is MB.):
image
image

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions