Skip to main content
Skip to main content
Edit this page

Allocation profiling

ClickHouse uses jemalloc as its global allocator. Jemalloc comes with some tools for allocation sampling and profiling.
To make allocation profiling more convenient, SYSTEM commands are provided along with four letter word (4LW) commands in Keeper.

Sampling allocations and flushing heap profiles​

If you want to sample and profile allocations in jemalloc, you need to start ClickHouse/Keeper with profiling enabled using the environment variable MALLOC_CONF:

jemalloc will sample allocations and store the information internally.

You can tell jemalloc to flush the current profile by running:

SYSTEM JEMALLOC FLUSH PROFILE

By default, the heap profile file will be generated in /tmp/jemalloc_clickhouse._pid_._seqnum_.heap where _pid_ is the PID of ClickHouse and _seqnum_ is the global sequence number for the current heap profile.
For Keeper, the default file is /tmp/jemalloc_keeper._pid_._seqnum_.heap, and follows the same rules.

A different location can be defined by appending the MALLOC_CONF environment variable with the prof_prefix option.
For example, if you want to generate profiles in the /data folder where the filename prefix will be my_current_profile, you can run ClickHouse/Keeper with the following environment variable:

The generated file will be appended to the prefix PID and sequence number.

Analyzing heap profiles​

After heap profiles have been generated, they need to be analyzed.
For that, jemalloc's tool called jeprof can be used. It can be installed in multiple ways:

  • Using the system's package manager
  • Cloning the jemalloc repo and running autogen.sh from the root folder. This will provide you with the jeprof script inside the bin folder
Note

jeprof uses addr2line to generate stacktraces which can be really slow.
If that's the case, it is recommended to install an alternative implementation of the tool.

There are many different formats to generate from the heap profile using jeprof. It is recommended to run jeprof --help for information on the usage and the various options the tool provides.

In general, the jeprof command is used as:

If you want to compare which allocations happened between two profiles you can set the base argument:

Examples​

  • if you want to generate a text file with each procedure written per line:
  • if you want to generate a PDF file with a call-graph:

Generating a flame graph​

jeprof allows you to generate collapsed stacks for building flame graphs.

You need to use the --collapsed argument:

After that, you can use many different tools to visualize collapsed stacks.

The most popular is FlameGraph which contains a script called flamegraph.pl:

Another interesting tool is speedscope that allows you to analyze collected stacks in a more interactive way.

Controlling allocation profiler during runtime​

If ClickHouse/Keeper is started with the profiler enabled, additional commands for disabling/enabling allocation profiling during runtime are supported. Using those commands, it's easier to profile only specific intervals.

To disable the profiler:

SYSTEM JEMALLOC DISABLE PROFILE

To enable the profiler:

SYSTEM JEMALLOC ENABLE PROFILE

It's also possible to control the initial state of the profiler by setting the prof_active option which is enabled by default.
For example, if you don't want to sample allocations during startup but only after, you can enable the profiler. You can start ClickHouse/Keeper with the following environment variable:

The profiler can be enabled later.

Additional options for the profiler​

jemalloc has many different options available, which are related to the profiler. They can be controlled by modifying the MALLOC_CONF environment variable. For example, the interval between allocation samples can be controlled with lg_prof_sample.
If you want to dump the heap profile every N bytes you can enable it using lg_prof_interval.

It is recommended to check jemallocs reference page for a complete list of options.

Other resources​

ClickHouse/Keeper expose jemalloc related metrics in many different ways.

Warning

It's important to be aware that none of these metrics are synchronized with each other and values may drift.

System table asynchronous_metrics​

Reference

System table jemalloc_bins​

Contains information about memory allocations done via the jemalloc allocator in different size classes (bins) aggregated from all arenas.

Reference

Prometheus​

All jemalloc related metrics from asynchronous_metrics are also exposed using the Prometheus endpoint in both ClickHouse and Keeper.

Reference

jmst 4LW command in Keeper​

Keeper supports the jmst 4LW command which returns basic allocator statistics: