Debug hanging / freezing process

Last updated:

|Edit this page

If a ClickHouse node is unresponsive and you don't know what is happening, you can easily check the stacktraces of all the working threads by running:

SQL
SELECT
arrayStringConcat(arrayMap(x -> demangle(addressToSymbol(x)), trace), '\n') AS trace_functions,
count()
FROM system.stack_trace
GROUP BY trace_functions
ORDER BY count()
DESC
SETTINGS allow_introspection_functions=1
FORMAT Vertical;

If you can't run any query but you have access to the node running the process, you can execute the following command to send a TSTP signal to every thread of the process:

Terminal
for i in $(ls -1 /proc/$(pidof clickhouse-server)/task/); do kill -TSTP $i; done

Stack traces will be printed in the ClickHouse logs.

Questions?

Was this page useful?

Next article

Horizontal scaling (sharding & replication)

Sharding is a horizontal cluster scaling strategy that puts parts of one ClickHouse database on different shards. This can help you to: Improve fault tolerance. Sharding lets you isolate individual host or replica set malfunctions. If you don't use sharding, then when one host or a set of replicas fails, the entire data they contain may become inaccessible. But if 1 shard out of 5 fails, for example, then 80% of the table data is still available. Improve the query performance. Requests…

Read next article