Sentry Clickhouse replication

[begemotik]

General question, any help would be much appreciated.

We've got Clickhouse deployed as part of Sentry chart with default values, nothing changed there. As I understand from the debug it spins up 3 clickhouse nodes with replication enabled. With that.

I expect that if any of the clickhouse replicas is not available for some time - it shouldn't impact sentry itself because data distributed and replicated across nodes, if host is not available data should be handled by snuba using other replica. Though when I stop one of the nodes I observe snuba components restarts and broken UI (logs from sentry-snuba-api):

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/snuba/./snuba/web/db_query.py", line 672, in raw_query
    result = execute_query_strategy(
  File "/usr/src/snuba/./snuba/util.py", line 273, in wrapper
    return func(*args, **kwargs)
  File "/usr/src/snuba/./snuba/web/db_query.py", line 555, in execute_query_with_readthrough_caching
    return cache_partition.get_readthrough(
  File "/usr/src/snuba/./snuba/state/cache/redis/backend.py", line 187, in get_readthrough
    raise e
  File "/usr/src/snuba/./snuba/state/cache/redis/backend.py", line 167, in get_readthrough
    value = self.__executor.submit(function).result(task_timeout)
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/snuba/./snuba/util.py", line 273, in wrapper
    return func(*args, **kwargs)
  File "/usr/src/snuba/./snuba/web/db_query.py", line 355, in execute_query_with_rate_limits
    return execute_query(
  File "/usr/src/snuba/./snuba/util.py", line 273, in wrapper
    return func(*args, **kwargs)
  File "/usr/src/snuba/./snuba/web/db_query.py", line 257, in execute_query
    result = reader.execute(
  File "/usr/src/snuba/./snuba/clickhouse/native.py", line 499, in execute
    execute_func(
  File "/usr/src/snuba/./snuba/clickhouse/native.py", line 268, in execute
    raise ClickhouseError(e.message, code=e.code) from e
snuba.clickhouse.errors.ClickhouseError: DB::Exception: All connection tries failed. Log:

Code: 210, e.displayText() = DB::NetException: Host not found: sentry-clickhouse-2.sentry-clickhouse-headless.sentry.svc.cluster.local (sentry-clickhouse-2.sentry-clickhouse-headless.sentry.svc.cluster.local:9000) (version 20.8.19.4 (official build))
Code: 210, e.displayText() = DB::NetException: Host not found: sentry-clickhouse-2.sentry-clickhouse-headless.sentry.svc.cluster.local (sentry-clickhouse-2.sentry-clickhouse-headless.sentry.svc.cluster.local:9000) (version 20.8.19.4 (official build))
Code: 210, e.displayText() = DB::NetException: Host not found: sentry-clickhouse-2.sentry-clickhouse-headless.sentry.svc.cluster.local (sentry-clickhouse-2.sentry-clickhouse-headless.sentry.svc.cluster.local:9000) (version 20.8.19.4 (official build)

clickhouse:
  enabled: true
  clickhouse:
    imageVersion: "20.8.19.4"
    configmap:
      remote_servers:
        internal_replication: true

Can someone point me out how to setup Clickhouse for Sentry in HA mode?