Inserted rows disappear even though insert is successful

[maxmarcon]

Hi!

We have a CH cluster with 3 replicas.
Occasionally (maybe once a day), we're replacing the data in a table with new data. For that we do:

clickhouse_client.execute("ALTER TABLE target_table DELETE WHERE bh_client_id = 1492 SETTINGS mutations_sync = 2")
# at this point, because of mutations_sync = 2, the delete operation should be completed on all replicas

for records in batches:
  inserted = clickhouse_client.execute('INSERT INTO target_table VALUES', records)
  assert len(records) == inserted

Problem is, even though the INSERT command reports that all records were inserted, quite often (around 50% of the times), if we inspect target_tabe for records with bh_client_id = 1492. We don't find any!

We're really puzzled by this error, we're no CH experts and any help or hint on what might be causing it would be greatly appreciated! 🙏

insert_quorum is set to 0.

One thing we do see in the server's error logs is the following:

2023.07.20 08:01:05.703697 [ 2318552 ] {} <Warning> HTTPServerRequest: Got an HTTP request with no content length and no chunked/multipart encoding, it may be impossible to distinguish graceful EOF from abnormal connection loss

I don't know if this is an issue with the driver not sending a proper HTTP request.

Versions

• Package: clickhouse-driver[lz4]==0.2.6.
• ClickHouse server version: 23.2.4.12
• Python version: 3.9.10