Skip to content

Commit

Permalink
[fix](load) Fix potential data loss during disk migration (apache#42296)
Browse files Browse the repository at this point in the history
The following operations may trigger this issue.
1. migration start
2. load start using old tablet,but wait migration lock in
`RowsetBuilder::prepare_txn`.
3. migration finish, old tablet -> new tablet
4. obtained migration lock and commit successfully using old tablet.
5. publish failed using old tablet, because old tablet has been dropped.
It cause the data loss.

Therefore, after acquiring the migration lock, check if the tablet has
already been shut down. If it has, it indicates that it is an old
tablet, and data should not be imported into the old tablet.
  • Loading branch information
liaoxin01 authored Oct 24, 2024
1 parent c1c8110 commit bd8f111
Showing 1 changed file with 9 additions and 0 deletions.
9 changes: 9 additions & 0 deletions be/src/olap/txn_manager.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,15 @@ TxnManager::TxnManager(StorageEngine& engine, int32_t txn_map_shard_size, int32_
Status TxnManager::prepare_txn(TPartitionId partition_id, const Tablet& tablet,
TTransactionId transaction_id, const PUniqueId& load_id,
bool ingest) {
// check if the tablet has already been shutdown. If it has, it indicates that
// it is an old tablet, and data should not be imported into the old tablet.
// Otherwise, it may lead to data loss during migration.
if (tablet.tablet_state() == TABLET_SHUTDOWN) {
return Status::InternalError<false>(
"The tablet's state is shutdown, tablet_id: {}. The tablet may have been dropped "
"or migrationed. Please check if the table has been dropped or try again.",
tablet.tablet_id());
}
return prepare_txn(partition_id, transaction_id, tablet.tablet_id(), tablet.tablet_uid(),
load_id, ingest);
}
Expand Down

0 comments on commit bd8f111

Please sign in to comment.