Skip to content

Commit

Permalink
feat: introduce GetLocal() and BatchGet cache for PipelinedMemDB (#1212)
Browse files Browse the repository at this point in the history 
* feat: introduce GetLocal() for MemBuffer

Signed-off-by: ekexium <[email protected]>

* downgrade tools to 0.18.0

Signed-off-by: ekexium <[email protected]>

* fix: set flushingMemDB = nil when an error is returned from a flush

Signed-off-by: ekexium <[email protected]>

* impl BatchGet for MemBuffer

Signed-off-by: you06 <[email protected]>

test membuffer batch get

Signed-off-by: you06 <[email protected]>

* add Prefetch & GetPrefetchCache

Signed-off-by: you06 <[email protected]>

* cache multi Prefetch call

Signed-off-by: you06 <[email protected]>

* add tests

Signed-off-by: you06 <[email protected]>

* workaround for golang ci lint failure

Signed-off-by: you06 <[email protected]>

* replace assert with require

Signed-off-by: you06 <[email protected]>

* workaround golangci lint

Signed-off-by: you06 <[email protected]>

* lint

Signed-off-by: you06 <[email protected]>

* remove prefetch interface, pipelined memdb will cache batch get result

Signed-off-by: you06 <[email protected]>

* update tidb

Signed-off-by: you06 <[email protected]>

* fix batch get cache when membuffer is empty

Signed-off-by: you06 <[email protected]>

* fix returned delete value

Signed-off-by: you06 <[email protected]>

* fix: handle resourceGroupTag of Flush

Signed-off-by: ekexium <[email protected]>

* fix: set resource group tag for committer if it's pipelined

Signed-off-by: ekexium <[email protected]>

* Update internal/unionstore/pipelined_memdb.go

Co-authored-by: ekexium <[email protected]>

* remove prefetch interface

Signed-off-by: you06 <[email protected]>

* release mutex

Signed-off-by: you06 <[email protected]>

* flush wait to avoid race

Signed-off-by: you06 <[email protected]>

* fix unstopped test

Signed-off-by: you06 <[email protected]>

* set resource group tags for committer

Signed-off-by: ekexium <[email protected]>

* skip test due to tikv image not updated yet

Signed-off-by: you06 <[email protected]>

* skip more test

Signed-off-by: you06 <[email protected]>

---------

Signed-off-by: ekexium <[email protected]>
Signed-off-by: you06 <[email protected]>
Co-authored-by: you06 <[email protected]>
  • Loading branch information
@ekexium @you06
ekexium and you06 authored Mar 16, 2024
1 parent 0606e74 commit 87a984a
Show file tree
Hide file tree
Showing 21 changed files with 496 additions and 155 deletions.
4 changes: 2 additions & 2 deletions examples/gcworker/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240228224816-df926f6c8641 // indirect
google.golang.org/grpc v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
)
Expand Down
4 changes: 2 additions & 2 deletions examples/rawkv/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240228224816-df926f6c8641 // indirect
google.golang.org/grpc v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
)
Expand Down
4 changes: 2 additions & 2 deletions examples/txnkv/1pc_txn/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240228224816-df926f6c8641 // indirect
google.golang.org/grpc v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
)
Expand Down
4 changes: 2 additions & 2 deletions examples/txnkv/async_commit/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240228224816-df926f6c8641 // indirect
google.golang.org/grpc v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
)
Expand Down
4 changes: 2 additions & 2 deletions examples/txnkv/delete_range/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240228224816-df926f6c8641 // indirect
google.golang.org/grpc v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
)
Expand Down
4 changes: 2 additions & 2 deletions examples/txnkv/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240228224816-df926f6c8641 // indirect
google.golang.org/grpc v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
)
Expand Down
4 changes: 2 additions & 2 deletions examples/txnkv/pessimistic_txn/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240228224816-df926f6c8641 // indirect
google.golang.org/grpc v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
)
Expand Down
4 changes: 2 additions & 2 deletions examples/txnkv/unsafedestoryrange/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,8 +45,8 @@ require (
golang.org/x/text v0.14.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240228224816-df926f6c8641 // indirect
google.golang.org/grpc v1.62.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
google.golang.org/protobuf v1.33.0 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
)
Expand Down
27 changes: 13 additions & 14 deletions integration_tests/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,15 @@ go 1.21

require (
github.com/ninedraft/israce v0.0.3
github.com/pingcap/errors v0.11.5-0.20231212100244-799fae176cfb
github.com/pingcap/errors v0.11.5-0.20240311081613-f97970b88865
github.com/pingcap/failpoint v0.0.0-20220801062533-2eaa32854a6c
github.com/pingcap/kvproto v0.0.0-20240227073058-929ab83f9754
github.com/pingcap/tidb v1.1.0-beta.0.20240126041650-de177d85b19e
github.com/pingcap/tidb v1.1.0-beta.0.20240312154839-cd6dbc2af910
github.com/pkg/errors v0.9.1
github.com/stretchr/testify v1.8.4
github.com/stretchr/testify v1.9.0
github.com/tidwall/gjson v1.14.1
github.com/tikv/client-go/v2 v2.0.8-0.20240205071126-11cb7985f0ec
github.com/tikv/pd/client v0.0.0-20240229062430-b207e514ece1
github.com/tikv/client-go/v2 v2.0.8-0.20240313022320-d59fea5757db
github.com/tikv/pd/client v0.0.0-20240229065730-92a31c12238e
go.uber.org/goleak v1.3.0
)

Expand Down Expand Up @@ -69,18 +69,18 @@ require (
github.com/pingcap/goleveldb v0.0.0-20191226122134-f82aafb29989 // indirect
github.com/pingcap/log v1.1.1-0.20230317032135-a0d097d16e22 // indirect
github.com/pingcap/sysutil v1.0.1-0.20230407040306-fb007c5aff21 // indirect
github.com/pingcap/tidb/pkg/parser v0.0.0-20240111112854-1ad36eb0ef29 // indirect
github.com/pingcap/tidb/pkg/parser v0.0.0-20240312154839-cd6dbc2af910 // indirect
github.com/pingcap/tipb v0.0.0-20240116032918-9bb28c43bbfc // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20221212215047-62379fc7944b // indirect
github.com/prometheus/client_golang v1.18.0 // indirect
github.com/prometheus/client_model v0.5.0 // indirect
github.com/prometheus/common v0.46.0 // indirect
github.com/prometheus/client_golang v1.19.0 // indirect
github.com/prometheus/client_model v0.6.0 // indirect
github.com/prometheus/common v0.48.0 // indirect
github.com/prometheus/procfs v0.12.0 // indirect
github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
github.com/rogpeppe/go-internal v1.11.0 // indirect
github.com/sasha-s/go-deadlock v0.3.1 // indirect
github.com/shirou/gopsutil/v3 v3.23.10 // indirect
github.com/shirou/gopsutil/v3 v3.24.1 // indirect
github.com/shoenig/go-m1cpu v0.1.6 // indirect
github.com/spf13/pflag v1.0.5 // indirect
github.com/tiancaiamao/gp v0.0.0-20221230034425-4025bc8a4d4a // indirect
Expand All @@ -97,7 +97,7 @@ require (
go.etcd.io/etcd/client/v3 v3.5.12 // indirect
go.uber.org/atomic v1.11.0 // indirect
go.uber.org/multierr v1.11.0 // indirect
go.uber.org/zap v1.26.0 // indirect
go.uber.org/zap v1.27.0 // indirect
golang.org/x/crypto v0.21.0 // indirect
golang.org/x/exp v0.0.0-20240205201215-2c58cdc269a3 // indirect
golang.org/x/net v0.22.0 // indirect
Expand All @@ -106,7 +106,6 @@ require (
golang.org/x/text v0.14.0 // indirect
golang.org/x/time v0.5.0 // indirect
golang.org/x/tools v0.19.0 // indirect
google.golang.org/genproto v0.0.0-20240213162025-012b6fc9bca9 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20240304212257-790db918fca8 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20240308144416-29370a3891b7 // indirect
google.golang.org/grpc v1.62.1 // indirect
Expand All @@ -118,7 +117,7 @@ require (

replace (
github.com/go-ldap/ldap/v3 => github.com/YangKeao/ldap/v3 v3.4.5-0.20230421065457-369a3bab1117
github.com/pingcap/tidb => github.com/you06/tidb v1.1.0-beta.0.20240220121437-9d6b908d9a92
github.com/pingcap/tidb/pkg/parser => github.com/you06/tidb/pkg/parser v0.0.0-20240220121437-9d6b908d9a92
github.com/pingcap/tidb => github.com/ekexium/tidb v1.1.0-beta.0.20240314050256-444418ecf47b
github.com/pingcap/tidb/pkg/parser => github.com/ekexium/tidb/pkg/parser v0.0.0-20240314050256-444418ecf47b
github.com/tikv/client-go/v2 => ../
)
Loading

0 comments on commit 87a984a

Please sign in to comment.