Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(snapshot): provide snapshot structure acceleration #324

Open
wants to merge 89 commits into
base: develop
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
89 commits
Select commit Hold shift + click to select a range
9485284
refactor: extract the zktrie logic into a new package, and prepare th…
mortal123 Apr 21, 2023
87d01e6
fix: reset the inner object for zktrie.Trie and zktrie.Secure
mortal123 Apr 21, 2023
8f18c57
Added tests for secure_trie_test and trie_test, skipping iterator and…
KevinYum Apr 23, 2023
c107000
adapt tests to zk-trie hash and encode contraints, remove benchmarks
KevinYum Apr 23, 2023
cbda0e0
feat: prove and verify for trie and secure trie
mortal123 Apr 23, 2023
ff4b17b
feat: struct BinaryPath to facilitate path representation
mortal123 Apr 23, 2023
406cf81
fix: proofs in statedb
mortal123 Apr 24, 2023
a6c009a
fix: set impl in trie
mortal123 Apr 24, 2023
4695cfc
fix: add zk trie magic bytes back
mortal123 Apr 25, 2023
fecd2f7
chore: secure trie add sanity checks for storage and account update
mortal123 Apr 25, 2023
9030b56
feat: add stack trie implementation
mortal123 Apr 25, 2023
85d7085
add test cases of proof_test for Trie
KevinYum Apr 26, 2023
ceeff53
fix: separate the use case of trie.Update and trie.UpdateAccount
mortal123 Apr 26, 2023
d9620b9
add test cases of proof_test for SecureTrie
KevinYum Apr 26, 2023
d77935d
fix: assign flag for leaf node in stack trie
mortal123 Apr 26, 2023
31ad213
chore: remove comments
mortal123 Apr 27, 2023
8c6e7dd
fix: fix bit order when update trie
mortal123 Apr 27, 2023
cf1d46b
chore: add key check in stacktrie
mortal123 Apr 27, 2023
095a0c0
add test cases of stacktrie_test
KevinYum Apr 27, 2023
dc98589
chore: replace zktrie with trie
mortal123 Apr 28, 2023
9df7803
chore: add binary path
mortal123 Apr 28, 2023
3c80d49
feat: trie iterator
mortal123 Apr 28, 2023
0cdb57f
feat: add proof range verify
mortal123 Apr 30, 2023
b16a518
fix test discrepancy introduced by commit e5af6e6
KevinYum May 2, 2023
c5df8ef
fix: remove debug log
mortal123 May 2, 2023
b857a85
feat: add iterator for secure trie
mortal123 May 2, 2023
2b5c2ac
chore: add key length checker in trie and stack_trie
mortal123 May 2, 2023
bec5e8e
fix: assign root hash for node iterator
mortal123 May 2, 2023
092de1e
fix: secure trie iterator bug
mortal123 May 2, 2023
065e203
Enforce trie_test and stacktrie_test cases to have key length of 32 b…
KevinYum May 2, 2023
1f07ad1
fix: make secure trie get key transform valid
mortal123 May 3, 2023
ef113e0
add test cases for iterator, align usage of key->secureKey conversion
KevinYum May 3, 2023
17e9f23
fix: save the unset internal node into cache
mortal123 May 4, 2023
826c66a
fix: range proof verification
mortal123 May 4, 2023
0eedd3a
fix: add trie update with kind method
mortal123 May 4, 2023
f217e05
chore: delete the snapshot disable
mortal123 May 4, 2023
43db248
fix: correcting account marshalling
mortal123 May 4, 2023
df91a1e
fix: check keys range in range proof verify
mortal123 May 4, 2023
945789f
add test cases for rangeVerify
KevinYum May 4, 2023
4c6e214
chore: fix minor issues and move poseidon hash into crypto packages
mortal123 May 5, 2023
48d8eaf
add basic benchmarks for geth and snap feature for comparison
KevinYum May 6, 2023
4de19cc
add snap sync
mortal123 May 8, 2023
895ac7f
fix iterator value decode bug
mortal123 May 8, 2023
fd8382d
enable cleans cache for trie database
mortal123 May 8, 2023
1816186
adopt proof tracer with secure trie
mortal123 May 9, 2023
6110df9
fix empty root hash
mortal123 May 9, 2023
c1339d8
allow snapshot diff layer do the accumulation when snapshot is genera…
mortal123 May 11, 2023
0e21b17
fix file import
mortal123 May 11, 2023
c67e3e4
make tests compilable
mortal123 May 11, 2023
b5850c2
dump log if zktrie is disabled
mortal123 May 11, 2023
af4ffc7
fix bugs for testing
mortal123 May 11, 2023
d596e08
add range proof test for account trie
KevinYum May 12, 2023
53e58c1
fix the checking method for trie node
mortal123 May 12, 2023
d7fb7c2
chore: go fmt with imports
mortal123 May 12, 2023
0120513
add node hash method
mortal123 May 12, 2023
a34513d
bug fix
mortal123 May 12, 2023
d61cef6
bug fix for zktrie proof
mortal123 May 12, 2023
ace8416
add test cases to sync(healer)
KevinYum May 12, 2023
0290fab
provide toString function for trie
mortal123 May 13, 2023
c5f6453
fix bug for hashing node proof key
mortal123 May 13, 2023
4ffc934
fix test code generate_test.go and genesis_test.go, handle correspond…
mortal123 May 13, 2023
6ca5353
fix testcases in statedb_test.go
mortal123 May 13, 2023
5bb7436
change compact encoding
mortal123 May 14, 2023
ee908e3
fix bugs and testcase (state sync)
mortal123 May 14, 2023
811e93a
disable DESTRUCT opcode testcase
mortal123 May 14, 2023
b3dce78
correcting the judgement of empty zktrie
mortal123 May 14, 2023
50640b3
add test cases for snap/sync_test (not passed yet)
KevinYum May 14, 2023
1a7417d
fix bugs for testcase and stacktrie
mortal123 May 14, 2023
7ada128
shrink test size for snap/sync_test to adapt poor trie performance
KevinYum May 15, 2023
239c07a
fix bugs for testcase
mortal123 May 15, 2023
77c4db5
fix tracer testcase
mortal123 May 15, 2023
fc90758
fix api testcase
mortal123 May 15, 2023
fee15c7
fix testcases in downloader and enable fast sync
mortal123 May 15, 2023
9dc5d2d
change the tree cap strategy
mortal123 May 16, 2023
71c9917
skip testcase related to zktrie database
mortal123 May 16, 2023
29b2943
Merge branch 'develop' of https://github.com/scroll-tech/go-ethereum …
mortal123 May 16, 2023
838aaaa
chore for pr check
mortal123 May 16, 2023
0fc49c5
makefile tests include zktrie
mortal123 May 16, 2023
868258f
fix bugs related account decode
mortal123 May 17, 2023
39b9d99
fix bugs of sync depth
mortal123 May 20, 2023
35cb810
add comment
mortal123 May 21, 2023
5deab05
enable zktrie snap sync
mortal123 May 23, 2023
ed09860
add benchmark for stacktrie
mortal123 May 23, 2023
987d7ad
eliminate inappropriate fmt printf
mortal123 May 23, 2023
1009e84
fix comment style in core/blockchain_test.go
mortal123 May 23, 2023
0059a2a
fix comment style
mortal123 May 23, 2023
ec5c3a9
merge from develop
mortal123 May 23, 2023
339e03f
fix account encoding in cmd snapshot
mortal123 May 24, 2023
8c6618f
Merge tag 'scroll-v3.3.1' of https://github.com/scroll-tech/go-ethere…
mortal123 May 29, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
chore: secure trie add sanity checks for storage and account update
  • Loading branch information
mortal123 committed May 9, 2023
commit fecd2f7fc75e761c25e8d2156f952ce9dd181a0a
32 changes: 15 additions & 17 deletions zktrie/secure_trie.go
Original file line number Diff line number Diff line change
Expand Up @@ -29,21 +29,21 @@ import (

var magicHash []byte = []byte("THIS IS THE MAGIC INDEX FOR ZKTRIE")

// wrap itrie for trie interface
// SecureTrie is a wrapper of Trie which make the key secure
type SecureTrie struct {
trie *itrie.ZkTrie
db *Database
}

func sanityCheckByte32Key(b []byte) {
if len(b) != 32 && len(b) != 20 {
panic(fmt.Errorf("do not support length except for 120bit and 256bit now. data: %v len: %v", b, len(b)))
func sanityCheckKeyBytes(b []byte, accountAddress bool, storageKey bool) {
if (accountAddress && len(b) == 20) || (storageKey && len(b) == 32) {
} else {
panic(fmt.Errorf(
"bytes length is not supported, accountAddress: %v, storageKey: %v, length: %v",
accountAddress, storageKey, len(b)))
}
}

// New creates a trie
// New bypasses all the buffer mechanism in *Database, it directly uses the
// underlying diskdb
func NewSecure(root common.Hash, db *Database) (*SecureTrie, error) {
if db == nil {
panic("zktrie.NewSecure called without a database")
Expand All @@ -66,19 +66,18 @@ func (t *SecureTrie) Get(key []byte) []byte {
}

func (t *SecureTrie) TryGet(key []byte) ([]byte, error) {
sanityCheckByte32Key(key)
sanityCheckKeyBytes(key, true, true)
return t.trie.TryGet(key)
}

func (t *SecureTrie) TryGetNode(path []byte) ([]byte, int, error) {
panic("implement me!")
}

// TryUpdateAccount will abstract the write of an account to the
// secure trie.
func (t *SecureTrie) TryUpdateAccount(key []byte, acc *types.StateAccount) error {
sanityCheckByte32Key(key)
value, flag := acc.MarshalFields()
// TryUpdateAccount will update the account value in trie
func (t *SecureTrie) TryUpdateAccount(key []byte, account *types.StateAccount) error {
sanityCheckKeyBytes(key, true, false)
value, flag := account.MarshalFields()
return t.trie.TryUpdate(key, flag, value)
}

Expand All @@ -94,10 +93,9 @@ func (t *SecureTrie) Update(key, value []byte) {
}
}

// NOTE: value is restricted to length of bytes32.
// we override the underlying itrie's TryUpdate method
// TryUpdate will update the storage value in trie. value is restricted to length of bytes32.
func (t *SecureTrie) TryUpdate(key, value []byte) error {
sanityCheckByte32Key(key)
sanityCheckKeyBytes(key, false, true)
return t.trie.TryUpdate(key, 1, []itypes.Byte32{*itypes.NewByte32FromBytes(value)})
}

Expand All @@ -109,7 +107,7 @@ func (t *SecureTrie) Delete(key []byte) {
}

func (t *SecureTrie) TryDelete(key []byte) error {
sanityCheckByte32Key(key)
sanityCheckKeyBytes(key, true, true)
return t.trie.TryDelete(key)
}

Expand Down