Skip to content

Commit

Permalink
Merge Release/v0.2.2 into develop (0xPolygonHermez#2353)
Browse files Browse the repository at this point in the history
* fix null effective_percentage

* fix forkID calculation

* fix script

* generate json-schema + docs for node config file and network_custom

* fix unittest

* Hotfixv0.1.4 to v0.2.0 (0xPolygonHermez#2255)

* Hotfix v0.1.4 to main (0xPolygonHermez#2250)

* fix concurrent web socket writes

* fix eth_syncing

* fix custom trace internal tx call error handling and update prover

* add test to custom tracer depth issue; fix internal call error and gas used

* fix custom tracer for internal tx with error and no more steps after it

* remove debug code

* Make max grpc message size configurable  (0xPolygonHermez#2179)

* make max grpc message size configurable

* fix state tests

* fix tests

* fix tests

* get SequencerNodeURI from SC if empty and not IsTrustedSequencer

* Optimize trace (0xPolygonHermez#2183)

* optimize trace

* fix memory reading

* update docker image

* update prover image

* fix converter

* fix memory

* fix step memory

* fix structlogs

* fix structlogs

* fix structlogs

* fix structlogs

* fix structlogs

* fix structlogs

* fix structlogs

* fix structlogs

* update prover image

* fix struclogs

* fix memory size

* fix memory size

* fix memory size

* refactor memory resize

* refactor memory resize

* move log for the best fitting tx (0xPolygonHermez#2192)

* fix load zkCounters from pool

* remove unnecessary log.info

* add custom tracer support to CREATES opcode without depth increase (0xPolygonHermez#2213)

* logs

* fix getting stateroot from previous batch (GetWIPBatch)

* logs

* Fix GetWipBatch when previous last batch is a forced batch

* fix forcedBatch trusted state

* Revert "fix getting stateroot from previous batch (GetWIPBatch)"

This reverts commit 860f0e7.

* force GHA

* add pool limits (0xPolygonHermez#2189)

* Hotfix/batch l2 data (0xPolygonHermez#2223)

* Fix BatchL2Data

* Force GHA

* remove failed txs from the pool limit check (0xPolygonHermez#2233)

* debug trace by batch number via external rpc requests (0xPolygonHermez#2235)

* fix trace batch remote requests in parallel limitation (0xPolygonHermez#2244)

* Added RPC.TraceBatchUseHTTPS config parameter

* fix executor version

---------

Co-authored-by: tclemos <[email protected]>
Co-authored-by: tclemos <[email protected]>
Co-authored-by: Toni Ramírez <[email protected]>
Co-authored-by: agnusmor <[email protected]>
Co-authored-by: agnusmor <[email protected]>
Co-authored-by: Thiago Coimbra Lemos <[email protected]>

* fix test

* fix test

---------

Co-authored-by: tclemos <[email protected]>
Co-authored-by: tclemos <[email protected]>
Co-authored-by: Toni Ramírez <[email protected]>
Co-authored-by: agnusmor <[email protected]>
Co-authored-by: agnusmor <[email protected]>
Co-authored-by: Thiago Coimbra Lemos <[email protected]>

* Effective GasPrice refactor+fixes (0xPolygonHermez#2247)

* effective GasPrice refactor

* bugs fixes and finalizer tests fixes

* fix typo

* fix calculate effective gasprice percentage

* fix test gas price

* Fix/0xPolygonHermez#2257 effective gas price receipt (0xPolygonHermez#2258)

* effective gas price returned by the rpc in the receipt

* linter

* bugfix: fixing l2blocks timestamp for the fist batch (0xPolygonHermez#2260)

* bugfix: fixing l2blocks timestamp for the fist batch

Signed-off-by: Nikolay Nedkov <[email protected]>

* fix finalizer unit test

---------

Signed-off-by: Nikolay Nedkov <[email protected]>

* add more comments, and removed fields PrivateKeyPath and PrivateKeyPassword from etherman.Config that are not in use

* add info to git action

* add info to git action

* fix github action

* updated comments

* updated comments

* Fix/0xPolygonHermez#2263 gas used (0xPolygonHermez#2264)

* fix fea2scalar and gas used

* suggestion

* fix fea2scalar

* suggestion

* Fix pending tx when duplicate nonce (0xPolygonHermez#2270)

* fix pending tx when duplicate nonce

* set pool.transaction.failed_reason to NULL when updating an existing tx

* add more log details when adding tx to AddrQueue

* fix query to add tx to the pool. Fix lint errors

* change failed_reason for tx discarded due duplicate nonce

* Only return a tx from the pool if tx is in pending status (0xPolygonHermez#2273)

* Return a tx from the pool only if it is

* fix TestGetTransactionByHash

---------

Co-authored-by: agnusmor <[email protected]>

* fix documentation with  config file

* improve: adding check to skip appending effectivePercentage if current forkId is under 5.

Signed-off-by: Nikolay Nedkov <[email protected]>

* Fiex effectiveGasprice unsigned txs with forkId lower than 5 (0xPolygonHermez#2278)

* feat: adding functionality to stop sequencer on specific batch num from config param.

Signed-off-by: Nikolay Nedkov <[email protected]>

* patch: adding print for X-Real-IP in JSON-RPC

Signed-off-by: Nikolay Nedkov <[email protected]>

* Fix checkIfSynced (0xPolygonHermez#2289)

* [Rehashing] Check logs order and fix blockhash and blockNumber in the log conversion (0xPolygonHermez#2280)

* fix and check order

* linter

* flushID synchronizer (0xPolygonHermez#2287)

* FlushID in synchronizer

* linter

* fix logs

* commnets

* executor error refactor (0xPolygonHermez#2299)

* handle invalid rlp ROM error (0xPolygonHermez#2297)

* add maxL2GasPrice (0xPolygonHermez#2294)

* add maxL2GasPrice

* fix

* fix

* add test

* document parameter

* update description

* Error refactor (0xPolygonHermez#2302)

* error refactor

* refactor

* Fix replaced tx as failed when duplicated nonce (0xPolygonHermez#2308)

* Fix UpdateTxStatus for replacedTx

* Fix adding tx with same nonce on AddrQueue

* log reprocess need (0xPolygonHermez#2309)

* log reprocess need

* Update finalizer.go

* Feature/2300 synchronizer detect if executor restart (0xPolygonHermez#2306)

* detect if executor restarts and stop synchonizer

* Update prover images (0xPolygonHermez#2311)

* update prover image

* update prover images

* change executor param

* Update testnet.prover.config.json

* Update test.permissionless.prover.config.json

* Update test.prover.config.json

* Update public.prover.config.json

* prover params

* prover params

* prover params

* update prover images

* add doc, and fix dockers to be able to use snap/restore feature (0xPolygonHermez#2315)

* add doc, and fix dockers to be able to use snap/restore feature

* add doc for snap/restore feature

---------

Co-authored-by: Toni Ramírez <[email protected]>

* Update docker-compose.yml

* Update docker-compose.yml

* do not add tx to the pool in case err != nil

* do not add tx into the pool if a fatal error in the executor happens during pre execution

* fix dbMultiWriteSinglePosition config value

* workarround for the error error closing batch

* workarround for the error error closing batch

* workarround for the error error closing batch

* workaround for the error of closing batch, another case

* `Worker`'s `AddTxTracker` Bug Fix (0xPolygonHermez#2343)

* bugfix: Resolve  Function Bug in Worker Module

Signed-off-by: Nikolay Nedkov <[email protected]>

* improve: improving the wait for pending txs to be for only the txs for the current address.

Signed-off-by: Nikolay Nedkov <[email protected]>

---------

Signed-off-by: Nikolay Nedkov <[email protected]>

* rename config files (0xPolygonHermez#2349)

* fix closing batch + logs (0xPolygonHermez#2348)

* fix closing batch + logs

* fix

* log description

* typo errors

* fix error: failed to store transactions for batch due to duplicate key

* test

* typo

* Update README.md

* Update release.yml

* fix conflict

---------

Signed-off-by: Nikolay Nedkov <[email protected]>
Co-authored-by: joanestebanr <[email protected]>
Co-authored-by: Alonso Rodriguez <[email protected]>
Co-authored-by: tclemos <[email protected]>
Co-authored-by: tclemos <[email protected]>
Co-authored-by: agnusmor <[email protected]>
Co-authored-by: agnusmor <[email protected]>
Co-authored-by: Thiago Coimbra Lemos <[email protected]>
Co-authored-by: joanestebanr <[email protected]>
Co-authored-by: Nikolay Nedkov <[email protected]>
  • Loading branch information
10 people authored Aug 2, 2023
1 parent 01d8bbe commit 0cbb920
Show file tree
Hide file tree
Showing 25 changed files with 252 additions and 111 deletions.
8 changes: 4 additions & 4 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,12 @@ jobs:
sed -i -e "s/image: zkevm-node/image: hermeznetwork\/zkevm-node:$GIT_TAG_NAME/g" testnet/docker-compose.yml
zip -r testnet.zip testnet
# MAINNET
mkdir -p mainnet/config/environments/testnet
mkdir -p mainnet/config/environments/mainnet
mkdir -p mainnet/db/scripts
cp config/environments/mainnet/* mainnet/config/environments/testnet
cp config/environments/mainnet/* mainnet/config/environments/mainnet
cp docker-compose.yml mainnet
cp db/scripts/init_prover_db.sql mainnet/db/scripts
mv mainnet/config/environments/testnet/example.env mainnet
mv mainnet/config/environments/mainnet/example.env mainnet
sed -i -e "s/image: zkevm-node/image: hermeznetwork\/zkevm-node:$GIT_TAG_NAME/g" mainnet/docker-compose.yml
zip -r mainnet.zip mainnet
Expand All @@ -61,4 +61,4 @@ jobs:
files: 'testnet.zip;mainnet.zip'
repo-token: ${{ secrets.TOKEN_RELEASE }}
release-tag: ${{ steps.tagName.outputs.tag }}


2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ RUN cd /src && make build
# CONTAINER FOR RUNNING BINARY
FROM alpine:3.18.0
COPY --from=build /src/dist/zkevm-node /app/zkevm-node
COPY --from=build /src/config/environments/testnet/testnet.node.config.toml /app/example.config.toml
COPY --from=build /src/config/environments/testnet/node.config.toml /app/example.config.toml
RUN apk update && apk add postgresql15-client
EXPOSE 8123
CMD ["/bin/sh", "-c", "/app/zkevm-node run"]
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,6 @@ It's recommended to use `make` for building, and testing the code, ... Run `make

## Contribute

Before opening a pull request, please read this [guide](CONTRIBUTING.md)
Before opening a pull request, please read this [guide](CONTRIBUTING.md).


4 changes: 2 additions & 2 deletions config/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -61,8 +61,8 @@ Config represents the configuration of the entire Hermez Node
The file is [TOML format]
You could find some examples:
- `config/environments/local/local.node.config.toml`: running a permisionless node
- `config/environments/mainnet/public.node.config.toml`
- `config/environments/public/public.node.config.toml`
- `config/environments/mainnet/node.config.toml`
- `config/environments/public/node.config.toml`
- `test/config/test.node.config.toml`: configuration for a trusted node used in CI
[TOML format]: https://en.wikipedia.org/wiki/TOML
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -112,5 +112,5 @@
"maxHashDBThreads": 8,
"ECRecoverPrecalc": true,
"ECRecoverPrecalcNThreads": 16,
"dbMultiWriteSinglePosition": true
"dbMultiWriteSinglePosition": false
}
Original file line number Diff line number Diff line change
Expand Up @@ -112,5 +112,5 @@
"maxHashDBThreads": 8,
"ECRecoverPrecalc": true,
"ECRecoverPrecalcNThreads": 16,
"dbMultiWriteSinglePosition": true
"dbMultiWriteSinglePosition": false
}
4 changes: 2 additions & 2 deletions config/gen_json_schema_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -25,8 +25,8 @@ The file is [TOML format](https://en.wikipedia.org/wiki/TOML#)
You could find some examples:
- `config/environments/local/local.node.config.toml`: running a permisionless node
- `config/environments/mainnet/public.node.config.toml`
- `config/environments/public/public.node.config.toml`
- `config/environments/mainnet/node.config.toml`
- `config/environments/public/node.config.toml`
- `test/config/test.node.config.toml`: configuration for a trusted node used in CI
*/
type MyTestConfig struct {
Expand Down
6 changes: 3 additions & 3 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ services:
environment:
- ZKEVM_NODE_ETHERMAN_URL=${ZKEVM_NODE_ETHERMAN_URL}
volumes:
- ${ZKEVM_ADVANCED_CONFIG_DIR:-./config/environments/testnet}/testnet.node.config.toml:/app/config.toml
- ${ZKEVM_ADVANCED_CONFIG_DIR:-./config/environments/testnet}/node.config.toml:/app/config.toml
command:
- "/bin/sh"
- "-c"
Expand All @@ -50,7 +50,7 @@ services:
environment:
- ZKEVM_NODE_ETHERMAN_URL=${ZKEVM_NODE_ETHERMAN_URL}
volumes:
- ${ZKEVM_ADVANCED_CONFIG_DIR:-./config/environments/testnet}/testnet.node.config.toml:/app/config.toml
- ${ZKEVM_ADVANCED_CONFIG_DIR:-./config/environments/testnet}/node.config.toml:/app/config.toml
command:
- "/bin/sh"
- "-c"
Expand Down Expand Up @@ -115,6 +115,6 @@ services:
- 50061:50061 # MT
- 50071:50071 # Executor
volumes:
- ${ZKEVM_ADVANCED_CONFIG_DIR:-./config/environments/testnet}/testnet.prover.config.json:/usr/src/app/config.json
- ${ZKEVM_ADVANCED_CONFIG_DIR:-./config/environments/testnet}/prover.config.json:/usr/src/app/config.json
command: >
zkProver -c /usr/src/app/config.json
8 changes: 4 additions & 4 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@ To configure a node you need 3 files:
This file is a [TOML](https://en.wikipedia.org/wiki/TOML#) formatted file.
You could find some examples here:
- `config/environments/local/local.node.config.toml`: running a permisionless node
- `config/environments/mainnet/public.node.config.toml`
- `config/environments/public/public.node.config.toml`
- `config/environments/mainnet/node.config.toml`
- `config/environments/public/node.config.toml`
- `test/config/test.node.config.toml`: configuration for a trusted node used in CI

For details about the contents you can read specifications [here](config-file/node-config-doc.md)
Expand Down Expand Up @@ -38,5 +38,5 @@ For details about the contents you can read specifications [here](config-file/cu
Please check [prover repository](https://github.com/0xPolygonHermez/zkevm-prover) for further information

Examples:
- `config/environments/mainnet/public.prover.config.json`
- `config/environments/testnet/testnet.prover.config.json`
- `config/environments/mainnet/prover.config.json`
- `config/environments/testnet/prover.config.json`
4 changes: 2 additions & 2 deletions docs/modes.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ By default the config files found in the repository will spin up the Node in JSO

This will syncronize with the Trusted Sequencer (run by Polygon).

Use the default [testnet config file](https://github.com/0xPolygonHermez/zkevm-node/blob/develop/config/environments/testnet/testnet.node.config.toml), and make sure the following values are set to:
Use the default [testnet config file](https://github.com/0xPolygonHermez/zkevm-node/blob/develop/config/environments/testnet/node.config.toml), and make sure the following values are set to:

```toml
[RPC]
Expand Down Expand Up @@ -78,7 +78,7 @@ Machine 1:

#### Machine 1

Use default [prover config](https://github.com/0xPolygonHermez/zkevm-node/blob/develop/config/environments/testnet/testnet.prover.config.json) but change the following values (`runProverServer` set to true, rest false):
Use default [prover config](https://github.com/0xPolygonHermez/zkevm-node/blob/develop/config/environments/testnet/prover.config.json) but change the following values (`runProverServer` set to true, rest false):

For *only* Prover Config (`only-prover-config.json`):

Expand Down
4 changes: 2 additions & 2 deletions docs/production-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -87,8 +87,8 @@ In the basic setup, there are Postgres being instanciated as Docker containers.
- Run dedicated instances for Postgres. To achieve this you will need to:
- Remove the Postgres services (`zkevm-pool-db` and `zkevm-state-db`) from the `docker-compose.yml`
- Instantiate Postgres elsewhere (note that you will have to create credentials and run some queries to make this work, following the config files and docker-compose should give a clear idea of what to do)
- Update the `testnet.node.config.toml` to use the correct URI for both DBs
- Update `testnet.prover.config.json` to use the correct URI for the state DB
- Update the `node.config.toml` to use the correct URI for both DBs
- Update `prover.config.json` to use the correct URI for the state DB
- Use a setup of Postgres that allows to have separated endpoints for read / write replicas

### JSON RPC
Expand Down
4 changes: 2 additions & 2 deletions event/event.go
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,8 @@ const (
EventID_FinalizerRestart EventID = "FINALIZER RESTART"
// EventID_FinalizerBreakEvenGasPriceBigDifference is triggered when the finalizer recalculates the break even gas price and detects a big difference
EventID_FinalizerBreakEvenGasPriceBigDifference EventID = "FINALIZER BREAK EVEN GAS PRICE BIG DIFFERENCE"
// EventID_SynchonizerRestart is triggered when the Synchonizer restarts
EventID_SynchonizerRestart EventID = "SYNCHRONIZER RESTART"
// EventID_SynchronizerRestart is triggered when the Synchonizer restarts
EventID_SynchronizerRestart EventID = "SYNCHRONIZER RESTART"
// Source_Node is the source of the event
Source_Node Source = "node"

Expand Down
2 changes: 2 additions & 0 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -127,10 +127,12 @@ require (
go.uber.org/atomic v1.9.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
golang.org/x/exp v0.0.0-20230206171751-46f607a40771 // indirect
golang.org/x/mod v0.9.0 // indirect
golang.org/x/sys v0.10.0 // indirect
golang.org/x/term v0.10.0 // indirect
golang.org/x/text v0.11.0 // indirect
golang.org/x/time v0.1.0 // indirect
golang.org/x/tools v0.7.0 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20230525234030-28d5490b6b19 // indirect
gopkg.in/ini.v1 v1.67.0 // indirect
gopkg.in/natefinch/npipe.v2 v2.0.0-20160621034901-c1b8fa8bdcce // indirect
Expand Down
4 changes: 4 additions & 0 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -825,6 +825,8 @@ golang.org/x/mod v0.4.1/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/mod v0.9.0 h1:KENHtAZL2y3NLMYZeHY9DW8HW8V+kQyJsY/V9JlKvCs=
golang.org/x/mod v0.9.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
Expand Down Expand Up @@ -1080,6 +1082,8 @@ golang.org/x/tools v0.1.3/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
golang.org/x/tools v0.1.7/go.mod h1:LGqMHiF4EqQNHR1JncWGqT5BVaXmza+X+BDGol+dOxo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/tools v0.7.0 h1:W4OVu8VVOaIO0yzWMNdepAulS7YfoS3Zabrm8DOXXU4=
golang.org/x/tools v0.7.0/go.mod h1:4pg6aUX35JBAogB10C9AtvVL+qowtN4pT3CGSQex14s=
golang.org/x/xerrors v0.0.0-20190410155217-1f06c39b4373/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
Expand Down
72 changes: 47 additions & 25 deletions sequencer/finalizer.go
Original file line number Diff line number Diff line change
Expand Up @@ -60,14 +60,15 @@ type finalizer struct {
maxBreakEvenGasPriceDeviationPercentage *big.Int
defaultMinGasPriceAllowed uint64
// Processed txs
pendingTransactionsToStore chan transactionToStore
pendingTransactionsToStoreWG *sync.WaitGroup
pendingTransactionsToStoreMux *sync.RWMutex
storedFlushID uint64
storedFlushIDCond *sync.Cond
proverID string
lastPendingFlushID uint64
pendingFlushIDCond *sync.Cond
pendingTxsToStore chan transactionToStore
pendingTxsToStoreWG *sync.WaitGroup
pendingTxsToStoreMux *sync.RWMutex
pendingTxsPerAddressTrackers map[common.Address]*pendingTxPerAddressTracker
storedFlushID uint64
storedFlushIDCond *sync.Cond
proverID string
lastPendingFlushID uint64
pendingFlushIDCond *sync.Cond
}

type transactionToStore struct {
Expand Down Expand Up @@ -113,6 +114,8 @@ func newFinalizer(
closingSignalCh ClosingSignalCh,
batchConstraints batchConstraints,
eventLog *event.EventLog,
pendingTxsToStoreMux *sync.RWMutex,
pendingTxsPerAddressTrackers map[common.Address]*pendingTxPerAddressTracker,
) *finalizer {
return &finalizer{
cfg: cfg,
Expand All @@ -139,9 +142,10 @@ func newFinalizer(
// event log
eventLog: eventLog,
maxBreakEvenGasPriceDeviationPercentage: new(big.Int).SetUint64(effectiveGasPriceCfg.MaxBreakEvenGasPriceDeviationPercentage),
pendingTransactionsToStore: make(chan transactionToStore, batchConstraints.MaxTxsPerBatch*pendingTxsBufferSizeMultiplier),
pendingTransactionsToStoreWG: new(sync.WaitGroup),
pendingTransactionsToStoreMux: &sync.RWMutex{},
pendingTxsToStore: make(chan transactionToStore, batchConstraints.MaxTxsPerBatch*pendingTxsBufferSizeMultiplier),
pendingTxsToStoreWG: new(sync.WaitGroup),
pendingTxsToStoreMux: pendingTxsToStoreMux,
pendingTxsPerAddressTrackers: pendingTxsPerAddressTrackers,
storedFlushID: 0,
// Mutex is unlocked when the condition is broadcasted
storedFlushIDCond: sync.NewCond(&sync.Mutex{}),
Expand Down Expand Up @@ -395,7 +399,7 @@ func (f *finalizer) checkProverIDAndUpdateStoredFlushID(storedFlushID uint64, pr
func (f *finalizer) storePendingTransactions(ctx context.Context) {
for {
select {
case tx, ok := <-f.pendingTransactionsToStore:
case tx, ok := <-f.pendingTxsToStore:
if !ok {
// Channel is closed
return
Expand All @@ -415,10 +419,19 @@ func (f *finalizer) storePendingTransactions(ctx context.Context) {

// Now f.storedFlushID >= tx.flushId, you can store tx
f.storeProcessedTx(ctx, tx)
f.pendingTransactionsToStoreWG.Done()
f.pendingTxsToStoreMux.Lock()
f.pendingTxsToStoreWG.Done()
f.pendingTxsPerAddressTrackers[tx.txTracker.From].wg.Done()
f.pendingTxsPerAddressTrackers[tx.txTracker.From].count--
// Needed to avoid memory leaks
if f.pendingTxsPerAddressTrackers[tx.txTracker.From].count == 0 {
delete(f.pendingTxsPerAddressTrackers, tx.txTracker.From)
}
f.pendingTxsToStoreMux.Unlock()

case <-ctx.Done():
// The context was cancelled from outside, Wait for all goroutines to finish, cleanup and exit
f.pendingTransactionsToStoreWG.Wait()
f.pendingTxsToStoreWG.Wait()
return
default:
time.Sleep(100 * time.Millisecond) //nolint:gomnd
Expand All @@ -433,7 +446,7 @@ func (f *finalizer) newWIPBatch(ctx context.Context) (*WipBatch, error) {

// Wait until all processed transactions are saved
startWait := time.Now()
f.pendingTransactionsToStoreWG.Wait()
f.pendingTxsToStoreWG.Wait()
endWait := time.Now()

log.Info("waiting for pending transactions to be stored took: ", endWait.Sub(startWait).String())
Expand Down Expand Up @@ -672,18 +685,18 @@ func (f *finalizer) handleProcessTransactionResponse(ctx context.Context, tx *Tx
flushId: result.FlushID,
}

f.pendingTransactionsToStoreMux.Lock()
f.pendingTransactionsToStoreWG.Add(1)
f.pendingTxsToStoreMux.Lock()
f.pendingTxsToStoreWG.Add(1)
if result.FlushID > f.lastPendingFlushID {
f.lastPendingFlushID = result.FlushID
f.pendingFlushIDCond.Broadcast()
}
f.pendingTransactionsToStoreMux.Unlock()
f.pendingTxsToStoreMux.Unlock()
select {
case f.pendingTransactionsToStore <- processedTransaction:
case f.pendingTxsToStore <- processedTransaction:
case <-ctx.Done():
// If context is cancelled before we can send to the channel, we must decrement the WaitGroup count
f.pendingTransactionsToStoreWG.Done()
f.pendingTxsToStoreWG.Done()
}

f.batch.countOfTxs++
Expand Down Expand Up @@ -721,26 +734,35 @@ func (f *finalizer) handleForcedTxsProcessResp(ctx context.Context, request stat
flushId: result.FlushID,
}

f.pendingTransactionsToStoreMux.Lock()
f.pendingTransactionsToStoreWG.Add(1)
f.pendingTxsToStoreMux.Lock()
f.pendingTxsToStoreWG.Add(1)
if result.FlushID > f.lastPendingFlushID {
f.lastPendingFlushID = result.FlushID
f.pendingFlushIDCond.Broadcast()
}
f.pendingTransactionsToStoreMux.Unlock()
f.pendingTxsToStoreMux.Unlock()
oldStateRoot = txResp.StateRoot

select {
case f.pendingTransactionsToStore <- processedTransaction:
case f.pendingTxsToStore <- processedTransaction:
case <-ctx.Done():
// If context is cancelled before we can send to the channel, we must decrement the WaitGroup count
f.pendingTransactionsToStoreWG.Done()
f.pendingTxsToStoreWG.Done()
}
}
}

// storeProcessedTx stores the processed transaction in the database.
func (f *finalizer) storeProcessedTx(ctx context.Context, txToStore transactionToStore) {
f.pendingTxsToStoreMux.Lock()
if _, ok := f.pendingTxsPerAddressTrackers[txToStore.txTracker.From]; !ok {
f.pendingTxsPerAddressTrackers[txToStore.txTracker.From] = new(pendingTxPerAddressTracker)
f.pendingTxsPerAddressTrackers[txToStore.txTracker.From].wg = &sync.WaitGroup{}
}
f.pendingTxsPerAddressTrackers[txToStore.txTracker.From].wg.Add(1)
f.pendingTxsPerAddressTrackers[txToStore.txTracker.From].count++
f.pendingTxsToStoreMux.Unlock()

if txToStore.response != nil {
log.Infof("storeProcessedTx: storing processed txToStore: %s", txToStore.response.TxHash.String())
} else {
Expand Down
Loading

0 comments on commit 0cbb920

Please sign in to comment.