All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog and this project adheres to Semantic Versioning.
- VWAP for Binance, ByBit
- KuCoin fetcher with vwap
- Update GitHub Action. The deployments are managed by helm charts
- prepare schema for vwap
- remove calculators
- fallback to default RPC provider gas estimation based on env setup
- set proper warning and error level for
5ire
balances
- remove unused aggregation code
- Update CI due to helm chart migration
- add support for 5ire blockchain
- fix CI
- add support for 5ire blockchain
- fix CI
- fix fetching list of validators on concordium
- additional debug logs for validators selection
- throw full error on massa estimation
- fix gas management for massa
- remove
maxFeePerGas
for massa
- fix log for
searchForConsensus
- do not throw error when verifying proposed data, when feed not exists
- check required signatures per chain
- fix
validatorExistInBank
- fix massa integration
- set expire to gas model
- use
Math.trunc
for better precision forMONGODB_PURGE_DAYS
- fix support floating numbers for
MONGODB_PURGE_DAYS
- support floating numbers for
MONGODB_PURGE_DAYS
eg 1.5 = 1 day + 12h
- clean up after adding TTL for price collections
- do not throw error on dropping collection when collection does not exist
- added back migration for dropping price collections. Collections should be dropped manually before updating to this version.
- removed migration for dropping price collection because it was too slow.
- add logs for migration to
8.5.*
- run migrations only on scheduler
- turn off purge, drop price collections and use TTL for removing data
- option to set purge limit by env
- added errors to debug issue with
Cannot read properties of undefined (reading 'precision')
- purge price collections
- fix cache for
MoCMeasurement
- fix usage of
amountInDecimals
- add
Money on chain /measurement
fetcher - add BOB blockchain support
- remove
console.log
- fix logic in leader selector V2
- change log level for yaml parser
- change path for release configuration
- assume release is active by default, eg. when flag not present
- Fix github actions
- option to turn on release on whole network at once
- by default check validators list every 60s
- leader selection changes:
- use validator urls (instead of evm addresses) to create ordered list of validators
- use only validators that are present on all blockchains
- on consensus dispatching do not check if you leader, but deprecate it after some time
- disable
PolygonIOStockSnapshotPrice
fetcher
- new
PolygonIOStockSnapshotPriceFetcher
- split fetchers into fetcher and getter
- removed stock fetcher service
- remove unused fetcher name:
PolygonIOPrice
- fix version number
- add check for leaves and proposed data
- return gas price in
Gwei
- use
Gwei
denominator forEvmTWAPGasPriceFetcher
prices
- fix TWAP price precision
- allow to use custom precision for on-chain data feeds
- ensure errors of fetching price does not prevent fetching prices from DB
- set correct
valueType
for gas model - set correct fetcher name for
PolygonIOCurrencySnapshotGrams
in schema
- fix
id
for gas model
- set time window for prices to be -60 +10
- store all possible fetched data into separate collections
- separate
PolygonIOCryptoSnapshotPriceFetcher
- separate
PolygonIOSingleCryptoPriceFetcher
- read feeds from DB instead of from API directly (part 1)
- remove deprecated uniswap code
- remove polygon crypto service (2 separate fetchers extracted)
- ensure we filter out extreme gas prices in
GasMonitor
- fix PolygonIOCurrencySnapshotGramsFetcher and MetalPriceApiFetcher fetcher
- add more logs for polygon IO fetchers
- fix invalid error about
[CryptoComparePriceMultiFetcher] empty fsyms
- backward compatible patch for
FIXED_RAND
base/quote
- backward compatible patch for
PolygonGas_TWAP10
base/quote
- use lower case for tokens in uniswapV3 repository
- added better logs for caching validators
- new
ByBitSpotFetcher
- new
BinanceFetcher
- fix conditions for feeds in multi fetchers
- fix conversion to number in Uniswap V3 fetcher and fix support for unsuccessful prices
- use proper name for
Sovryn
fetcher
- fix
SovrynMultiProcessor
to return fixed array of results that matchesfeedFetchers
input.
- add additional logs for debugging fetchers
- add additional logs for debugging fetchers
- add additional logs for debugging fetchers
- add additional logs for debugging fetchers
- Support for
UniswapV3
dex fetcher - Migration 7.28.0 adding liquidities and tokens for rootstock
- disable Sovryn Pool Scanner
- new
uniswapV3LiquidityWorker
- new
SovrynPriceFetcher
- new
UniswapV3MultiFetcher
- Added all fetcher in FetcherName
- display
verifyKey
in info
- filter out signatures that are unknown to staking bank
- filter out signatures that are unknown to staking bank
- display
verifyKey
in info
- filter out signatures that are unknown to staking bank
- fix schema for
PolygonIOCurrencySnapshotGramsFetcher
- Use SSL by default for concordium provider plus option to use insecure
- Update Makefile and cicd actions due to Non Prod migration to Frankfurt
- revert
throwError
option
- do throw error on empty yaml config
- use unchecked versions of concordium modules
- added back fixed support for Massa
- update concordium auto-generated files
- add
PolygonIOCurrencySnapshotFetcher
to schema - fix
MetalsDevApi
name in schema
- make
PolygonIOCurrencySnapshotFetcher
to return price in grams instead of troy oz
- support for zk-Link Nova
- price history index
- price history
- new
PolygonIOCurrencySnapshotFetcher
- new
GoldApiPriceFetcher
- new
MetalPriceApiFetcher
- new
MetalsDevApiPriceFetcher
- remove
KaikoSpotPriceFetcher
- on invalid fetcher, display error but do not halt the worker
- fix
CryptoCompareMultiProcessor
: support case where id of feed might be in different case than feed name
- MultiversX: hash key for
getManyPriceDataRaw
- remove last
/
from RPC
- display network id on info
- display all on-chain triggers in API response
- global flag for uniswap status
- small code fixes for repositories
- better logs
- trunk gas after multiplication
- logs cleanup
- remove max gas setting
- revert RPC estimation to be bale to use higher max gas
- dump consensus data on error
- option to override
maxFeePerGas
andmaxPriorityFeePerGas
- removed
massa
code
- change app type to
ES Module
- update typescript to v5
- check for empty string
newrelic.js
filename
from settings
- print
id
inBlockchainProviderFactory
- support Massa for on-chain data
- add support for
\\n
for MultiversX PK
- disable uniswap agent if there are no settings
- removed deprecated master chain settings
- multiversx fixes
- order signtures
- constant network ID
- user proper feed contract
- cast price to bigint
- wait for tx
- tx cost
- TWAP gas price fetcher
- new relic
- make
/info
default "ping" endpoint
- non evm support for on-chain solution
- general
BLOCKCHAIN_PROVIDER_URL
andBLOCKCHAIN_PROVIDER_URLS
env, used for BSC, TODO: cleanup tech-ops - solana related code
- support for base blockchain for on-chain solution
- fix version number
- option to set multiplier for gas
- explicit use current nonce when sending tx
- feeds contract abi
- fix gas estimation for arbitrum dispatcher
- remove displaying error message for BNB in
/info
endpoint
- pull list of validators from any blockchain to check the leader
- order validators addresses alphabetically
- do not run
MetricsWorker
ifNEW_RELIC_LABELS
are not set
- ensure memory data are cloned properly
- pull validators list from target blockchain
- do not overshoot heartbeat
- cache required number of signatures
- add PolygonDeviationDispatcher
- on-chain leader decentralisation
- do not save intervals if you not a leader (each leader will check feed at least once)
- do not run minter/leader if we can not dispatch
- slow down fetching uniswap prices when no prices
- On-Chain deviation feeds
OnChainDataFetcher
: add support for returning structs
- do not include values that are zeros
- purge blocks older than 6 months
- update umbrella SDK
- update ethers to v5.7.2
- use mapping (instead of file) to monitor last tx
- force "@noble/hashes"
~v1.2.0
because1.3.0
causesError: Cannot find module 'node:crypto'
- set manually gasLimit for arbitrum
- use
StaticJsonRpcProvider
to reduce RPC calls - adjust gas settings for arbitrum
- fix TS2339: Property 'timestamp' does not exist on type 'TransformableInfo'
- cache masterchain data required to run consensus
- set default timeout to 5s for
chain.getStatus()
- remove deprecated leader selection
- Fixed GitHub Action "CI"
- ignore keys longer than 32 bytes
- fixed errors with
express-serve-static-core
package causing:
@types/express-serve-static-core/index.d.ts(1197,33): error TS1005: ';' expected.
@types/express/node_modules/@types/express-serve-static-core/index.d.ts(1265,1): error TS1160: Unterminated template literal.
- better logs messages
- Remove
type
for avaxcalculatePayableOverrides
- Use min padding as consensus round length
- display registry and rpc over multichain settings in
/info
endpoint - display uniswap contract addresses in
/info
endpoint
- Start chain dispatcher based on settings
- Dispatch avalanche blocks
- Use connections options to connect on Redis
- Updated docker compose to include block dispatcher
- Upgrade BullMQ to ~2.3.2
- Ensure dispatcher does not submit same consensus twice
- Ensure leader selection is backward compatible
- Use old isLeader selection when chain architecture is not for dispatcher worker.
- Skip CoingeckoPriceMultiFetcher tests.
- Added new pub key on action:
CI
.
- Created service to fetch state from a list of chainIds
LeaderSelector
service- Created ConsensusData model
- Created api price freshness settings
- Use LeaderSelector to get leader when sign and mint a block
- Run workers in concurrency mode
- Uniswap agent only gets successful prices
- Publish different pegasus versions to dockerhub using tags
- Bump ethers to 5.5.1
- Bump @umb-network/toolbox to 5.10.0
- Changed BlockMinter to use Gas Estimator from SDK
- Deleted GasEstimator
- Coingecko multi price fetcher
- Multi-feeds processors sub-services for FeedProcessor
- Coingecko single price fetcher
- Enabled FeedProcessor integration test
- Updated package.json version
- Optional MongoDB Price Storage & Aggregation engine
- Update
name
env to useNEW_RELIC_APP_NAME
- Updated the way logs are shown in Uniswap price Scanner
- Enabled BlockSigner unit test
- Enabled FeedProcessor unit test
- Optimized loadFeeds function to use etag and caching
- Better signature collection & discrepancy handling
- Update Git workflows when merging to main
- Added
RPC_SELECTION_STRATEGY
in env
- Updated options prices feeds to include new fields
- Upgrade SDK to 5.7.0
- Changed Blockchain class to use RPC selection from SDK
- Check if balance is enough before execute transaction
- Re-enable RPC Selection before minting/signing
- Removed RPCSelector class
- Divide the Polygon snapshot list into batches
- optimize
/info
endpoint calls
- Check if gas is enough before execute transaction
- unique codes for timeouts
- make sure we do not call ourselves for signature
- fix sorting signatures (no case sensitivity)
- Minimum timeout for StatusCheck 5000 -> 10000 ms
- Ensures that the UniswapPoolScanner agent can start without OTA enabled
- Uniswap Verified Pool optimized query index
- Uniswap OTA updates
- Uniswap MongoDB price management
- Swagger documentation on /docs.
- Added RPC Selection before minting
- Reapply Uniswap
- fetching prices from UniswapV3
- dump validators response in debug mode
- /debug/feeds endpoint
- uniswapV3 integration
- Increased JSON payload size
- Added RandomNumberFetcher
- Reverted RandomNumberFetcher
- Added RandomNumberFetcher
- CORS Support.
- Consensus Round countdown indexed to 1.
- Settings file URL mapper for production environment
- Additional consensus logging.
- Consensus retry logic.
- Increased consensus discrepancy threshold.
- Ability to handle Internal validators separatly.
- Decouple the docker hub image update from the automatic deployment
- Consensus Optimization feature flag.
- Multiple feeds file handler
- Price Aggregator auto-pruning on insert to prevent current regular OOM issues.
- Added Yearn vaults support
- Add Consensus Optimization to reduce the number of discrepant dropped keys;
- Change default feeds URLs to BSC feeds files
- Refactored Options to use Calculator
- Add a
retryStrategy
to the Redis connection, to attempt a re-connection up to a 10-second interval; - Add a
maxRetryTime
for Redis re-connection interval. Default 10 seconds (10000 ms); - Add options price API key in the /info endpoint;
- Refactored FeedProcessor leaf building to Service Object
- Add "OP:" to options leaves names
- Support new multichain feature on Makefile and github actions
- fixed a case where the OptionsPrice Fetcher interrupts minting process when fails a request
- Added a new fetcher for the OptionPrices
- Added a default value for discrepancy in consensus
- add
KaikoPriceStream
stream fetcher
- now it just logs and does not break feed processing if does not find fetcher
- Ignore validators with high discrepancies
- turn off power check
- exclude validator from consensus if check for its status fails
- change gas calculation
- unify way for executing "normal" and canceling transactions
- update
ethers
- add additional logs for
BlockMintingWorker
- add handling for
nonce has already been used
error
- add additional logs for tracking number of leaves and FCDs
- change enqueueing rules
- update ethers to version that supports EIP-1559 London
- use SDK v4.0.1
- as temporary fix change
jobId
- add InfluxDB to docker-compose core
- support for
Chain
with 2 signatures requirement - update toolbox
- adds JobId to ensure jobs are unique (limits task queue growing indefinitely)
- removed
ValidatorRegistry
settings
- better gas estimation
- detailed logs about gas
- add gas price to logs
- clear logs after debugging
- more logs for minting tx process for debugging purpose
- more logs for minting tx process
- DB Migration service
- add version to API block endpoint
- store all signed blocks and let explorer choose valid one
- return list of all blocks for
blockId
call - wait for new block to be minted before set timeout for submitted tx
- remove
RevertedBlockResolver
- fix
obfuscate
function
- add more details to
/info
endpoint - add protection to not override minted blocks
- Add SANDBOX workflow
- wait for canceling tx to be minted
- fix linters errors
- fix default paths to feeds files
- better error handling for
predict the future
error
- Updated /debug endpoint
- Polygon.io for crypto
- change error message for
SignatureCollector
to reflect issue better
- make log level configurable
- fix gas calculation
- new fetcher CoinmarketcapHistoDay
- new fetcher CoinmarketcapHistoHour
OnChain
feed collector
- restrict packages versions with
~
- Remove last block if it was not minted
- save blocks at Block Sign request
- refactored block saving mechanism for code reuse
- typo fix. revered -> reverted
- split BlockSigner into services for better QA
- potentially fix issue with discrepancies by refactoring code
- leaf collection is no longer used
- DB migrations
- Ignore out-of-date crypto prices in CryptoCompareWSClient
- Fixed VWAP calculation
- Fixed aggregate price calculation
- CryptoCompare re-connect timeout
- remove loadFeeds to use toolbox's loadFeeds
- calculate gas price dynamically
- TX cancellation for pending TXs
- worker's jobs are now deleted from redis when completed
- Minted blocks are reported to NewRelic
- remove unnecessary consensus data, that block explorer is getting directly from blockchain
- Moves constants to its own directory
- added reporting of reverted blocks to NewRelic
- added reporting of data source errors to NewRelic
- added reporting of transaction errors to NewRelic
- support CryptoCompare Multiple Symbols Price API
- cicd workflow run to set production tag
- add new worker to report metrics to NewRelic
- Improve tests execution and cleanup
- Ensure Prod build is not executed if tests failed
- Add develop workflow for the E2E testing (SDK and Reff App)
- Add Badges for actions and Argocd
- collect consensus data from events
- support storing only latest FCDs
- support full
Chain
storage optimisation
- delete deprecated blocks on boot
- remove
MerkleTree
class (use one from SDK)
- Average CPI implemented
- Fixed BlockSigner and the case when validator fails to calculate values
- Support hundreds of US equities
- logging for consensus retries and version check
- added new event
PriceDiscrepancy
reported to NewRelic - self-adjusting consensus, that can ignore discrepancy pairs
- set
dataTimestamp
with an offset
- use
dataTimestamp
as part of consensus - use
chain.getStatus
to pull all consensus data
ValidatorRegistry
- cryptocompare logging
- Resubscribe stale subscriptions
- Check cryptocompare heartbeat
- Save all values and timestamps on redis
- Debug endpoint
- code prettier
- Keep at least one value provided by cryptocompare
- Fixed SIGNATURE_TIMEOUT
- SIGNATURE_TIMEOUT is defaulted to 15 seconds
- optimized getValidators
- configure CryptoCompare price expiry
- added extra logs
- Added StatsD Client
- submit prices based on a timestamp
- better error handling in /info
- timout for /signature endpoint
- More logging to BlockSigner
- Disconnect CryptoCompare WS every 4 hours
- feed files can be loaded from a remote host
- check for the leader based on the next block
- NewRelic is disabled by default
- feedsOnChain.yaml
- CryptoCompare data source for UMB
- Extra logging with coloring
- Multiple optimizations to speed up consensus
- The leader includes his signature first
- No EX for CryptoCompare WS
- Fixed MongoDB queries to be compatible with 4.0.0
- deal with discrepancies
- /blocks endpoint
- renamed HOLO to HOT
- documented all env variables
- multi-node set-up
- do not cache
ValidatorRegistry
- Repository Migration
- Support case when blocks can be reverted
- precision in FeedProcessor
- do not cache
Chain
address, pull it from registry - environment info
- update toolbox version
- FeedProcessor assigned the same label to all values
- add validator to
/info
endpoint
- fix
/info
endpoint
- updated package.json version
- fixed an bug in ChainContract::submit when Signer was not provided
- bumped toolbox version
- initial version
- use registry to resolve Chain address
- support numeric first class data
- cryptocompare API support
- Genesis Volatility support
- support multiple data sources
- validate feeds.json through schema
- use
@umb-network/toolbox
for fetching ABIs for contracts