Skip to content

Commit

Permalink
[explorer] batch queryEvent calls using the max page limit size of 10…
Browse files Browse the repository at this point in the history
…00 to fix APY calculations (MystenLabs#10117)

## Description 
When we fetch validator events in the Explorer, we say that the limit is
`30 * numValidators` which on testnet is 2850. The problem is that the
`queryEvents` API has a maximum page size of `1000`, so the limit we
were providing was causing the API to error out.

To fix this, we can batch the API calls together when the limit exceeds
the maximum page size. So if we did have 2850 events, we'd fetch 1000 ->
fetch another 1000 -> fetch the remaining 850. One thing to note is that
there are only 190 validator events (2 completed epochs * 95 validators)
thus far, so this batching functionality isn't actually being used at
the moment.

## Test Plan 
- Manual testing on devnet with:
   -  4 validators and a query limit of 1
   - 4 validators and a query limit of 1000
   - 4 validators with an event limit of 5 and a query limit of 1
- Manual testing on testnet with:
   - 95 validators and a query limit of 50
   - 95 validators and a query limit of 1000
- 95 validators with an event limit of 5 and a query limit of 2 (limit
truncation case)

---
If your changes are not user-facing and not a breaking change, you can
skip the following section. Otherwise, please indicate what changed, and
then add to the Release Notes section as highlighted during the release
process.

### Type of Change (Check all that apply)

- [X] user-visible impact
- [ ] breaking change for a client SDKs
- [ ] breaking change for FNs (FN binary must upgrade)
- [ ] breaking change for validators or node operators (must upgrade
binaries)
- [ ] breaking change for on-chain data layout
- [ ] necessitate either a data wipe or data migration

### Release notes
N/A
  • Loading branch information
williamrobertson13 authored Mar 30, 2023
1 parent 1cd7ced commit 121f00f
Show file tree
Hide file tree
Showing 4 changed files with 51 additions and 23 deletions.
7 changes: 2 additions & 5 deletions apps/core/src/hooks/useGetRollingAverageApys.ts
Original file line number Diff line number Diff line change
Expand Up @@ -60,14 +60,11 @@ export function useGetRollingAverageApys(numberOfValidators: number | null) {

const apyByValidator =
useMemo<ApyByValidator | null>(() => {
if (
!validatorEpochEvents?.data ||
!validatorEpochEvents?.data?.data
) {
if (!validatorEpochEvents?.data) {
return null;
}
const apyGroups: ApyGroups = {};
validatorEpochEvents.data.data.forEach(({ parsedJson }) => {
validatorEpochEvents.data.forEach(({ parsedJson }) => {
const { stake, pool_staking_reward, validator_address } =
parsedJson as ParsedJson;

Expand Down
59 changes: 45 additions & 14 deletions apps/core/src/hooks/useGetValidatorsEvents.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,33 +2,64 @@
// SPDX-License-Identifier: Apache-2.0

import { useRpcClient } from '../api/RpcClientContext';
import { type EventId, VALIDATORS_EVENTS_QUERY } from '@mysten/sui.js';
import {
type EventId,
VALIDATORS_EVENTS_QUERY,
SuiEvent,
} from '@mysten/sui.js';
import { useQuery } from '@tanstack/react-query';

type GetValidatorsEvent = {
cursor?: EventId | null;
limit: number | null;
order: 'ascending' | 'descending';
};

// NOTE: This copys the query limit from our Rust JSON RPC backend, this needs to be kept in sync!
const QUERY_MAX_RESULT_LIMIT = 1000;

//TODO: get validatorEvents by validator address
export function useGetValidatorsEvents({
cursor,
limit,
order,
}: GetValidatorsEvent) {
export function useGetValidatorsEvents({ limit, order }: GetValidatorsEvent) {
const rpc = useRpcClient();
// since we are getting events base on the number of validators, we need to make sure that the limit is not null and cache by the limit
// number of validators can change from network to network
// Since we are getting events based on the number of validators, we need to make sure that the limit
// is not null and cache by the limit number of validators can change from network to network
return useQuery(
['validatorEvents', limit, cursor, order],
() =>
rpc.queryEvents({
['validatorEvents', limit, order],
async () => {
if (!limit) {
// Do some validation at the runtime level for some extra type-safety
// https://tkdodo.eu/blog/react-query-and-type-script#type-safety-with-the-enabled-option
throw new Error(
`Limit needs to always be defined and non-zero! Received ${limit} instead.`
);
}

if (limit > QUERY_MAX_RESULT_LIMIT) {
let hasNextPage = true;
let currCursor: EventId | null | undefined;
const results: SuiEvent[] = [];

while (hasNextPage && results.length < limit) {
const validatorEventsResponse = await rpc.queryEvents({
query: { MoveEventType: VALIDATORS_EVENTS_QUERY },
cursor: currCursor,
limit: Math.min(limit, QUERY_MAX_RESULT_LIMIT),
order,
});

hasNextPage = validatorEventsResponse.hasNextPage;
currCursor = validatorEventsResponse.nextCursor;
results.push(...validatorEventsResponse.data);
}
return results.slice(0, limit);
}

const validatorEventsResponse = await rpc.queryEvents({
query: { MoveEventType: VALIDATORS_EVENTS_QUERY },
cursor,
limit,
order,
}),
});
return validatorEventsResponse.data;
},
{ enabled: !!limit }
);
}
4 changes: 2 additions & 2 deletions apps/explorer/src/pages/validator/ValidatorDetails.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ function ValidatorDetails() {
const validatorRewards = useMemo(() => {
if (!validatorEvents || !id) return 0;
const rewards = getValidatorMoveEvent(
validatorEvents.data,
validatorEvents,
id
)?.pool_staking_reward;
return +rewards || 0;
Expand All @@ -60,7 +60,7 @@ function ValidatorDetails() {

const apy = rollingAverageApys?.[id] || 0;
const tallyingScore =
validatorEvents?.data.find(
validatorEvents?.find(
({ parsedJson }) => parsedJson?.validator_address === id
)?.parsedJson?.tallying_rule_global_score || null;
return (
Expand Down
4 changes: 2 additions & 2 deletions apps/explorer/src/pages/validators/Validators.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -227,7 +227,7 @@ function ValidatorPageResult() {
if (!validatorEvents) return 0;
let totalRewards = 0;

validatorEvents.data.forEach(({ parsedJson }) => {
validatorEvents.forEach(({ parsedJson }) => {
totalRewards += +parsedJson!.pool_staking_reward;
});

Expand All @@ -239,7 +239,7 @@ function ValidatorPageResult() {
return validatorsTableData(
data.activeValidators,
data.atRiskValidators,
validatorEvents.data,
validatorEvents,
rollingAverageApys
);
}, [data, validatorEvents, rollingAverageApys]);
Expand Down

0 comments on commit 121f00f

Please sign in to comment.