[rpc-loadgen][11/n] increase default gas budget (MystenLabs#10312)

## Description 

Describe the changes or additions included in this PR.

## Test Plan 

How did you test the new or updated feature?

---
If your changes are not user-facing and not a breaking change, you can
skip the following section. Otherwise, please indicate what changed, and
then add to the Release Notes section as highlighted during the release
process.

### Type of Change (Check all that apply)

- [ ] user-visible impact
- [ ] breaking change for a client SDKs
- [ ] breaking change for FNs (FN binary must upgrade)
- [ ] breaking change for validators or node operators (must upgrade
binaries)
- [ ] breaking change for on-chain data layout
- [ ] necessitate either a data wipe or data migration

### Release notes

crates/sui-rpc-loadgen/README.md

@@ -1,10 +1,11 @@
 # sui-rpc-loadgen: Load Generator for SUI RPC Servers
 
-`sui-rpc-loadgen`  is a utility that facilitates the generation of read and write loads on single or multiple Sui RPC servers. Its primary functions include performance testing and data correctness verification.
+`sui-rpc-loadgen` is a utility that facilitates the generation of read and write loads on single or multiple Sui RPC servers. Its primary functions include performance testing and data correctness verification.
 
 ## Features
+
 - **Easily extendable** to support any read/write endpoint
-- **Concurrent load generation** with multiple threads,  making it suitable for load testing high-traffic RPC servers.
+- **Concurrent load generation** with multiple threads, making it suitable for load testing high-traffic RPC servers.
 - **Cross-verifying** results across multiple RPC Servers, ensuring data consistency and accuracy.
 - **Performance comparison** between vanilla Full node RPC and Enhanced Full node RPC
 
@@ -21,16 +22,20 @@ To try this locally, refer to [sef](../sui-test-validator/README.md). Recommend
 ### Example 1: Get All Checkpoints
 
 The following command initiates a single thread (num-threads == 1) to retrieve all checkpoints from the beginning (sequence 0) to the latest, executing the operation exactly once (repeat == 0):
+
 ```bash
-cargo run --bin sui-rpc-loadgen -- --urls "http://127.0.0.1:9000" "http://127.0.0.1:9124" --num-threads 1 get-checkpoints --start 0 --repeat 0 --interval-in-ms 0 --verify-transactions true
+cargo run --bin sui-rpc-loadgen -- --urls "http://127.0.0.1:9000" "http://127.0.0.1:9124" --num-threads 1 get-checkpoints --start 0 --repeat 0 --interval-in-ms 0
 ```
+
 This command is equivalent to the simplified version below:
+
 ```bash
 cargo run --bin sui-rpc-loadgen -- --urls "http://127.0.0.1:9000" "http://127.0.0.1:9124" --num-threads 1 get-checkpoints
 ```
+
 Both commands achieve the same outcome: fetching all checkpoints using one thread, without repeating the operation.
 
-By default, this command also verify all the transactions in the checkpoint, specify `--verify-transactions false` to disable fetching transactions. Note that this is incompatible with `--verify-objects true` as we do need to fetch transactions to get objects for the checkpoint.
+By default, this command also verify all the transactions in the checkpoint, specify `--skip-verify-transactions` to disable fetching transactions. Note that this must used with `--skip-verify-objects` as we do need to fetch transactions to get objects for the checkpoint.
 
 **Note** you must put `--num-threads ` after the urls, otherwise the command will not be parsed correctly
 
@@ -39,14 +44,17 @@ By default, this command also verify all the transactions in the checkpoint, spe
 ```bash
 cargo run --bin sui-rpc-loadgen -- --urls "http://127.0.0.1:9000" --num-threads 1 pay-sui --repeat 100
 ```
+
 **NOTE**: right now `pay-sui` only supports 1 thread but multi-threading support can be added pretty easily by assigning different gas coins to different threads
 
 ### Example 3: Query Transaction Blocks
+
 ```bash
 cargo run --bin sui-rpc-loadgen -- --urls "http://127.0.0.1:9000" "http://127.0.0.1:9000" --num-threads 4 query-transaction-blocks --address-type from
 ```
 
 ### Multi Get Objects
+
 ```bash
 cargo run --bin sui-rpc-loadgen -- --urls "http://127.0.0.1:9000" "http://127.0.0.1:9000" --num-threads 4 multi-get-objects
 ```
@@ -57,9 +65,11 @@ cargo run --bin sui-rpc-loadgen -- --urls "http://127.0.0.1:9000" "http://127.0.
 ```
 
 # Useful commands
+
 ```bash
 cat sui-rpc-loadgen.b844f547-d354-4871-b958-1ea3fe23a0a8.log.2023-03-23 | awk '/Finished processing/{print $7}' | sort -n | uniq | awk 'BEGIN{last=0}{for(i=last+1;i<$1;i++) print i; last=$1} END{print last}' | tee missing_numbers.txt && wc -l missing_numbers.txt
 ```
+
 Checks which checkpoints among threads have not been processed yet. The last one should be the largest checkpoint being processed.
 
 `wc -l missing_numbers.txt` - counts how many checkpoints to go

crates/sui-rpc-loadgen/src/main.rs

@@ -74,15 +74,15 @@ pub enum ClapCommand {
         #[clap(short, long)]
         end: Option<u64>,
 
-        #[clap(long, default_value_t = true)]
-        verify_transactions: bool,
+        #[clap(long)]
+        skip_verify_transactions: bool,
 
-        #[clap(long, default_value_t = true)]
-        verify_objects: bool,
+        #[clap(long)]
+        skip_verify_objects: bool,
 
         // Whether to record data from checkpoint
-        #[clap(long, default_value_t = true)]
-        record: bool,
+        #[clap(long)]
+        skip_record: bool,
 
         #[clap(flatten)]
         common: CommonOptions,
@@ -173,11 +173,17 @@ async fn main() -> Result<(), Box<dyn Error>> {
             common,
             start,
             end,
-            verify_transactions,
-            verify_objects,
-            record,
+            skip_verify_transactions,
+            skip_verify_objects,
+            skip_record,
         } => (
-            Command::new_get_checkpoints(start, end, verify_transactions, verify_objects, record),
+            Command::new_get_checkpoints(
+                start,
+                end,
+                !skip_verify_transactions,
+                !skip_verify_objects,
+                !skip_record,
+            ),
             common,
             false,
         ),

crates/sui-rpc-loadgen/src/payload/get_checkpoints.rs

@@ -91,12 +91,14 @@ impl<'a> ProcessPayload<'a, &'a GetCheckpoints> for RpcCommandProcessor {
                 .concat();
 
                 if op.record {
+                    debug!("adding addresses and object ids from response");
                     self.add_addresses_from_response(&transaction_responses);
                     self.add_object_ids_from_response(&transaction_responses);
                 };
             }
 
             if op.record {
+                debug!("adding transaction digests from response");
                 self.add_transaction_digests(transaction_digests);
             };
 

crates/sui-rpc-loadgen/src/payload/rpc_command_processor.rs

@@ -35,9 +35,9 @@ use crate::payload::{
     Processor, QueryTransactionBlocks, SignerInfo,
 };
 
-pub(crate) const DEFAULT_GAS_BUDGET: u64 = 10_000;
-pub(crate) const DEFAULT_LARGE_GAS_BUDGET: u64 = 100_000_000;
-pub(crate) const MAX_NUM_NEW_OBJECTS_IN_SINGLE_TRANSACTION: usize = 2000;
+pub(crate) const DEFAULT_GAS_BUDGET: u64 = 500_000_000;
+pub(crate) const DEFAULT_LARGE_GAS_BUDGET: u64 = 50_000_000_000;
+pub(crate) const MAX_NUM_NEW_OBJECTS_IN_SINGLE_TRANSACTION: usize = 120;
 
 #[derive(Clone)]
 pub struct RpcCommandProcessor {
@@ -178,6 +178,7 @@ impl RpcCommandProcessor {
         // TODO: be more granular
         let digests: Vec<TransactionDigest> = self.transaction_digests.iter().map(|x| *x).collect();
         if !digests.is_empty() {
+            debug!("dumping transaction digests to file {:?}", digests.len());
             write_data_to_file(
                 &digests,
                 &format!("{}/{}", &self.data_dir, CacheType::TransactionDigest),
@@ -187,6 +188,7 @@ impl RpcCommandProcessor {
 
         let addresses: Vec<SuiAddress> = self.addresses.iter().map(|x| *x).collect();
         if !addresses.is_empty() {
+            debug!("dumping addresses to file {:?}", addresses.len());
             write_data_to_file(
                 &addresses,
                 &format!("{}/{}", &self.data_dir, CacheType::SuiAddress),
@@ -203,6 +205,7 @@ impl RpcCommandProcessor {
         }
 
         if !object_ids.is_empty() {
+            debug!("dumping object_ids to file {:?}", object_ids.len());
             write_data_to_file(
                 &object_ids,
                 &format!("{}/{}", &self.data_dir, CacheType::ObjectID),
@@ -240,12 +243,24 @@ impl Processor for RpcCommandProcessor {
 
     async fn prepare(&self, config: &LoadTestConfig) -> Result<Vec<Payload>> {
         let clients = self.get_clients().await?;
+        let Command {
+            repeat_n_times,
+            repeat_interval,
+            ..
+        } = &config.command;
         let command_payloads = match &config.command.data {
             CommandData::GetCheckpoints(data) => {
                 if !config.divide_tasks {
                     vec![config.command.clone(); config.num_threads]
                 } else {
-                    divide_checkpoint_tasks(&clients, data, config.num_threads).await
+                    divide_checkpoint_tasks(
+                        &clients,
+                        data,
+                        config.num_threads,
+                        *repeat_n_times,
+                        *repeat_interval,
+                    )
+                    .await
                 }
             }
             CommandData::QueryTransactionBlocks(data) => {
@@ -379,6 +394,8 @@ async fn divide_checkpoint_tasks(
     clients: &[SuiClient],
     data: &GetCheckpoints,
     num_chunks: usize,
+    repeat_n_times: usize,
+    repeat_interval: Duration,
 ) -> Vec<Command> {
     let start = data.start;
     let end = match data.end {
@@ -411,6 +428,8 @@ async fn divide_checkpoint_tasks(
                 data.verify_objects,
                 data.record,
             )
+            .with_repeat_interval(repeat_interval)
+            .with_repeat_n_times(repeat_n_times)
         })
         .collect()
 }

crates/sui-rpc-loadgen/src/payload/validation.rs

@@ -5,7 +5,6 @@ use futures::future::join_all;
 use itertools::Itertools;
 use std::collections::HashSet;
 use std::fmt::Debug;
-use sui_json_rpc::api::QUERY_MAX_RESULT_LIMIT;
 use sui_json_rpc_types::{
     SuiObjectDataOptions, SuiObjectResponse, SuiTransactionBlockEffectsAPI,
     SuiTransactionBlockResponse, SuiTransactionBlockResponseOptions,
@@ -15,12 +14,13 @@ use sui_types::base_types::{ObjectID, TransactionDigest};
 use tracing::error;
 use tracing::log::warn;
 
+const LOADGEN_QUERY_MAX_RESULT_LIMIT: usize = 25;
+
 pub(crate) fn cross_validate_entities<U>(entities: &Vec<Vec<U>>, entity_name: &str)
 where
     U: PartialEq + Debug,
 {
     if entities.len() < 2 {
-        error!("Unable to cross validate as {} less than 2", entity_name);
         return;
     }
 
@@ -128,7 +128,7 @@ pub(crate) fn chunk_entities<U>(entities: &[U], chunk_size: Option<usize>) -> Ve
 where
     U: Clone + PartialEq + Debug,
 {
-    let chunk_size = chunk_size.unwrap_or(QUERY_MAX_RESULT_LIMIT);
+    let chunk_size = chunk_size.unwrap_or(LOADGEN_QUERY_MAX_RESULT_LIMIT);
     entities
         .iter()
         .cloned()
@@ -158,13 +158,13 @@ pub(crate) async fn multi_get_object(
     object_ids: &[ObjectID],
 ) -> Vec<Vec<SuiObjectResponse>> {
     let objects: Vec<Vec<SuiObjectResponse>> = join_all(clients.iter().map(|client| async move {
-        let object_ids = if object_ids.len() > QUERY_MAX_RESULT_LIMIT {
+        let object_ids = if object_ids.len() > LOADGEN_QUERY_MAX_RESULT_LIMIT {
             warn!(
                 "The input size for multi_get_object_with_options has exceed the query limit\
-         {QUERY_MAX_RESULT_LIMIT}: {}, time to implement chunking",
+         {LOADGEN_QUERY_MAX_RESULT_LIMIT}: {}, time to implement chunking",
                 object_ids.len()
             );
-            &object_ids[0..QUERY_MAX_RESULT_LIMIT]
+            &object_ids[0..LOADGEN_QUERY_MAX_RESULT_LIMIT]
         } else {
             object_ids
         };