Reorg to support client/server architecture (risc0#808)

## Why

The zkVM is moving towards a client/server architecture for two main
reasons:
1. Streamline the installation of the zkVM for most users.
2. Allow multiple versions of the zkVM (and hence circuits) to be used
by clients

The first is achieved by using prebuilt binaries that include HW
acceleration which is usually expensive to build from source. The second
is to rely on a stable interface between client and server, so that
newer clients can work with older servers. Both are supported by
decoupling the expensive zkVM bits from the user bits so that the
dependency graph of the client is minimized. It's also possible in the
future to support clients written in other languages.

## Reorg

The biggest change is a reorganization of the code:
* zkvm
  * guest
  * host
    * client
    * server
      * exec
      * prove

## Prover Traits

The `Prover` trait has been split into `Prover` and `DynProverImpl`.
There is now a distinction among client-side `Prover`s, and server-side
`ProverImpl`s. A `Prover` is a higher-level interface which includes
execution. A `ProverImpl` is only concerned with the proving phase.

A `ProverOpts` is added to the prove functions so that users can
configure the prover. Right now it only supports setting the hashing
function.

There are three client `Trait` impls:
* Bonsai
* External (calls `r0vm`, not yet complete)
* Local (only available when `prove` feature is enabled).

## Feature Flags

Currently the `prove` feature is enabled by default. In the future, once
the `ExternalProver` is complete and it's easy for users to install the
prebuilt `r0vm` via `cargo risczero install`, the `prove` feature will
not be enabled by default and should only be used by `r0vm`.

A new `client` feature is added to toggle support for the client-side.
The intention is for `default = ["client"]` once client/server is
complete. Pure verify (useful for WASM) is supported with
`default-features = false`.

## Changes

* `RISC0_DEV_MODE` now requires the value to be one of `["1", "true",
"yes"]` to be considered enabled.
* The `provers()` registry is dropped, along with `get_prover()`. Use
`get_prover_impl()` to select a `ProverImpl` based on `ProverOpts`.
* Drop support for low-level custom syscalls on the `ExecutorEnv`. It's
difficult and potentially dangerous to expose this across the
client/server interface.
* Dev mode is implemented as a `ProverImpl` rather than a `Prover` so
that it works independent of the hash function (or other `ProverOpts`)
that are selected.

.vscode/settings.json

@@ -42,4 +42,4 @@
     "risc0/zkvm/methods/guest/Cargo.toml",
     "risc0/zkvm/methods/std/Cargo.toml"
   ]
-}
+}

bonsai/ethereum-relay/src/tests/utils.rs

@@ -20,7 +20,7 @@ pub(crate) mod tests {
         SessionId,
     };
     use ethers::types::{Address, Bytes, H256, U256};
-    use risc0_zkvm::{receipt::InnerReceipt, Receipt};
+    use risc0_zkvm::{InnerReceipt, Receipt};
     use uuid::Uuid;
     use wiremock::{
         matchers::{method, path},

bonsai/ethereum/src/lib.rs

@@ -47,7 +47,7 @@ mod tests {
         },
         utils::{hex, Anvil, AnvilInstance},
     };
-    use risc0_zkvm::{receipt::InnerReceipt, Receipt};
+    use risc0_zkvm::{InnerReceipt, Receipt};
 
     use crate::{
         i_bonsai_relay,

bonsai/rest-api-mock/src/prover.rs

@@ -19,8 +19,7 @@ use std::{
 
 use anyhow::Context;
 use risc0_zkvm::{
-    receipt::InnerReceipt, Executor, ExecutorEnv, MemoryImage, Program, Receipt, MEM_SIZE,
-    PAGE_SIZE,
+    Executor, ExecutorEnv, InnerReceipt, MemoryImage, Program, Receipt, MEM_SIZE, PAGE_SIZE,
 };
 use tokio::sync::mpsc;
 
@@ -102,7 +101,7 @@ impl Prover {
                     .map_err(|e| {
                         anyhow::anyhow!("failed to build executor environment: {:?}", e)
                     })?;
-                let mut exec = Executor::new(env, mem_img);
+                let mut exec = Executor::new(env, mem_img)?;
                 let session = exec.run()?;
 
                 let receipt = Receipt {

examples/waldo/core/src/image.rs

@@ -128,7 +128,9 @@ impl<const N: u32> ImageMerkleTree<N> {
     }
 
     #[cfg(not(target_os = "zkvm"))]
-    pub fn vector_oracle_callback<'a>(&'a self) -> impl Fn(&[u8]) -> Vec<u8> + 'a {
+    pub fn vector_oracle_callback<'a>(
+        &'a self,
+    ) -> impl Fn(risc0_zkvm::Bytes) -> risc0_zkvm::Result<risc0_zkvm::Bytes> + 'a {
         self.0.vector_oracle_callback()
     }
 }

examples/waldo/core/src/merkle.rs

@@ -75,12 +75,14 @@ impl<Element> MerkleTree<Element>
 where
     Element: Hashable<ShaHasher> + Serialize,
 {
-    pub fn vector_oracle_callback<'a>(&'a self) -> impl Fn(&[u8]) -> Vec<u8> + 'a {
+    pub fn vector_oracle_callback<'a>(
+        &'a self,
+    ) -> impl Fn(risc0_zkvm::Bytes) -> risc0_zkvm::Result<risc0_zkvm::Bytes> + 'a {
         |data| {
             // TODO: Using bincode here, but it would likely be better on the guest side to
             // use the risc0 serde crate. I should try to use one of
             // those (again).
-            let index: usize = bincode::deserialize::<u32>(data)
+            let index: usize = bincode::deserialize::<u32>(&data)
                 .unwrap()
                 .try_into()
                 .unwrap();
@@ -89,7 +91,7 @@ where
             let proof = self.prove(index);
 
             assert!(proof.verify(&self.root(), &value));
-            bincode::serialize(&(value, proof)).unwrap()
+            Ok(bincode::serialize(&(value, proof)).unwrap().into())
         }
     }
 }

risc0/circuit/rv32im/benches/eval_check.rs

@@ -52,7 +52,9 @@ pub fn eval_check(c: &mut Criterion) {
     for po2 in [2, 8, 16, 22].iter() {
         let params = EvalCheckParams::new(*po2);
         let hal = std::rc::Rc::new(risc0_zkp::hal::metal::MetalHalSha256::new());
-        let eval = risc0_circuit_rv32im::metal::MetalEvalCheckSha256::new(hal.clone());
+        let eval = risc0_circuit_rv32im::metal::MetalEvalCheck::<
+            risc0_zkp::hal::metal::MetalHashSha256,
+        >::new(hal.clone());
         group.bench_function(BenchmarkId::new("metal", po2), |b| {
             b.iter(|| {
                 eval_check_impl(&params, hal.as_ref(), &eval);

risc0/circuit/rv32im/src/metal.rs

@@ -22,7 +22,7 @@ use risc0_core::field::{
 use risc0_zkp::{
     core::log2_ceil,
     hal::{
-        metal::{BufferImpl as MetalBuffer, MetalHal, MetalHash, MetalHashSha256},
+        metal::{BufferImpl as MetalBuffer, MetalHal, MetalHash},
         EvalCheck,
     },
     INV_RATE,
@@ -50,8 +50,6 @@ impl<MH: MetalHash> MetalEvalCheck<MH> {
     }
 }
 
-pub type MetalEvalCheckSha256 = MetalEvalCheck<MetalHashSha256>;
-
 impl<MH: MetalHash> EvalCheck<MetalHal<MH>> for MetalEvalCheck<MH> {
     #[tracing::instrument(skip_all)]
     fn eval_check(

risc0/r0vm/src/bin/r0vm.rs

@@ -16,8 +16,7 @@ use std::{fs, path::PathBuf, rc::Rc};
 
 use clap::{Args, Parser, ValueEnum};
 use risc0_zkvm::{
-    prove::{get_prover, Prover},
-    Executor, ExecutorEnv, VerifierContext,
+    get_prover_impl, DynProverImpl, Executor, ExecutorEnv, ProverOpts, VerifierContext,
 };
 
 /// Runs a RISC-V ELF binary within the RISC Zero ZKVM.
@@ -114,7 +113,7 @@ fn main() {
         } else if let Some(ref image_path) = args.binfmt.image {
             let image_contents = fs::read(image_path).unwrap();
             let image = bincode::deserialize(&image_contents).unwrap();
-            Executor::new(env, image)
+            Executor::new(env, image).unwrap()
         } else {
             unreachable!()
         };
@@ -149,17 +148,15 @@ fn main() {
 }
 
 impl Cli {
-    fn get_prover(&self) -> Rc<dyn Prover> {
-        if let Some(dev_mode) = std::env::var("RISC0_DEV_MODE").ok() {
-            if dev_mode == "1" || dev_mode == "true" || dev_mode == "yes" {
-                return get_prover("$devmode");
-            }
-        }
-
-        let prover_name = match self.hashfn {
-            HashFn::Sha256 => "$default",
-            HashFn::Poseidon => "$poseidon",
+    fn get_prover(&self) -> Rc<dyn DynProverImpl> {
+        let hashfn = match self.hashfn {
+            HashFn::Sha256 => "sha-256",
+            HashFn::Poseidon => "poseidon",
         };
-        get_prover(prover_name)
+        let opts = ProverOpts {
+            hashfn: hashfn.to_string(),
+        };
+
+        get_prover_impl(&opts).unwrap()
     }
 }

risc0/r0vm/tests/dev_mode.rs

@@ -45,7 +45,7 @@ fn dev_mode() {
     let receipt = run_dev_mode();
     temp_env::with_var("RISC0_DEV_MODE", Some("1"), || {
         receipt.verify(risc0_zkvm_methods::MULTI_TEST_ID).unwrap();
-        assert_eq!(receipt.inner, risc0_zkvm::receipt::InnerReceipt::Fake);
+        assert_eq!(receipt.inner, risc0_zkvm::InnerReceipt::Fake);
     });
 }
 

risc0/r0vm/tests/external_prover.rs

@@ -14,18 +14,14 @@
 
 use anyhow::Result;
 use assert_cmd::cargo::cargo_bin;
-use risc0_zkvm::{
-    prove::{ExternalProver, Prover},
-    serde::to_vec,
-    ExecutorEnv, Receipt,
-};
+use risc0_zkvm::{serde::to_vec, ExecutorEnv, ExternalProver, Prover, Receipt};
 use risc0_zkvm_methods::{multi_test::MultiTestSpec, MULTI_TEST_ELF, MULTI_TEST_ID};
 
 fn prove_nothing() -> Result<Receipt> {
     let input = to_vec(&MultiTestSpec::DoNothing).unwrap();
     let env = ExecutorEnv::builder().add_input(&input).build().unwrap();
     let r0vm_path = cargo_bin("r0vm");
-    let prover = ExternalProver::new("r0vm", r0vm_path, "sha-256");
+    let prover = ExternalProver::new("r0vm", r0vm_path);
     prover.prove_elf(env, MULTI_TEST_ELF)
 }
 

risc0/zkvm/Cargo.toml

@@ -10,11 +10,20 @@ repository = { workspace = true }
 [[bench]]
 name = "fib"
 harness = false
+required-features = ["prove"]
 
 [[bench]]
 name = "guest_run"
 harness = false
 
+[[example]]
+name = "fib"
+required-features = ["prove"]
+
+[[example]]
+name = "loop"
+required-features = ["prove"]
+
 [build-dependencies]
 prost-build = { version = "0.11", optional = true }
 protobuf-src = { version = "1.1", optional = true }
@@ -41,7 +50,7 @@ serde = { version = "1.0", default-features = false, features = [
 addr2line = { version = "0.20", optional = true }
 bincode = { version = "1.3", optional = true }
 bonsai-sdk = { workspace = true, optional = true }
-elf = { version = "0.7", optional = true }
+bytes = { version = "1.4", features = ["serde"], optional = true }
 risc0-binfmt = { workspace = true }
 risc0-circuit-rv32im = { workspace = true }
 risc0-core = { workspace = true }
@@ -83,10 +92,11 @@ tar = "0.4"
 test-log = { version = "0.2", features = ["trace"] }
 
 [features]
+client = ["dep:bincode", "dep:bonsai-sdk", "dep:bytes", "std"]
 cuda = ["prove", "risc0-circuit-rv32im/cuda", "risc0-zkp/cuda"]
 dual = []
 metal = ["prove", "risc0-circuit-rv32im/metal", "risc0-zkp/metal"]
-default = ["prove"]
+default = ["client", "prove"]
 disable-dev-mode = []
 profiler = [
   "dep:addr2line",
@@ -96,9 +106,10 @@ profiler = [
   "dep:protobuf-src",
 ]
 prove = [
+  "client",
   "dep:addr2line",
-  "dep:bonsai-sdk",
   "dep:bincode",
+  "dep:bytes",
   "dep:num-traits",
   "dep:generic-array",
   "dep:getrandom",

risc0/zkvm/benches/fib.rs

@@ -15,7 +15,7 @@
 use criterion::{
     black_box, criterion_group, criterion_main, BatchSize, BenchmarkId, Criterion, Throughput,
 };
-use risc0_zkvm::{prove::default_prover, Executor, ExecutorEnv, VerifierContext};
+use risc0_zkvm::{get_prover_impl, Executor, ExecutorEnv, ProverOpts, VerifierContext};
 use risc0_zkvm_methods::FIB_ELF;
 
 fn setup(iterations: u32) -> Executor<'static> {
@@ -34,7 +34,8 @@ enum Scope {
 pub fn bench(c: &mut Criterion) {
     let mut group = c.benchmark_group("fib");
 
-    let prover = default_prover();
+    let opts = ProverOpts::default();
+    let prover = get_prover_impl(&opts).unwrap();
     let ctx = VerifierContext::default();
 
     for iterations in [100, 1000, 10_000] {

risc0/zkvm/build.rs

@@ -16,6 +16,10 @@ fn main() {
     #[cfg(feature = "profiler")]
     {
         std::env::set_var("PROTOC", protobuf_src::protoc());
-        prost_build::compile_protos(&["src/exec/profile.proto"], &["src/exec/"]).unwrap();
+        prost_build::compile_protos(
+            &["src/host/server/exec/profile.proto"],
+            &["src/host/server/exec"],
+        )
+        .unwrap();
     }
 }

risc0/zkvm/examples/fib.rs

@@ -16,8 +16,7 @@ use std::rc::Rc;
 
 use clap::Parser;
 use risc0_zkvm::{
-    prove::{default_prover, Prover},
-    Executor, ExecutorEnv, VerifierContext,
+    get_prover_impl, DynProverImpl, Executor, ExecutorEnv, ProverOpts, VerifierContext,
 };
 use risc0_zkvm_methods::FIB_ELF;
 use tracing_subscriber::{prelude::*, EnvFilter};
@@ -49,13 +48,13 @@ fn main() {
         .init();
 
     let args = Args::parse();
-    let prover = default_prover();
+    let prover = get_prover_impl(&ProverOpts::default()).unwrap();
     let metrics = top(prover, args.iterations, args.skip_prover);
     println!("{metrics:?}");
 }
 
 #[tracing::instrument(skip_all)]
-fn top(prover: Rc<dyn Prover>, iterations: u32, skip_prover: bool) -> Metrics {
+fn top(prover: Rc<dyn DynProverImpl>, iterations: u32, skip_prover: bool) -> Metrics {
     let env = ExecutorEnv::builder()
         .add_input(&[iterations])
         .build()

risc0/zkvm/examples/loop.rs

@@ -17,10 +17,8 @@ use std::{process::Command, rc::Rc, time::Instant};
 use clap::Parser;
 use human_repr::{HumanCount, HumanDuration};
 use risc0_zkvm::{
-    prove::{default_prover, Prover},
-    receipt::Receipt,
-    serde::to_vec,
-    Executor, ExecutorEnv, Session, VerifierContext,
+    get_prover_impl, serde::to_vec, DynProverImpl, Executor, ExecutorEnv, ProverOpts, Receipt,
+    Session, VerifierContext,
 };
 use risc0_zkvm_methods::{
     bench::{BenchmarkSpec, SpecWithIters},
@@ -46,7 +44,7 @@ fn main() {
             .with(tracing_forest::ForestLayer::default())
             .init();
 
-        let prover = default_prover();
+        let prover = get_prover_impl(&ProverOpts::default()).unwrap();
 
         let start = Instant::now();
         let (session, receipt) = top(prover.clone(), iterations);
@@ -115,7 +113,7 @@ fn run_with_iterations(iterations: usize) {
 }
 
 #[tracing::instrument(skip_all)]
-fn top(prover: Rc<dyn Prover>, iterations: u64) -> (Session, Receipt) {
+fn top(prover: Rc<dyn DynProverImpl>, iterations: u64) -> (Session, Receipt) {
     let spec = SpecWithIters(BenchmarkSpec::SimpleLoop, iterations);
     let env = ExecutorEnv::builder()
         .add_input(&to_vec(&spec).unwrap())