feature(forc-pkg): Support IPFS-sourced dependencies (FuelLabs#4299)

## Description
Closes FuelLabs#3926.

(edited by @kayagokalp to reflect latest status of the PR)

This PR adds IPFS support for forc-pkg. To do so, forc relies on
existing ipfs local node without embedding a node to forc itself. If
local system does not have an ipfs node running, a public gateway
(https://ipfs.io) is used as a fallback mechanism and packages are
fetched through that.

## Some Context and Future Work

For our package registry stuff to work properly we will eventually need
to host an IPFS node to make sure that our published packages are pinned
by at least one node in the network. That is the case as long as we
don't have a way to incentivize people pinning the packages they
published. What will happen is that a package will be published by a
developer and after we detect the published package our own IPFS node
will pin it to make sure it is always accessible.

Also some benchmarks are done for public gateway during development. It
looks like for initial fetch operations from public gateway spends some
time looking for the package pinned by the IPFS node simulating our
incoming fuel IPFS node which is explained above. It is manageable but
fetching from the fuel IPFS node is instant as it already got the
package pinned. We can consider falling back to our own node's gateway
api rather than a public one to smooth the process.

Finally maybe we can explore embedded ipfs node option as a follow-up,
due to the status of IPFS with Rust, it is not very obvious (maybe not
event possible atm) how to be able to be fully compatible with kubo, for
fetching/publishing our packages. But this is still an open question
worth exploring.

## Testing

### To test with a local node:

1. You can install kubo, you can follow the instructions in their
website while installing https://docs.ipfs.tech/install/command-line/
2. Run the `ipfs daemon` service with `ipfs daemon`
3. Add an ipfs source, I have a package already pinned by my IPFS node
that you can use:
```toml
[dependencies]
test_lib = { ipfs = "QmfZ3uH7dFEDkYN5RQfyu4m7L8uk8kGiLkNwzHqsrormSj" }
```
4. Run `forc build`

### To test public gateway fallback:

1. Either do this before starting `ipfs daemon` or stop your local
daemon with `ipfs shutdown`.
Add an ipfs source, I have a package already pinned by my IPFS node that
you can use:
```toml
[dependencies]
test_lib = { ipfs = "QmfZ3uH7dFEDkYN5RQfyu4m7L8uk8kGiLkNwzHqsrormSj" }
```
2. Run `forc build`


---------

Co-authored-by: kayagokalp <[email protected]>

docs/book/src/forc/dependencies.md

@@ -1,10 +1,10 @@
 # Dependencies
 
-Forc has a dependency management system which can pull packages using git. This allows users to build and share Forc libraries.
+Forc has a dependency management system which can pull packages using git and ipfs. This allows users to build and share Forc libraries.
 
 ## Adding a dependency
 
-If your `Forc.toml` doesn't already have a `[dependencies]` table, add one. Below, list the package name alongside its source. Currently, `forc` supports both `git` and `path` sources.
+If your `Forc.toml` doesn't already have a `[dependencies]` table, add one. Below, list the package name alongside its source. Currently, `forc` supports `git`, `ipfs` and `path` sources.
 
 If a `git` source is specified, `forc` will fetch the git repository at the given URL and then search for a `Forc.toml` for a package with the given name anywhere inside the git repository.
 
@@ -24,8 +24,17 @@ Depending on a local library using `path`:
 custom_lib = { path = "../custom_lib" }
 ```
 
+For `ipfs` sources, `forc` will fetch the specified `cid` using either a local ipfs node or a public gateway. `forc` automatically tries to connect to local `ipfs` node and if it fails, fallbacks to using `https://ipfs.io/`, as a gateway.
+
+The following example adds a dependency with an `ipfs` source.
+
+```toml
+[dependencies]
+custom_lib = { ipfs = "QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG" }
+```
+
 Once the package is added, running `forc build` will automatically download added dependencies.
 
 ## Updating dependencies
 
-To update dependencies in your Forc directory you can run `forc update`. For `path` dependencies this will have no effect. For `git` dependencies with a `branch` reference, this will update the project to use the latest commit for the given branch.
+To update dependencies in your Forc directory you can run `forc update`. For `path` and `ipfs` dependencies this will have no effect. For `git` dependencies with a `branch` reference, this will update the project to use the latest commit for the given branch.

forc-pkg/Cargo.toml

@@ -11,14 +11,18 @@ repository.workspace = true
 [dependencies]
 ansi_term = "0.12"
 anyhow = "1"
+cid = "0.10"
 fd-lock = "3.0"
 forc-tracing = { version = "0.40.1", path = "../forc-tracing" }
 forc-util = { version = "0.40.1", path = "../forc-util" }
 fuel-abi-types = "0.1"
+futures = "0.3"
 git2 = { version = "0.16.1", features = ["vendored-libgit2", "vendored-openssl"] }
 gix-url = { version = "0.16.0", features = ["serde1"] }
 hex = "0.4.3"
+ipfs-api-backend-hyper = { version = "0.6", features = ["with-builder"] }
 petgraph = { version = "0.6", features = ["serde-1"] }
+reqwest = "0.11.7"
 semver = { version = "1.0", features = ["serde"] }
 serde = { version = "1.0", features = ["derive"] }
 serde_ignored = "0.1"
@@ -28,6 +32,7 @@ sway-error = { version = "0.40.1", path = "../sway-error" }
 sway-types = { version = "0.40.1", path = "../sway-types" }
 sway-utils = { version = "0.40.1", path = "../sway-utils" }
 sysinfo = "0.29.0"
+tar = "0.4.38"
 toml = "0.5"
 tracing = "0.1"
 url = { version = "2.2", features = ["serde"] }

forc-pkg/src/manifest.rs

@@ -197,6 +197,7 @@ pub struct DependencyDetails {
     pub(crate) tag: Option<String>,
     pub(crate) package: Option<String>,
     pub(crate) rev: Option<String>,
+    pub(crate) ipfs: Option<String>,
 }
 
 /// Parameters to pass through to the `sway_core::BuildConfig` during compilation.
@@ -1021,13 +1022,20 @@ mod tests {
             tag: None,
             package: None,
             rev: None,
+            ipfs: None,
         };
 
         let dependency_details_branch = DependencyDetails {
             path: None,
             ..dependency_details_path_branch.clone()
         };
 
+        let dependency_details_ipfs_branch = DependencyDetails {
+            path: None,
+            ipfs: Some("QmVxgEbiDDdHpG9AesCpZAqNvHYp1P3tWLFdrpUBWPMBcc".to_string()),
+            ..dependency_details_path_branch.clone()
+        };
+
         let dependency_details_path_tag = DependencyDetails {
             version: None,
             path: Some("example_path/".to_string()),
@@ -1036,20 +1044,28 @@ mod tests {
             tag: Some("v0.1.0".to_string()),
             package: None,
             rev: None,
+            ipfs: None,
         };
 
         let dependency_details_tag = DependencyDetails {
             path: None,
             ..dependency_details_path_tag.clone()
         };
 
+        let dependency_details_ipfs_tag = DependencyDetails {
+            path: None,
+            ipfs: Some("QmVxgEbiDDdHpG9AesCpZAqNvHYp1P3tWLFdrpUBWPMBcc".to_string()),
+            ..dependency_details_path_branch.clone()
+        };
+
         let dependency_details_path_rev = DependencyDetails {
             version: None,
             path: Some("example_path/".to_string()),
             git: None,
             branch: None,
             tag: None,
             package: None,
+            ipfs: None,
             rev: Some("9f35b8e".to_string()),
         };
 
@@ -1058,6 +1074,12 @@ mod tests {
             ..dependency_details_path_rev.clone()
         };
 
+        let dependency_details_ipfs_rev = DependencyDetails {
+            path: None,
+            ipfs: Some("QmVxgEbiDDdHpG9AesCpZAqNvHYp1P3tWLFdrpUBWPMBcc".to_string()),
+            ..dependency_details_path_branch.clone()
+        };
+
         let expected_mismatch_error = "Details reserved for git sources used without a git field";
         assert_eq!(
             dependency_details_path_branch
@@ -1066,20 +1088,41 @@ mod tests {
                 .map(|e| e.to_string()),
             Some(expected_mismatch_error.to_string())
         );
+        assert_eq!(
+            dependency_details_ipfs_branch
+                .validate()
+                .err()
+                .map(|e| e.to_string()),
+            Some(expected_mismatch_error.to_string())
+        );
         assert_eq!(
             dependency_details_path_tag
                 .validate()
                 .err()
                 .map(|e| e.to_string()),
             Some(expected_mismatch_error.to_string())
         );
+        assert_eq!(
+            dependency_details_ipfs_tag
+                .validate()
+                .err()
+                .map(|e| e.to_string()),
+            Some(expected_mismatch_error.to_string())
+        );
         assert_eq!(
             dependency_details_path_rev
                 .validate()
                 .err()
                 .map(|e| e.to_string()),
             Some(expected_mismatch_error.to_string())
         );
+        assert_eq!(
+            dependency_details_ipfs_rev
+                .validate()
+                .err()
+                .map(|e| e.to_string()),
+            Some(expected_mismatch_error.to_string())
+        );
         assert_eq!(
             dependency_details_branch
                 .validate()
@@ -1113,6 +1156,7 @@ mod tests {
             tag: None,
             package: None,
             rev: None,
+            ipfs: None,
         };
 
         let git_source_string = "https://github.com/FuelLabs/sway".to_string();
@@ -1124,6 +1168,7 @@ mod tests {
             tag: Some("v0.1.0".to_string()),
             package: None,
             rev: None,
+            ipfs: None,
         };
         let dependency_details_git_branch = DependencyDetails {
             version: None,
@@ -1133,6 +1178,7 @@ mod tests {
             tag: None,
             package: None,
             rev: None,
+            ipfs: None,
         };
         let dependency_details_git_rev = DependencyDetails {
             version: None,
@@ -1142,11 +1188,24 @@ mod tests {
             tag: None,
             package: None,
             rev: Some("9f35b8e".to_string()),
+            ipfs: None,
+        };
+
+        let dependency_details_ipfs = DependencyDetails {
+            version: None,
+            path: None,
+            git: None,
+            branch: None,
+            tag: None,
+            package: None,
+            rev: None,
+            ipfs: Some("QmVxgEbiDDdHpG9AesCpZAqNvHYp1P3tWLFdrpUBWPMBcc".to_string()),
         };
 
         assert!(dependency_details_path.validate().is_ok());
         assert!(dependency_details_git_tag.validate().is_ok());
         assert!(dependency_details_git_branch.validate().is_ok());
         assert!(dependency_details_git_rev.validate().is_ok());
+        assert!(dependency_details_ipfs.validate().is_ok());
     }
 }

forc-pkg/src/pkg.rs

@@ -1,7 +1,7 @@
 use crate::{
     lock::Lock,
     manifest::{BuildProfile, Dependency, ManifestFile, MemberManifestFiles, PackageManifestFile},
-    source::{self, Source},
+    source::{self, IPFSNode, Source},
     CORE, PRELUDE, STD,
 };
 use anyhow::{anyhow, bail, Context, Error, Result};
@@ -240,6 +240,8 @@ pub struct PkgOpts {
     pub output_directory: Option<String>,
     /// Outputs json abi with callpath instead of struct and enum names.
     pub json_abi_with_callpaths: bool,
+    /// The IPFS node to be used for fetching IPFS sources.
+    pub ipfs_node: IPFSNode,
 }
 
 #[derive(Default, Clone)]
@@ -582,18 +584,29 @@ impl BuildPlan {
             &member_manifests,
             build_options.pkg.locked,
             build_options.pkg.offline,
+            build_options.pkg.ipfs_node.clone(),
         )
     }
 
     /// Create a new build plan for the project by fetching and pinning all dependenies.
     ///
     /// To account for an existing lock file, use `from_lock_and_manifest` instead.
-    pub fn from_manifests(manifests: &MemberManifestFiles, offline: bool) -> Result<Self> {
+    pub fn from_manifests(
+        manifests: &MemberManifestFiles,
+        offline: bool,
+        ipfs_node: IPFSNode,
+    ) -> Result<Self> {
         // Check toolchain version
         validate_version(manifests)?;
         let mut graph = Graph::default();
         let mut manifest_map = ManifestMap::default();
-        fetch_graph(manifests, offline, &mut graph, &mut manifest_map)?;
+        fetch_graph(
+            manifests,
+            offline,
+            &ipfs_node,
+            &mut graph,
+            &mut manifest_map,
+        )?;
         // Validate the graph, since we constructed the graph from scratch the paths will not be a
         // problem but the version check is still needed
         validate_graph(&graph, manifests)?;
@@ -626,6 +639,7 @@ impl BuildPlan {
         manifests: &MemberManifestFiles,
         locked: bool,
         offline: bool,
+        ipfs_node: IPFSNode,
     ) -> Result<Self> {
         // Check toolchain version
         validate_version(manifests)?;
@@ -665,7 +679,13 @@ impl BuildPlan {
         let mut manifest_map = graph_to_manifest_map(manifests, &graph)?;
 
         // Attempt to fetch the remainder of the graph.
-        let _added = fetch_graph(manifests, offline, &mut graph, &mut manifest_map)?;
+        let _added = fetch_graph(
+            manifests,
+            offline,
+            &ipfs_node,
+            &mut graph,
+            &mut manifest_map,
+        )?;
 
         // Determine the compilation order.
         let compilation_order = compilation_order(&graph)?;
@@ -1292,7 +1312,10 @@ fn find_path_root(graph: &Graph, mut node: NodeIx) -> Result<NodeIx> {
                     })?;
                 node = parent;
             }
-            source::Pinned::Git(_) | source::Pinned::Registry(_) | source::Pinned::Member(_) => {
+            source::Pinned::Git(_)
+            | source::Pinned::Ipfs(_)
+            | source::Pinned::Member(_)
+            | source::Pinned::Registry(_) => {
                 return Ok(node);
             }
         }
@@ -1309,6 +1332,7 @@ fn find_path_root(graph: &Graph, mut node: NodeIx) -> Result<NodeIx> {
 fn fetch_graph(
     member_manifests: &MemberManifestFiles,
     offline: bool,
+    ipfs_node: &IPFSNode,
     graph: &mut Graph,
     manifest_map: &mut ManifestMap,
 ) -> Result<HashSet<NodeIx>> {
@@ -1317,6 +1341,7 @@ fn fetch_graph(
         added_nodes.extend(&fetch_pkg_graph(
             member_pkg_manifest,
             offline,
+            ipfs_node,
             graph,
             manifest_map,
             member_manifests,
@@ -1342,6 +1367,7 @@ fn fetch_graph(
 fn fetch_pkg_graph(
     proj_manifest: &PackageManifestFile,
     offline: bool,
+    ipfs_node: &IPFSNode,
     graph: &mut Graph,
     manifest_map: &mut ManifestMap,
     member_manifests: &MemberManifestFiles,
@@ -1376,6 +1402,7 @@ fn fetch_pkg_graph(
     fetch_deps(
         fetch_id,
         offline,
+        ipfs_node,
         proj_node,
         path_root,
         graph,
@@ -1393,6 +1420,7 @@ fn fetch_pkg_graph(
 fn fetch_deps(
     fetch_id: u64,
     offline: bool,
+    ipfs_node: &IPFSNode,
     node: NodeIx,
     path_root: PinnedId,
     graph: &mut Graph,
@@ -1441,6 +1469,7 @@ fn fetch_deps(
                     path_root,
                     name: &pkg.name,
                     offline,
+                    ipfs_node,
                 };
                 let source = pkg.source.pin(ctx, manifest_map)?;
                 let name = pkg.name.clone();
@@ -1473,16 +1502,18 @@ fn fetch_deps(
         })?;
 
         let path_root = match dep_pinned.source {
-            source::Pinned::Member(_) | source::Pinned::Git(_) | source::Pinned::Registry(_) => {
-                dep_pkg_id
-            }
+            source::Pinned::Member(_)
+            | source::Pinned::Git(_)
+            | source::Pinned::Ipfs(_)
+            | source::Pinned::Registry(_) => dep_pkg_id,
             source::Pinned::Path(_) => path_root,
         };
 
         // Fetch the children.
         added.extend(fetch_deps(
             fetch_id,
             offline,
+            ipfs_node,
             dep_node,
             path_root,
             graph,
@@ -2712,8 +2743,14 @@ fn test_root_pkg_order() {
     let manifest_file = ManifestFile::from_dir(&manifest_dir).unwrap();
     let member_manifests = manifest_file.member_manifests().unwrap();
     let lock_path = manifest_file.lock_path().unwrap();
-    let build_plan =
-        BuildPlan::from_lock_and_manifests(&lock_path, &member_manifests, false, false).unwrap();
+    let build_plan = BuildPlan::from_lock_and_manifests(
+        &lock_path,
+        &member_manifests,
+        false,
+        false,
+        Default::default(),
+    )
+    .unwrap();
     let graph = build_plan.graph();
     let order: Vec<String> = build_plan
         .member_nodes()