Merge branch 'main' into rp/adr-monitoring

.github/workflows/markdown-lint.yml

@@ -0,0 +1,22 @@
+name: Markdown Lint
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+  release:
+    types: [published]
+
+jobs:
+  markdown-lint:
+    name: Markdown Lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v3
+      - uses: actions/setup-node@v3
+        with:
+          node-version: 18
+      - run: |
+          npm install -g [email protected]
+          markdownlint --config .markdownlint.yaml **/*.md

.markdownlint.yaml

@@ -0,0 +1,3 @@
+"default": true # Default state for all rules
+"MD013": false # Disable rule for line length
+"MD033": false # Disable rule banning inline HTML

CONTRIBUTING.md

@@ -1,6 +1,6 @@
 # Contributing
 
-Thank you for your interest in contributing to Celestia-Node! 
+Thank you for your interest in contributing to Celestia-Node!
 
 All work on the code base should be motivated by [our Github
 issues](https://github.com/celestiaorg/celestia-node/issues). If you
@@ -34,7 +34,8 @@ Each stage of the process is aimed at creating feedback cycles which align contr
 
 ## PR Naming
 
-PRs should be titled as following: 
+PRs should be titled as following:
+
 ```txt
 pkg: Concise title of PR
 ```
@@ -64,20 +65,20 @@ is the author/s of the change.
 
 The main development branch is `main`.
 
-Every release is maintained in a release branch named `vX.Y`. On each respective release branch, we tag the releases 
+Every release is maintained in a release branch named `vX.Y`. On each respective release branch, we tag the releases
 vX.Y.0, vX.Y.1 and so forth.
 
 Note all pull requests should be squash merged except for merging to a release branch (named `vX.Y`). This keeps the commit history clean and makes it
 easy to reference the pull request where a change was introduced.
 
 ### Development Procedure
 
-The latest state of development is on `main`, which must never fail `make test`. _Never_ force push `main`.
+The latest state of development is on `main`, which must never fail `make test`. *Never* force push `main`.
 
 To begin contributing, create a development branch on your fork.
 
-Make changes, and before submitting a pull request, update the `CHANGELOG_PENDING.md` to record your change. Also, `git 
-rebase` on top of the latest `main`. 
+Make changes, and before submitting a pull request, update the `CHANGELOG_PENDING.md` to record your change. Also, `git
+rebase` on top of the latest `main`.
 
 Sometimes (often!) pull requests get out-of-date with main, as other people merge different pull requests to main. It is
 our convention that pull request authors are responsible for updating their branches with `main`. (This also means that you shouldn't update someone else's branch for them; even if it seems like you're doing them a favor, you may be interfering with their git flow in some way!)
@@ -121,4 +122,4 @@ Unit tests are located in `_test.go` files as directed by [the Go testing
 package](https://golang.org/pkg/testing/). If you're adding or removing a
 function, please check there's a `TestType_Method` test for it.
 
-Run: `make test`
+Run: `make test`

README.md

@@ -20,9 +20,10 @@ Continue reading [here](https://blog.celestia.org/celestia-mvp-release-data-avai
 |-------------|----------------|
 | Go version  | 1.18 or higher |
 
-## System Requirements 
+## System Requirements
+
+See the official docs page for system requirements per node type:
 
-See the official docs page for system requirements per node type: 
 * [Bridge](https://docs.celestia.org/nodes/bridge-node#hardware-requirements)
 * [Light](https://docs.celestia.org/nodes/light-node#hardware-requirements)
 * [Full](https://docs.celestia.org/nodes/full-storage-node#hardware-requirements)
@@ -44,9 +45,9 @@ Celestia-node public API is documented [here](https://docs.celestia.org/develope
 
 ## Node types
 
-- **Bridge** nodes - relay blocks from the celestia consensus network to the celestia data availability (DA) network
-- **Full** nodes - fully reconstruct and store blocks by sampling the DA network for shares
-- **Light** nodes - verify the availability of block data by sampling the DA network for shares
+* **Bridge** nodes - relay blocks from the celestia consensus network to the celestia data availability (DA) network
+* **Full** nodes - fully reconstruct and store blocks by sampling the DA network for shares
+* **Light** nodes - verify the availability of block data by sampling the DA network for shares
 
 More information can be found [here](https://github.com/celestiaorg/celestia-node/blob/main/docs/adr/adr-003-march2022-testnet.md#legend).
 
@@ -64,9 +65,9 @@ celestia <node_type> start
 
 ## Package-specific documentation
 
-- [Header](./service/header/doc.go)
-- [Share](./service/share/doc.go)
-- [DAS](./das/doc.go)
+* [Header](./service/header/doc.go)
+* [Share](./service/share/doc.go)
+* [DAS](./das/doc.go)
 
 ## Code of Conduct
 

das/daser_test.go

@@ -273,7 +273,6 @@ func TestDASer_stopsAfter_BEFP(t *testing.T) {
 
 	// create fraud service and break one header
 	f := fraud.NewService(ps, mockGet.GetByHeight, ds)
-	require.NoError(t, f.RegisterUnmarshaler(fraud.BadEncoding, fraud.UnmarshalBEFP))
 	mockGet.headers[1] = header.CreateFraudExtHeader(t, mockGet.headers[1], bServ)
 
 	// create and start DASer

docs/adr/README.md

@@ -29,4 +29,4 @@ Note the context/background should be written in the present tense.
 
 To start a new ADR, you can use this template: [adr-template.md](./adr-template.md)
 
-### Table of Contents:
+## Table of Contents

docs/adr/adr-001-predevnet-celestia-node.md

@@ -18,7 +18,7 @@
 
 This ADR describes a basic pre-devnet design for a "Celestia Node" that was decided at the August 2021 Kyiv offsite that will ideally be completed in early November 2021 and tested in the first devnet.
 
-The goal of this design is to get a basic structure of "Celestia Node" interoperating with a "Celestia Core" consensus node by November 2021 (devnet). 
+The goal of this design is to get a basic structure of "Celestia Node" interoperating with a "Celestia Core" consensus node by November 2021 (devnet).
 
 After basic interoperability on devnet, there will be an effort to merge consensus functionality into the "Celestia Node" design as a modulor service that can be added on top of the basic functions of a "Celestia Node".
 
@@ -27,15 +27,16 @@ After basic interoperability on devnet, there will be an effort to merge consens
 A "Celestia Node" will be distinctly different than a "Celestia Core" node in the initial implementation for devnet, with plans to merge consensus core functionality into the general design of a "Celestia Node", just added as an additional `ConsensusService` on the node.
 
 For devnet, we require two modes for the Celestia Node: `light` and `full`, where the `light` node performs data availability sampling and the `full` node processes, stores, and serves new blocks from either Celestia Core consensus nodes *(required for devnet)* or from other Celestia `full` nodes *(optional for devnet)*.
- 
+
 **For devnet, a `light` Celestia Node must be able to do the following:**
+
 * propagate relevant block information (in the form of `ExtendedHeader`s and `BadEncodingFraudProof`s) to its "Celestia Node" peers
 * verify `ExtendedHeader`s
 * perform and serve sampling and `SharesByNamespace` requests *(note: light nodes serve the `Shares` that they've already requested and stored by default as a result of the way bitswap works -- with bitswap, if a node has something another node wants, it will serve it)*
 * request `State` to get `AccountBalance` in order to submit transactions
 
-
 **For devnet, a `full` Celestia Node must be able to do everything a `light` Celestia Node does, in addition to the following:**
+
 * receive "raw" (un-erasure coded) blocks from a "Celestia Core" node by subscribing to `NewBlockEvents` using the `/block` RPC endpoint of "Celestia Core" node
 * erasure code the block / verify erasure coding
 * create an `ExtendedHeader` with the raw block header, the generated `DataAvailabilityHeader` (DAH), as well as the `ValidatorSet` and serve this `ExtendedHeader` to the Celestia network
@@ -51,60 +52,61 @@ A user will be able to initialise a Celestia Node as either a `full` or `light`
 
 A `full` node encompasses the functionality of a `light` node along with additional services that allow it to interact with a Celestia Core node.
 
-A `light` node will provide the following services: 
+A `light` node will provide the following services:
+
 * `ExtendedHeaderService` -- a service that can be registered on the node, and started/stopped that contains every process related to retrieving `ExtendedHeader`s (actively or passively), as well as storing them.
-    * `ExtendedHeaderExchange` (request/response)
-    * `ExtendedHeaderSub` (broadcast)
-    * `ExtendedHeaderStore`
-* `FraudProofService` *(optional for devnet)* -- a service that can be registered on the node, and started/stopped that contains every process related to retrieving `BadEncodingFraudProof`s and `StateFraudProof`s. 
-    * `FraudProofSub` (broadcast)
-    * `FraudProofStore`
+  * `ExtendedHeaderExchange` (request/response)
+  * `ExtendedHeaderSub` (broadcast)
+  * `ExtendedHeaderStore`
+* `FraudProofService` *(optional for devnet)* -- a service that can be registered on the node, and started/stopped that contains every process related to retrieving `BadEncodingFraudProof`s and `StateFraudProof`s.
+  * `FraudProofSub` (broadcast)
+  * `FraudProofStore`
 * `ShareService` -- a service that can be registered on the node, and started/stopped that contains every process related to retrieving shares randomly (sampling) or by namespace from the network, as well as storage for those shares.
-    * `ShareExchange` (request/response)
-    * `ShareStore`
+  * `ShareExchange` (request/response)
+  * `ShareStore`
 * `StateService` *(optional for devnet)* -- a service that can be registered on the node, and started/stopped that contains every process related to retrieving state for a given block height or account.
-    * `StateExchange` (request/response)
+  * `StateExchange` (request/response)
 * `TransactionService` *(dependent on `StateService` implementation, but optional for devnet)* -- a simple server that can be registered on the node, and started and stopped that handles for endpoints like `/submit_tx`
-    * `SubmitTx` (request/response)
+  * `SubmitTx` (request/response)
+
+A `full` node will provide the following services:
 
-A `full ` node will provide the following services: 
 * `ExtendedHeaderService` -- a service that can be registered on the node, and started/stopped that contains every process related to **generating**, propagating/retrieving `ExtendedHeader`s (actively or passively), as well as storing them.
-    * `ExtendedHeaderExchange` (request/response)
-    * `ExtendedHeaderVerification` (`full` nodes only)
-    * `ExtendedHeaderSub` (broadcast)
-    * `ExtendedHeaderStore`
+  * `ExtendedHeaderExchange` (request/response)
+  * `ExtendedHeaderVerification` (`full` nodes only)
+  * `ExtendedHeaderSub` (broadcast)
+  * `ExtendedHeaderStore`
 * `FraudProofService` *(optional for devnet)* -- a service that can be registered on the node, and started/stopped that contains every process related to generating, broadcasting, and storing both `BadEncodingFraudProof`s and `StateFraudProof`s.
-    * **`FraudProofGeneration`**
-    * `FraudProofSub` (broadcast)
-    * `FraudProofStore`
+  * **`FraudProofGeneration`**
+  * `FraudProofSub` (broadcast)
+  * `FraudProofStore`
 * `ShareService` -- a service that can be registered on the node, and started/stopped that contains every process related to requesting and providing shares. Note, `full` nodes will not have a separate `ShareStore` as the store the full blocks.
-    * `ShareExchange` (request/response)
+  * `ShareExchange` (request/response)
 * `BlockService`
-    * `BlockErasureCoding` 
-    * `NewBlockEventSubscription` (`full` node <> `Celestia Core` node request)
-    * `BlockExchange` *(optional for devnet)* (`full` node <> `full` node request/response) 
-    * `BlockStore`
+  * `BlockErasureCoding`
+  * `NewBlockEventSubscription` (`full` node <> `Celestia Core` node request)
+  * `BlockExchange` *(optional for devnet)* (`full` node <> `full` node request/response)
+  * `BlockStore`
 * `StateService` *(optional for devnet)*
-    * `StateExchange` (`full` node <> Celestia Core request/response)
+  * `StateExchange` (`full` node <> Celestia Core request/response)
 * `TransactionService` *(dependent on `StateService` implementation, but optional for devnet)*
-    * `SubmitTx`
-
+  * `SubmitTx`
 
-For devnet, it should be possible for Celestia `full` Nodes to receive information directly from Celestia Core nodes or from each other. 
+For devnet, it should be possible for Celestia `full` Nodes to receive information directly from Celestia Core nodes or from each other.
 
-## Considerations 
+## Considerations
 
 ### State Fraud Proofs
+
 For the Celestia Node to be able to propagate `StateFraudProof`s, we must modify Celestia Core to store blocks with invalid state and serve them to both the Celestia Node and the Celestia App, **and** the Celestia App must be able to generate and serve `StateFraudProof`s via RPC to Celestia nodes.
 
 This feature is not necessarily required for devnet (so state exection functionality for Celestia Full Nodes can be stubbed out), but it would be nice to have for devnet as we will likely allow Celestia Full Nodes to speak with other Celestia Full Nodes instead of running a trusted Celestia Core node simultaenously and relying on it for information.
 
-A roadmap to implementation could look like the following: 
+A roadmap to implementation could look like the following:
 
 The Celestia Full Node would adhere to the ABCI interface in order to communicate with the Celestia App (similar to the way Optimint does it). The Celestia Full Node would send State requests to the Celestia App in order for the Celestia app to replay the transactions in the block and verify the state. Alternatively, Celestia Full Nodes would also be able to replay the transactions in order to verify state / generate state fraud proofs on its own.
 
-For devnet, it is okay to stub out state verification functionality. For example, a Celestia Full Node would download reserve transactions, but not replay them. 
-
+For devnet, it is okay to stub out state verification functionality. For example, a Celestia Full Node would download reserve transactions, but not replay them.
 
 ### Fraud Proofs
 
@@ -116,33 +118,31 @@ For devnet, the Celestia Node will not be able to generate state fraud proofs as
 
 At the moment, we will be using [bitswap](https://github.com/ipfs/go-bitswap) to retrieve samples and shares from the network. The way Bitswap works requires nodes that have the requested data to serve it. This is not necessarily ideal for a "light node" to do as supporting serving samples/shares would expand the resource requirements for a light node.
 
-Other negatives of bitswap include: 
+Other negatives of bitswap include:
+
 * it somewhat couples storage with networking (e.g. it will be difficult to just compute the inner proof nodes on demand instead of storing them) by default
 * it requires multiple roundtrips for what could be a single (or at least fewer) roundtrip(s) if we wrote our own protocol; this is particularly relevant in the case where Celestia nodes will run on their own and download the whole block via the p2p network from other Celestia nodes (instead of from tendermint via RPC): the [current implementation](https://github.com/celestiaorg/celestia-core/blob/052d1269e0ec1de029e1cf3fc02d2585d7f9df10/p2p/ipld/read.go#L23-L30) using bitswap and ipfs is quite inefficient compared to a protocol that was not bitswap on a share-level
 * more code we do not directly have control over and more dependencies
 
-In the future, we should consider moving away from bitswap to either [GraphSync](https://github.com/ipfs/go-graphsync) or a custom protocol. 
+In the future, we should consider moving away from bitswap to either [GraphSync](https://github.com/ipfs/go-graphsync) or a custom protocol.
 
 ## Consequences
 
-While this design is not ideal, it will get us to a devnet more quickly and allow us to iterate rather than try to design and implement the perfect Celestia node from the start. 
+While this design is not ideal, it will get us to a devnet more quickly and allow us to iterate rather than try to design and implement the perfect Celestia node from the start.
 
-**Positive**
+### Positive
 
 Iterative process will allow us to test out non-p2p-related functionality much sooner, and will allow us to incrementally approach debugging rather than trying to get the design/implementation perfect in one go.
 
-**Negative**
+### Negative
 
-* We will end up throwing out a lot of the implementation work we do now since our eventual goal is to merge consensus functionality into the concept of a "Celestia Node". 
+* We will end up throwing out a lot of the implementation work we do now since our eventual goal is to merge consensus functionality into the concept of a "Celestia Node".
 * The current design requires erasure coding to be done twice and stores data twice (raw block in Celestia Core and erasure coded block in Celestia Node) which is redundant and should be consolidated in the future.
 
-
 ## Open Questions
 
 Should Celestia Core nodes also generate and serve fraud proofs? Or only serve the invalid blocks to Celestia Full Nodes?
 
 ## Status
 
 Proposed
-
-