update networking spec with aggregate proposer/builder types, update …

…TOCs

specs/sharding/beacon-chain.md

@@ -17,7 +17,7 @@
   - [Shard Work Status](#shard-work-status)
 - [Preset](#preset)
   - [Misc](#misc-1)
-  - [Shard block samples](#shard-block-samples)
+  - [Shard blob samples](#shard-blob-samples)
   - [Precomputed size verification points](#precomputed-size-verification-points)
   - [Gwei values](#gwei-values)
 - [Configuration](#configuration)

specs/sharding/p2p-interface.md

@@ -13,10 +13,11 @@
   - [Misc](#misc)
 - [Gossip domain](#gossip-domain)
   - [Topics and messages](#topics-and-messages)
-    - [Shard block subnets](#shard-block-subnets)
-      - [`shard_block_{subnet_id}`](#shard_block_subnet_id)
+    - [Shard blob subnets](#shard-blob-subnets)
+      - [`shard_blob_{subnet_id}`](#shard_blob_subnet_id)
     - [Global topics](#global-topics)
-      - [`shard_block_header`](#shard_block_header)
+      - [`shard_blob_header`](#shard_blob_header)
+      - [`shard_blob_tx`](#shard_blob_tx)
       - [`shard_proposer_slashing`](#shard_proposer_slashing)
 
 <!-- END doctoc generated TOC please keep comment here to allow auto update -->
@@ -34,32 +35,33 @@ The adjustments and additions for Shards are outlined in this document.
 
 | Name | Value | Description |
 | ---- | ----- | ----------- |
-| `SHARD_BLOCK_SUBNET_COUNT` | `64` | The number of `shard_block_{subnet_id}` subnets used in the gossipsub protocol. |
+| `SHARD_BLOB_SUBNET_COUNT` | `64` | The number of `shard_blob_{subnet_id}` subnets used in the gossipsub protocol. |
 
 ## Gossip domain
 
 ### Topics and messages
 
 Following the same scheme as the [Phase0 gossip topics](../phase0/p2p-interface.md#topics-and-messages), names and payload types are:
 
-| Name                             | Message Type              |
-|----------------------------------|---------------------------|
-| `shard_block_{subnet_id}`        | `SignedShardBlock`        |
-| `shard_block_header`             | `SignedShardBlockHeader`  |
-| `shard_proposer_slashing`        | `ShardProposerSlashing`   |
+| Name                            | Message Type             |
+|---------------------------------|--------------------------|
+| `shard_blob_{subnet_id}`        | `SignedShardBlob`        |
+| `shard_blob_header`             | `SignedShardBlobHeader`  |
+| `shard_blob_tx`                 | `SignedShardBlobHeader`  |
+| `shard_proposer_slashing`       | `ShardProposerSlashing`  |
 
 The [DAS network specification](./das-p2p.md) defines additional topics.
 
-#### Shard block subnets
+#### Shard blob subnets
 
-Shard block subnets are used by builders to make their blobs available after selection by shard proposers.
+Shard blob subnets are used by builders to make their blobs available after selection by shard proposers.
 
-##### `shard_block_{subnet_id}`
+##### `shard_blob_{subnet_id}`
 
-Shard block data, in the form of a `SignedShardBlock` is published to the `shard_block_{subnet_id}` subnets.
+Shard blob data, in the form of a `SignedShardBlob` is published to the `shard_blob_{subnet_id}` subnets.
 
 ```python
-def compute_subnet_for_shard_block(state: BeaconState, slot: Slot, shard: Shard) -> uint64:
+def compute_subnet_for_shard_blob(state: BeaconState, slot: Slot, shard: Shard) -> uint64:
     """
     Compute the correct subnet for a shard blob publication.
     Note, this mimics compute_subnet_for_attestation().
@@ -69,82 +71,104 @@ def compute_subnet_for_shard_block(state: BeaconState, slot: Slot, shard: Shard)
     slots_since_epoch_start = Slot(slot % SLOTS_PER_EPOCH)
     committees_since_epoch_start = committees_per_slot * slots_since_epoch_start
 
-    return uint64((committees_since_epoch_start + committee_index) % SHARD_BLOCK_SUBNET_COUNT)
+    return uint64((committees_since_epoch_start + committee_index) % SHARD_BLOB_SUBNET_COUNT)
 ```
 
-The following validations MUST pass before forwarding the `signed_block` on the horizontal subnet or creating samples for it.
+The following validations MUST pass before forwarding the `signed_blob`,
+on the horizontal subnet or creating samples for it. Alias `blob = signed_blob.message`.
 
-We define some aliases to the nested contents of `signed_block`:
-```python
-block: ShardBlock = signed_block.message
-signed_blob: SignedShardBlob = block.signed_blob
-blob: ShardBlob = signed_blob.message
-```
-
-- _[IGNORE]_ The `block` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
+- _[IGNORE]_ The `blob` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
   i.e. validate that `blob.slot <= current_slot`
   (a client MAY queue future blobs for processing at the appropriate slot).
-- _[IGNORE]_ The `blob` is new enough to be still be processed --
+- _[IGNORE]_ The `blob` is new enough to still be processed --
   i.e. validate that `compute_epoch_at_slot(blob.slot) >= get_previous_epoch(state)`
-- _[REJECT]_ The shard should have a committee at slot --
+- _[REJECT]_ The shard blob is for an active shard --
+  i.e. `blob.shard < get_active_shard_count(state, compute_epoch_at_slot(blob.slot))`
+- _[REJECT]_ The `blob.shard` MUST have a committee at the `blob.slot` --
   i.e. validate that `compute_committee_index_from_shard(state, blob.slot, blob.shard)` doesn't raise an error
 - _[REJECT]_ The shard blob is for the correct subnet --
-  i.e. `compute_subnet_for_shard_block(state, blob.slot, blob.shard) == subnet_id`
-- _[IGNORE]_ The block is the first block with valid signature received for the `(block.proposer_index, blob.slot, blob.shard)` combination.
+  i.e. `compute_subnet_for_shard_blob(state, blob.slot, blob.shard) == subnet_id`
+- _[IGNORE]_ The blob is the first blob with valid signature received for the `(blob.proposer_index, blob.slot, blob.shard)` combination.
 - _[REJECT]_ The blob is not too large, the data MUST NOT be larger than the SSZ list-limit, and a client MAY be more strict.
 - _[REJECT]_ The `blob.body.data` MUST NOT contain any point `p >= MODULUS`. Although it is a `uint256`, not the full 256 bit range is valid.
-- _[REJECT]_ The block proposer signature, `signed_block.signature`, is valid with respect to the `proposer_index` pubkey.
 - _[REJECT]_ The blob builder exists and has sufficient balance to back the fee payment.
-- _[REJECT]_ The blob builder signature, `signed_blob.signature`, is valid with respect to the `builder_index` pubkey.
-- _[REJECT]_ The block is proposed by the expected `proposer_index` for the block's slot
+- _[REJECT]_ The blob signature is valid for the aggregate of proposer and builder, `signed_blob.signature`,
+  i.e. `bls.FastAggregateVerify([builder_pubkey, proposer_pubkey], blob_signing_root, signed_blob.signature)`.
+- _[REJECT]_ The blob is proposed by the expected `proposer_index` for the blob's `slot` and `shard`,
   in the context of the current shuffling (defined by `blob.body.beacon_block_root`/`slot`).
   If the `proposer_index` cannot immediately be verified against the expected shuffling,
-  the block MAY be queued for later processing while proposers for the block's branch are calculated --
+  the blob MAY be queued for later processing while proposers for the blob's branch are calculated --
   in such a case _do not_ `REJECT`, instead `IGNORE` this message.
 
 #### Global topics
 
-There are two additional global topics for Sharding.
-
-One is used to propagate shard block headers (`shard_block_header`) to all nodes on the network.
-Another one is used to propagate shard proposer slashings (`shard_proposer_slashing`).
+There are three additional global topics for Sharding.
 
-##### `shard_block_header`
+- `shard_blob_header`: co-signed headers, to be included on-chain, and signaling builders to publish full data.
+- `shard_blob_tx`: builder-signed headers, also known as "data transaction".
+- `shard_proposer_slashing`: slashings of duplicate shard proposals
 
-Shard header data, in the form of a `SignedShardBlockHeader` is published to the global `shard_block_header` subnet.
-Shard block headers select shard blob bids by builders, and should be timely to ensure builders can publish the full shard block timely.
+##### `shard_blob_header`
 
-The following validations MUST pass before forwarding the `signed_block_header` (with inner `message` as `header`) on the network.
+Shard header data, in the form of a `SignedShardBlobHeader` is published to the global `shard_blob_header` subnet.
+Shard blob headers select shard blob bids by builders,
+and should be timely to ensure builders can publish the full shard blob before subsequent attestations.
 
-We define some aliases to the nested contents of `signed_block_header`:
-```python
-block_header: ShardBlockHeader = signed_block_header.message
-signed_blob_header: SignedShardBlobHeader = header.signed_blob_header
-blob_header: ShardBlobHeader = signed_blob_header.message
-```
+The following validations MUST pass before forwarding the `signed_blob_header` on the network. Alias `header = signed_blob_header.message`
 
-- _[IGNORE]_ The header is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
-  i.e. validate that `blob_header.slot <= current_slot`
+- _[IGNORE]_ The `header` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
+  i.e. validate that `header.slot <= current_slot`
   (a client MAY queue future headers for processing at the appropriate slot).
-- _[IGNORE]_ The header is new enough to be still be processed --
-  i.e. validate that `compute_epoch_at_slot(blob_header.slot) >= get_previous_epoch(state)`
-- _[IGNORE]_ The header is the first header with valid signature received for the `(block_header.proposer_index, blob_header.slot, blob_header.shard)` combination.
-- _[REJECT]_ The `shard` MUST have a committee at the `slot` --
-  i.e. validate that `compute_committee_index_from_shard(state, blob_header.slot, blob_header.shard)` doesn't raise an error
-- _[REJECT]_ The proposer signature, `signed_shard_block_header.signature`, is valid with respect to the `block_header.proposer_index` pubkey.
-- _[REJECT]_ The header is proposed by the expected `proposer_index` for the block's slot
+- _[IGNORE]_ The header is new enough to still be processed --
+  i.e. validate that `compute_epoch_at_slot(header.slot) >= get_previous_epoch(state)`
+- _[REJECT]_ The shard header is for an active shard --
+  i.e. `header.shard < get_active_shard_count(state, compute_epoch_at_slot(header.slot))`
+- _[REJECT]_ The `header.shard` MUST have a committee at the `header.slot` --
+  i.e. validate that `compute_committee_index_from_shard(state, header.slot, header.shard)` doesn't raise an error
+- _[IGNORE]_ The header is the first header with valid signature received for the `(header.proposer_index, header.slot, header.shard)` combination.
+- _[REJECT]_ The blob builder exists and has sufficient balance to back the fee payment.
+- _[REJECT]_ The header signature is valid for the aggregate of proposer and builder, `signed_blob_header.signature`,
+  i.e. `bls.FastAggregateVerify([builder_pubkey, proposer_pubkey], blob_signing_root, signed_blob_header.signature)`.
+- _[REJECT]_ The header is proposed by the expected `proposer_index` for the blob's `header.slot` and `header.shard`
   in the context of the current shuffling (defined by `header.body_summary.beacon_block_root`/`slot`).
   If the `proposer_index` cannot immediately be verified against the expected shuffling,
-  the block MAY be queued for later processing while proposers for the block's branch are calculated --
+  the blob MAY be queued for later processing while proposers for the blob's branch are calculated --
   in such a case _do not_ `REJECT`, instead `IGNORE` this message.
 
+##### `shard_blob_tx`
+
+Shard data-transactions, in the form of a `SignedShardBlobHeader` is published to the global `shard_blob_tx` subnet.
+These shard blob headers are signed solely by the blob-builder.
+
+The following validations MUST pass before forwarding the `signed_blob_header` on the network. Alias `header = signed_blob_header.message`
+
+- _[IGNORE]_ The `header` is not from a future slot (with a `MAXIMUM_GOSSIP_CLOCK_DISPARITY` allowance) --
+  i.e. validate that `header.slot <= current_slot`
+  (a client MAY queue future headers for processing at the appropriate slot).
+- _[IGNORE]_ The header is new enough to still be processed --
+  i.e. validate that `compute_epoch_at_slot(header.slot) >= get_previous_epoch(state)`
+- _[REJECT]_ The shard header is for an active shard --
+  i.e. `header.shard < get_active_shard_count(state, compute_epoch_at_slot(header.slot))`
+- _[REJECT]_ The `header.shard` MUST have a committee at the `header.slot` --
+  i.e. validate that `compute_committee_index_from_shard(state, header.slot, header.shard)` doesn't raise an error
+- _[IGNORE]_ The header is the first header with valid signature received for the `(header.builder_index, header.slot, header.shard)` combination.
+- _[REJECT]_ The blob builder exists and has sufficient balance to back the fee payment.
+- _[IGNORE]_ The header fee SHOULD be higher than previously seen headers for `(header.slot, header.shard)`, from any builder.
+  Propagating nodes MAY increase fee increments in case of spam.
+- _[REJECT]_ The header signature is valid for ONLY the builder, `signed_blob_header.signature`,
+  i.e. `bls.Verify(builder_pubkey, blob_signing_root, signed_blob_header.signature)`. The signature is not an aggregate with the proposer.
+- _[REJECT]_ The header is designated for proposal by the expected `proposer_index` for the blob's `header.slot` and `header.shard`
+  in the context of the current shuffling (defined by `header.body_summary.beacon_block_root`/`slot`).
+  If the `proposer_index` cannot immediately be verified against the expected shuffling,
+  the blob MAY be queued for later processing while proposers for the blob's branch are calculated --
+  in such a case _do not_ `REJECT`, instead `IGNORE` this message.
 
 ##### `shard_proposer_slashing`
 
 Shard proposer slashings, in the form of `ShardProposerSlashing`, are published to the global `shard_proposer_slashing` topic.
 
 The following validations MUST pass before forwarding the `shard_proposer_slashing` on to the network.
 - _[IGNORE]_ The shard proposer slashing is the first valid shard proposer slashing received
-  for the proposer with index `proposer_slashing.signed_reference_1.message.proposer_index`.
-  The `slot` and `shard` are ignored, there are no per-shard slashings.
+  for the proposer with index `proposer_slashing.proposer_index`.
+  The `proposer_slashing.slot` and `proposer_slashing.shard` are ignored, there are no repeated or per-shard slashings.
 - _[REJECT]_ All of the conditions within `process_shard_proposer_slashing` pass validation.