Skip to content

Commit

Permalink
Update EIP-7495: simplify design, renaming, formatting
Browse files Browse the repository at this point in the history 
Merged by EIP-Bot.
  • Loading branch information
@wemeetagain
wemeetagain authored May 16, 2024
1 parent e95cfaa commit d43a39d
Show file tree
Hide file tree
Showing 3 changed files with 80 additions and 126 deletions.
140 changes: 47 additions & 93 deletions EIPS/eip-7495.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,62 +12,64 @@ created: 2023-08-18

## Abstract

This EIP introduces a new [Simple Serialize (SSZ) type](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md) to represent `StableContainer[N]` values. A `StableContainer[N]` extends an SSZ `Container` with stable serialization and merkleization even when individual fields become optional or new fields are introduced in the future.
This EIP introduces two new [Simple Serialize (SSZ) types](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md) to enable forward-compatible containers.

Furthermore, `MerkleizeAs[B]` is introduced to support specialized sub-types of `Container` and `StableContainer[N]` while retaining the merkleization of the base type. This is useful, e.g., for fork-specific data structures that only use a subset of the base fields. Verifiers of Merkle proofs for these data structures will not break on new forks.
A `StableContainer[N]` extends an SSZ `Container` with stable merkleization and forward-compatible serialization even when individual fields are deprecated or new fields are introduced in the future.

Furthermore, `Profile[B]` is introduced to support specialized sub-types of `StableContainer[N]` while retaining the merkleization of the base type. This is useful, e.g., for fork-specific data structures that only use a subset of the base fields and/or when required base fields are known. Verifiers of Merkle proofs for these data structures will not break on new forks.

## Motivation

Stable containers are currently not representable in SSZ. Adding support provides these benefits:
Stable containers and profiles are currently not representable in SSZ. Adding support provides these benefits:

1. **Stable signatures:** Signing roots derived from a `StableContainer[N]` never change. In the context of Ethereum, this is useful for transaction signatures that are expected to remain valid even when future updates introduce additional transaction fields. Likewise, the overall transaction root remains stable and can be used as a perpetual transaction ID.

2. **Stable merkle proofs:** Merkle proof verifiers that check specific fields of a `StableContainer[N]` do not need continuous updating when future updates introduce additional fields. Common fields always merkleize at the same [generalized indices](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/merkle-proofs.md).
2. **Stable merkle proofs:** Merkle proof verifiers that check specific fields of a `StableContainer[N]` or `Profile[B]` do not need continuous updating when future updates introduce additional fields. Common fields always merkleize at the same [generalized indices](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/merkle-proofs.md).

3. **Optional fields:** Current SSZ formats do not support optional fields, prompting designs to use zero values instead. With `StableContainer[N]`, the SSZ serialization is compact; inactive fields do not consume space.
3. **Optional fields:** Current SSZ formats do not support optional fields, prompting designs to use zero values instead. With `StableContainer[N]` and `Profile[B]`, the SSZ serialization is compact; inactive fields do not consume space.

## Specification

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 and RFC 8174.

### Type definition
**Note:** In this document, `Optional[T]` exclusively refers to Python's `typing.Optional`. Specifically, `Optional[T]` is NOT an SSZ type itself!

### `StableContainer[N]`

Similar to the regular [SSZ `Container`](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#composite-types), `StableContainer[N]` defines an ordered heterogeneous collection of fields. `N` indicates the potential maximum number of fields to which it can ever grow in the future. `N` MUST be `> 0`.

As part of a `StableContainer[N]`, fields of type `Optional[T]` MAY be defined. Such fields can either represent a present value of SSZ type `T`, or indicate absence of a value (indicated by `None`). The [default value](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#default-values) of an `Optional[T]` is `None`.
All fields of a `StableContainer[N]` MUST be of of type `Optional[T]`. Such fields can either represent a present value of SSZ type `T`, or indicate absence of a value (indicated by `None`). The [default value](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#default-values) of an `Optional[T]` is `None`.

```python
class Example(StableContainer[32]):
a: uint64
a: Optional[uint64]
b: Optional[uint32]
c: uint16
c: Optional[uint16]
```

For the purpose of serialization, `StableContainer[N]` is always considered ["variable-size"](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#variable-size-and-fixed-size) regardless of the individual field types.

### Stability guarantees
#### Stability guarantees

The serialization and merkleization of a `StableContainer[N]` remains stable as long as:

- The maximum capacity `N` does not change
- The order of fields does not change
- New fields are always added to the end
- Required fields remain required `T`, or become an `Optional[T]`
- Optional fields remain `Optional[T]`, or become a required `T`

When an optional field becomes required, existing messages still have stable serialization and merkleization, but will be rejected on deserialization if not present.
- New fields are always appended to the end
- All fields have immutable SSZ schemas, or recursively adopt `StableContainer[N]`
- `List`/`Bitlist` capacities do not change; shortening is possible via application logic

### JSON serialization
#### JSON serialization

JSON serialization follows the [canonical JSON mapping](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#json-mapping) of SSZ `Container`.

Fields of type `Optional[T]` with a `None` value SHALL be omitted when serializing to JSON.

### Binary serialization
#### Binary serialization

Serialization of `StableContainer[N]` is defined similarly to the [existing logic](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#vectors-containers-lists) for `Container`. Notable changes are:

- A [`Bitvector[N]`](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#composite-types) is constructed, indicating active fields within the `StableContainer[N]`. For required fields `T` and optional fields `Optional[T]` with a present value (not `None`), a `True` bit is included. For optional fields `Optional[T]` with a `None` value, a `False` bit is included. The `Bitvector[N]` is padded with `False` bits up through length `N`
- A [`Bitvector[N]`](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#composite-types) is constructed, indicating active fields within the `StableContainer[N]`. For fields with a present value (not `None`), a `True` bit is included. For fields with a `None` value, a `False` bit is included. The `Bitvector[N]` is padded with `False` bits up through length `N`
- Only active fields are serialized, i.e., fields with a corresponding `True` bit in the `Bitvector[N]`
- The serialization of the `Bitvector[N]` is prepended to the serialized active fields
- If variable-length fields are serialized, their offsets are relative to the start of serialized active fields, after the `Bitvector[N]`
Expand Down Expand Up @@ -98,17 +100,15 @@ fixed_parts = [part if part != None else variable_offsets[i] for i, part in enum
return serialize(active_fields) + b"".join(fixed_parts + variable_parts)
```

### Deserialization
#### Deserialization

Deserialization of a `StableContainer[N]` starts by deserializing a `Bitvector[N]`. That value MUST be validated:

- For each required field, the corresponding bit in the `Bitvector[N]` MUST be `True`
- For each optional field, the corresponding bit in the `Bitvector[N]` is not restricted
- All extra bits in the `Bitvector[N]` that exceed the number of fields MUST be `False`

The rest of the data is [deserialized](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#deserialization) same as a regular [SSZ `Container`](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#vectors-containers-lists), consulting the `Bitvector[N]` to determine what optional fields are present in the data. Absent fields are skipped during deserialization and assigned `None` values.
The rest of the data is [deserialized](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#deserialization) same as a regular [SSZ `Container`](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#vectors-containers-lists), consulting the `Bitvector[N]` to determine which fields are present in the data. Absent fields are skipped during deserialization and assigned `None` values.

### Merkleization
#### Merkleization

The [merkleization specification](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#merkleization) is extended with the following helper functions:

Expand All @@ -122,55 +122,45 @@ Merkleization `hash_tree_root(value)` of an object `value` is extended with:

- `mix_in_aux(merkleize(([hash_tree_root(element) if is_active_field(element) else Bytes32() for element in value.data] + [Bytes32()] * N)[:N]), hash_tree_root(value.active_fields))` if `value` is a `StableContainer[N]`.

### `MerkleizeAs[B]`
### `Profile[B]`

`MerkleizeAs[B]` can be applied to a more specialized sub-type of a base type `B` with SSZ type `Container` or `StableContainer[N]`. The following specializations are allowed:
`Profile[B]` also defines an ordered heterogeneous collection of fields, a subset of fields of a base `StableContainer[N]` `B`.

- Fields in `MerkleizeAs[B]` correspond to a field with the same field name in `B`, but may appear in a different order; this only affects serialization, the canonical order in `B` is always used for merkleization
- For fields of type `T` in the base type `B`, the corresponding fields in `MerkleizeAs[B]` may have type `T` or `MerkleizeAs[T]`; All fields that are required in `B` must be present in `MerkleizeAs[B]`
The rules for construction are:

If base type `B` is a `StableContainer[N]`, additionally, the following specializations are allowed:
- Fields in `Profile[B]` correspond to fields with the same field name in `B`.
- Fields in `Profile[B]` must be in the same order as in `B`
- Fields in the base type `B` MAY be kept `Optional` in `Profile[B]`
- Fields in the base type `B` MAY be required in `Profile[B]` by unwrapping them from `Optional`
- Fields in the base type `B` MAY be omitted in `Profile[B]`, disallowing their presence in the sub-type
- Fields in the base type `B` of type `Optional[T]` with `T` being a nested `StableContainer` MAY have types `Optional[Profile[T]]` (if kept optional), or `Profile[T]` (if it is required)

- For fields of type `Optional[T]` in the base type `B`, the corresponding fields in `MerkleizeAs[B]` may have type `T` / `MerkleizeAs[T]` (to make them required in the sub-type), or `Optional[T]` / `Optional[MerkleizeAs[T]]` (to keep them optional in the sub-type)
- Fields of type `Optional[T]` in the base type `B` may be absent in `MerkleizeAs[B]` (to disallow their presence in the sub-type)
#### Serialization

Serialization of `MerkleizeAs[B]` is based on its own field order, disregarding the order of base type `B`. If base type `B` is a `StableContainer[N]`, the leading `Bitvector` is replaced by a sparse reprsentation that only includes information about optional fields, and also follows the field order in `MerkleizeAs[B]` instead of the order within the base type `B`. Bits for required fields of `MerkleizeAs[B]` as well as the zero-padding to capacity `N` are not included. If there are no optional fields in `MerkleizeAs[B]`, no `Bitvector` is omitted.
Serialization of `Profile[B]` is similar to the one of its base `StableContainer[N]`, except that the leading `Bitvector` is replaced by a sparse representation that only includes information about fields that are optional in `Profile[B]`. Bits for required fields of `Profile[B]` as well as the zero-padding to capacity `N` are not included. If there are no optional fields in `Profile[B]`, the `Bitvector` is omitted.

Merkleization of `MerkleizeAs[B]` always follows the merkleization of base type `B`.
`Profile[B]` is considered ["variable-size"](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#variable-size-and-fixed-size) iff it contains any `Optional[T]` or any "variable-size" fields.

`MerkleizeAs[B]` is considered ["variable-size"](https://github.com/ethereum/consensus-specs/blob/4afe39822c9ad9747e0f5635cca117c18441ec1b/ssz/simple-serialize.md#variable-size-and-fixed-size) iff it contains any `Optional[T]` or any "variable-size" fields.
#### Merkleization

Merkleization of `Profile[B]` always follows the merkleization of base type `B`.

```python
# Defines the common merkleization format and a portable serialization format
class Shape(StableContainer[4]):
side: Optional[uint16]
color: uint8
color: Optional[uint8]
radius: Optional[uint16]

# Inherits merkleization format from `Shape`, but is serialized more compactly
class Square(MerkleizeAs[Shape]):
class Square(Profile[Shape]):
side: uint16
color: uint8

# Inherits merkleization format from `Shape`, but is serialized more compactly
class Circle(MerkleizeAs[Shape]):
radius: uint16
class Circle(Profile[Shape]):
color: uint8

# Defines a container with immutable scheme that contains two `StableContainer`
class ShapePair(Container):
shape_1: Shape
shape_2: Shape

# Inherits merkleization format from `ShapePair`, and serializes more compactly
class SquarePair(MerkleizeAs[ShapePair]):
shape_1: Square
shape_2: Square

# Inherits merkleization format from `ShapePair`, and reorders fields
class CirclePair(MerkleizeAs[ShapePair]):
shape_2: Circle
shape_1: Circle
radius: uint16
```

Serialization examples:
Expand All @@ -187,47 +177,11 @@ Serialization examples:

```Shape(side=None, color=1, radius=0x42)```

- `420001`
- `014200`

```Circle(radius=0x42, color=1)```

- `08000000 0c000000 03420001 03690001`

```
ShapePair(
shape_1=Shape(side=0x42, color=1, radius=None),
shape_2=Shape(side=0x69, color=1, radius=None),
)
```
- `420001 690001`
```
SquarePair(
shape_1=Square(side=0x42, color=1),
shape_2=Square(side=0x69, color=1),
)
```
- `08000000 0c000000 06014200 06016900`
```
ShapePair(
shape_1=Shape(side=None, color=1, radius=0x42),
shape_2=Shape(side=None, color=1, radius=0x69),
)
```
- `690001 420001`
```
CirclePair(
shape_1=Circle(radius=0x42, color=1),
shape_2=Circle(radius=0x69, color=1),
)
```
While this serialization of `MerkleizeAs[B]` is more compact, note that it is not forward compatible and that context information that determines the underlying data type has to be indicated out of bands. If forward compatibility is required, `MerkleizeAs[B]` SHALL be converted to its base type `B` and subsequently serialized according to `B`.
While this serialization of `Profile[B]` is more compact, note that it is not forward-compatible and that context information that determines the underlying data type has to be indicated out of bands. If forward-compatibility is required, `Profile[B]` SHALL be converted to its base type `B` and subsequently serialized according to `B`.

## Rationale

Expand All @@ -237,9 +191,9 @@ Current SSZ types are only stable within one version of a specification, i.e., o

To avoid restricting design space, the scheme has to support extension with new fields, obsolescence of old fields, and new combinations of existing fields. When such adjustments occur, old messages must still deserialize correctly and must retain their original Merkle root.

### What are the problems solved by `MerkleizeAs[B]`?
### What are the problems solved by `Profile[B]`?

The forward compatible merkleization of `StableContainer` may be desirable even in situations where only a single sub-type is valid at any given time, e.g., as determined by the fork schedule. In such situations, message size can be reduced and type safety increased by exchanging `MerkleizeAs[B]` instead of the underlying base type. This can be useful, e.g., for consensus data structures such as `BeaconState`, to ensure that Merkle proofs for its fields remain compatible across forks.
The forward-compatible merkleization of `StableContainer` may be desirable even in situations where only a single sub-type is valid at any given time, e.g., as determined by the fork schedule. In such situations, message size can be reduced and type safety increased by exchanging `Profile[B]` instead of the underlying base type. This can be useful, e.g., for consensus data structures such as `BeaconState`, to ensure that Merkle proofs for its fields remain compatible across forks.

### Why not `Union[T, U, V]`?

Expand All @@ -259,7 +213,7 @@ Additionally, every time that the number of fields reaches a new power of 2, the

## Backwards Compatibility

`StableContainer[N]` and `MerkleizeAs[B]` are new SSZ types and do not conflict with other SSZ types currently in use.
`StableContainer[N]` and `Profile[B]` are new SSZ types and do not conflict with other SSZ types currently in use.

## Test Cases

Expand Down
Loading

0 comments on commit d43a39d

Please sign in to comment.