Initial design to reduce metadata overhead

[dryajov]

This document proposes a change to reduce metadata overhead in manifest files as well as consolidating the slot blocks handling with the network block handling.

[benbierens]

I like this plan. I have read the doc briefly and I'll read it again some more shortly, but I have a question for you:
Is it required that Merkle trees are binary, aka max 2 elements per node?
I understand how the leaf index as binary gives you the path from root to leaf, and how the hashes along the way are useful for inclusion proofs. But I was thinking... what if we allow 4 elements per node? the leaf index as binary would still represent the path from root to leaf except in chunks of 2 bits instead of 1. The same holds true for 3 bits and 8 elements per node, etc. An approach like this might greatly reduce the number of levels to our trees for large datasets, and I don't see a downside to this. Your thoughts please! :D

[tbekas]

@benbierens I think that 2 elements per node is optimal when it comes to the size of the proofs. Think about the extreme case when you have N leaves and N elements per node. The size of the proof for each of the N leaves will be O(N).

@dryajov I like the Idea. I'm not convinced yet that we need to store the entire tree. I think we could (should) store only a merkle proof next to a block and that should suffice for all the use cases in my opinion.

It will take only 2 times more on the disk, but I think it would reduce the complexity of the flows and possibly increase the performance of disk operations (sequential reads only).

[tbekas]

Probably you meant persisted to the repo/local store

[markspanbroek]

I like it! Makes a lot sense to use a merkle root instead of the block hashes in the manifest.

[markspanbroek]

Would a Codex node that stores only the data for a single slot, still announce that data under the merkle root of the entire dataset?

[dryajov]

Good question - yes, so far I don't see a better way of doing so. There isn't a good way of announcing block ranges over a DHT, but even in that case, blocks in slots aren't continuous. Definitely not ideal, and something to investigate further.

[dryajov]

@benbierens I think that 2 elements per node is optimal when it comes to the size of the proofs. Think about the extreme case when you have N leaves and N elements per node. The size of the proof for each of the N leaves will be O(N).

This, the size of proofs increases with the arity of the tree.

@dryajov I like the Idea. I'm not convinced yet that we need to store the entire tree. I think we could (should) store only a merkle proof next to a block and that should suffice for all the use cases in my opinion.

It will take only 2 times more on the disk, but I think it would reduce the complexity of the flows and possibly increase the performance of disk operations (sequential reads only).

The problem with this approach is that, in the new flow, you only know the root of the tree and the index of the leaf, you don't know the actual leaf or block hash before hand, the merkle tree serves as a kind of an index, where you read the tree and fetch the leaf (block hash) and after you can read it from the repo.