Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CIP: Standardize data expiry time for pruned nodes #16

Merged
merged 4 commits into from
Nov 27, 2023

Conversation

musalbas
Copy link
Member

@musalbas musalbas commented Nov 27, 2023

@musalbas musalbas marked this pull request as draft November 27, 2023 11:21

On the Celestia data availability network, there is only one topic under which storage nodes can be discovered, where all nodes discovered under that topic are expected to be non-pruned nodes.

An additional new topic will be created for pruned nodes, so that nodes discovered under the existing topic will not unexpectedly have missing block data when being queried.
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we name the exact topics here @Wondertan?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can name them full and archival

Where full is the default (pruned) and archival is historical

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a side note, I was going to name them archival in this text, but then I saw that Ethereum refers to archival nodes as nodes that store all historical blocks but only prune state.

So I thought maybe it's better to be explicit and refer to them as block archival nodes; or non-pruned nodes. To avoid confusion.

But if you're saying full and archival are the actual topic names in libp2p, I don't think that matters.


On the Celestia data availability network, there is only one topic under which storage nodes can be discovered, where all nodes discovered under that topic are expected to be non-pruned nodes.

An additional new topic will be created for pruned nodes, so that nodes discovered under the existing topic will not unexpectedly have missing block data when being queried.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can name them full and archival

Where full is the default (pruned) and archival is historical


On the Celestia data availability network, there is only one topic under which storage nodes can be discovered, where all nodes discovered under that topic are expected to be non-pruned nodes.

An additional new topic will be created for pruned nodes, so that nodes discovered under the existing topic will not unexpectedly have missing block data when being queried.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The pruned node topic will not be an additional topic but rather the current topic that we have which is full. The additional topic will be for archival nodes (archival).

We should call storage nodes archival to better indicate that they retain + serve historical blocks.

Copy link
Member Author

@musalbas musalbas Nov 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO it should be the other way round: the existing topic should be for archival nodes, otherwise you will break backwards compatibility, because you will clutter the existing topic with non-archival nodes, which will not be backwards compatible with older nodes that are discovering nodes in the existing topic, and thinking they have all the historical blocks.

We could keep full for archival and introduce an additional topic such as pruned.

title: Standardize data expiry time for pruned nodes
description: Standardize default data expiry time for pruned nodes to 30 days.
author: Mustafa Al-Bassam (@musalbas)
discussions-to: URL
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs a discussion link. Maybe open one in the forum

## Copyright

Copyright and related rights waived via [CC0](../LICENSE).
<!-- markdownlint-disable MD013 -->
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can delete this line

Copy link
Member Author

@musalbas musalbas Nov 27, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the implications of deleting that line? Should that also be removed from the template then?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will do so now. it's for markdown linters to ignore some rules.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@musalbas
Copy link
Member Author

@musalbas musalbas marked this pull request as ready for review November 27, 2023 12:28
@YazzyYaz YazzyYaz merged commit ae3cc2f into celestiaorg:main Nov 27, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants