Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tremor binary encoding #1276

Open
Licenser opened this issue Oct 21, 2021 · 0 comments
Open

tremor binary encoding #1276

Licenser opened this issue Oct 21, 2021 · 0 comments
Labels
_complexity:high A task with a high complexity that should be hard enhancement New feature or request needs-rfc This change needs to follow the RFC process _size:humongous A large task that will take a lot of work to complete

Comments

@Licenser
Copy link
Member

Describe the problem you are trying to solve

For networking and storage, it would be good to have a stable binary representation of tremor value. This might also enable (or at least make it more feasible) to pass data in and out of environments such as WASM runtimes.

Describe the solution you'd like

There is no clear solution here yet since this is a topic that requires a RFC, that said there are some requirements:

  1. the encoding needs to be binary stable under equivalence (meaning that something like {"a": 1, "b": 2} and {"b": 2, "a": 1} need to encode to the same binary value.
  2. It should be fast (yea that's somewhat of a given 😂)
  3. It needs to support versioning (since we will evolve it might matter to be able to detect what version it comes from)
  4. It needs to be architecturally independent (big vs little-endian encoding)
  5. It is worth considering our primary use-case and optimizing for it (I.e. map keys are usually small etc.)

Notes

This is a large, long-running issue and might evolve into a in-memory format for tremor value that allows more then just storage (as that would make decoding really cheap) - not something needed from day one but it is worth keeping the option in mind

@Licenser Licenser added enhancement New feature or request needs-rfc This change needs to follow the RFC process _complexity:high A task with a high complexity that should be hard _size:humongous A large task that will take a lot of work to complete labels Oct 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
_complexity:high A task with a high complexity that should be hard enhancement New feature or request needs-rfc This change needs to follow the RFC process _size:humongous A large task that will take a lot of work to complete
Projects
None yet
Development

No branches or pull requests

1 participant