Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tabular Data Package -> SQL database should also normalise #18

Open
willu47 opened this issue Nov 4, 2019 · 0 comments
Open

Tabular Data Package -> SQL database should also normalise #18

willu47 opened this issue Nov 4, 2019 · 0 comments

Comments

@willu47
Copy link
Member

willu47 commented Nov 4, 2019

Currently, data stored in a tabular data package is not normalised which makes conversion to an SQLite database inefficient.

The values of sets are used as keys to the table, rather than integer keys, which means that it is poor way to store this data, and we obtain few of the benefits of a relational database.

For example, parameter X might look like this:

REGION VALUE
SIMPLICITY 23.30
REGION 1 154.32
REGION 2 293.2

where the region names are stored in the REGION table:

VALUE
SIMPLICITY
REGION1
REGION2

and a foreign key link is defined in the JSON schema between the parameter X table REGION column and the REGION table.

We would obtain more efficient storage through better use of integer values primary keys in the set tables, and references to these using integer valued foreign keys. For example

parameter X:

fk_REGION VALUE
1 23.30
2 154.32
3 293.2

REGION table:

primary_key VALUE
1 SIMPLICITY
2 REGION1
3 REGION2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant