Trying to have the sanest naming schema for Pkgforge's Versions #1454

Azathothas · 2024-12-17T05:29:00Z

Azathothas
Dec 17, 2024

Hi, I am from (@pkgforge).
As per https://repology.org/docs/requirements , Our repositories currently meet the following requirements:

Data format

https://docs.pkgforge.dev/repositories/pkgforge-edge/metadata

https://docs.pkgforge.dev/repositories/pkgforge-stable/metadata

https://docs.pkgforge.dev/repositories/pkgforge-community/metadata

Availability

The metadata urls are listed in the above section, the main urls are:

pkgforge-edge : https://bin.pkgforge.dev/aarch64/METADATA.AIO.json | https://bin.pkgforge.dev/x86_64/METADATA.AIO.json

pkgforge-stable : https://bincache.pkgforge.dev/aarch64/METADATA.AIO.json | https://bincache.pkgforge.dev/x86_64/METADATA.AIO.json

pkgforge-community : https://soarpkgs.pkgforge.dev/metadata/METADATA.json

Completeness

I think we mostly satisfy this as well, but if we don't please tell us what's missing

And to address this:

Finally, we don't support personal repositories, PPAs, overlays and alike, or collections thereof, due to poor quality, absence of guarantees and moderation and high probability of becoming abandoned. For the same reason, we impose minimal requirements on maturity - that is, a repository should contain at least a few hundred packages and be maintained for at least half a year.

pkgforge, the org itself was created on Nov 04, 2024. However, Toolpacks (One of the repositories), the project that would eventually be turned into pkgforge has existed since ~ July, 2023, this can be verified on the Wayback Machine. You can read the lore here: https://docs.pkgforge.dev/orgs/readme/projects/toolpacks-bincache/faq#history-and-lore
So, I am not really sure whether meet this particular requirement or not...

And about number of packages, we currently have (prebuilt only):

While the total pkgs (including non-prebuilts) is ~ 4700 : https://docs.pkgforge.dev/soar/readme/packages#total

Unfortunately, our repositories, fail the version requirements:

We have fake, or rather auto guessed version
We use YYYYMMDDTHHMMSS (of the build not the commit) as version, if our automation can't guess the version [Example: 241213T193116)

So our repositories most certainly can't be added to repology yet..
I read:

And we decided on this:

Use upstream's version as-is, if it's available (Source from Github Release Tag, or Git tag) & we aren't building from source (Most of our pkgs are built from the latest git checkout)
If upstream's version is available & we are building from source, use: ${LAST_VERSION}.postgitYYYYMMDD.${GIT_SHA}, where the YYYYMMDD is the date of the ${GIT_SHA} & NOT build date
If upstream's version is NOT available & we are building from source, use: 0-unstable.postgitYYYYMMDD.${GIT_SHA}

We need the ${GIT_SHA} for security/reference reasons, we can't exclude/drop it.
${GIT_SHA} would be fetched using something like:

$ git ls-remote "https://github.com/repology/repology-updater" HEAD | awk '{print substr($1, 1, 7)}'
8dfc3bc

And if possible, can we use a more precise date: YYYYMMDDTHHMMSS instead of just YYYYMMDD ?
241213 --> 241213T193116

Would that satisfy the version requirements, or you have some better ideas?

Answered by AMDmi3

Dec 17, 2024

We need repology support for the same reason homebrew does.

If I'm not mistaken, homebrew always (or mostly) build from sources which are official releases, so it makes full use of repologys version comparison. It doesn't look the same.

Erroneous data

Did a quick grep by versions from https://bin.pkgforge.dev/aarch64/METADATA.AIO.json

the rest are fetched from upstream as most of them already release static binaries.

Okay, this makes sense. Just to be sure, are versions going to be set by hand in these cases, or will latest upstream releases be automatically tracked?

If we build from latest commit, You recommend we use: latest.${DATE_OF_latest_commit_we_built_from}.${SHORT_COMMIT…

View full answer

AMDmi3 · 2024-12-17T16:37:28Z

AMDmi3
Dec 17, 2024
Maintainer

If you build from latest git checkout, latest version is fine, you don't need to change that. Faking the version based on latest release is a bad idea as master may have been been diverged from the release branch.

Looking at the data, the main issue is that you seem to use tags as version, this won't work

    "version": "release-1.37.0",
    "version": "release-5.0.2_v2.0.10",
    "version": "reptyr-0.10.0",
    "version": "tcpdump-4.99.5",
    "version": "util-v0.16.0",
    "version": "v-v0.1.2",
    "version": "v0.0.1",
    "version": "v0.0.1-beta.9",
    "version": "v0.0.10",
    "version": "v0.0.15",
    "version": "v0.0.19",
    "version": "v0.0.2",
    "version": "v0.0.21",

1 reply

Azathothas Dec 17, 2024
Author

If you build from latest git checkout, latest version is fine

As in I should just use latest...?
But I still want to embed other information, so is latest.postgitYYYYMMDDTHHMMSS.${GIT_SHA} is fine as well?

Looking at the data, the main issue is that you seem to use tags as version, this won't work

Yes, I will fix all the issues before I officially submit an issue for inclusion into repology.

AMDmi3 · 2024-12-17T17:37:55Z

AMDmi3
Dec 17, 2024
Maintainer

As in I should just use latest...?
But I still want to embed other information, so is latest.postgitYYYYMMDDTHHMMSS.${GIT_SHA} is fine as well?

Sure. Documentation refers to "real" versions which can be compared, while you have rolling releases which are not compared (similar to 9999s in gentoo), and Repology does not care of the version format as long as it can identify these are rolling releases. latest prefix is enough for that, so it would make sense to drop postgit. In fact, shorter versions are preferred for readability, but latest.YYYYMMDDTHHMMSS.HHHHHHH is probably still ok.

Yes, I will fix all the issues before I officially submit an issue for inclusion into repology.

I still do not quite get, are you building packages from tags, from the tip of the main branch, or mixed? If tags are involved, how do you chose tag, as not all tags are releases. Also if you're going to extract version from the tag, how are you going to do it reliably, as it's not technically possible due to arbitrary tag format?

Also, currently 2/3 of your versions are latest (others probably are built from the latest tag?), why do you need Repology support if it's not going to compare versions and report vulnerabilities?

7 replies

Azathothas Dec 17, 2024
Author

latest prefix is enough for that, so it would make sense to drop postgit

I had read #345 & assumed, having a postgit prefix would make your job easier. But if rolling releases aren't compared, then I think I will indeed use a shorter format.

I still do not quite get, are you building packages from tags, from the tip of the main branch, or mixed?

Currently, 90% of the packages use the old build scripts, which weren't written with versioning in mind. So, any package that's built, it's from the tip of the main branch. The version that's in the metadata is generated later, which prefers GitHub releases and then git tags & finally just latest if neither are possible. Thus, why these are fake (incorrect) versions.

Also, currently 2/3 of your versions are latest (others probably are built from the latest tag?), why do you need Repology support if it's not going to compare versions and report vulnerabilities?

Erroneous data, maybe around 30-40% of the packages we build from source (latest commit), the rest are fetched from upstream as most of them already release static binaries.
We need repology support for the same reason homebrew does.

To summarize:

If we build from latest commit, You recommend we use: latest.${DATE_OF_latest_commit_we_built_from}.${SHORT_COMMIT_SHA} [Example: latest.241217T180221.ded6174c]
If we just fetch prebuilt-asset from GitHub etc., we use their release-version/tag as-is

AMDmi3 Dec 17, 2024
Maintainer

We need repology support for the same reason homebrew does.

If I'm not mistaken, homebrew always (or mostly) build from sources which are official releases, so it makes full use of repologys version comparison. It doesn't look the same.

Erroneous data

Did a quick grep by versions from https://bin.pkgforge.dev/aarch64/METADATA.AIO.json

the rest are fetched from upstream as most of them already release static binaries.

Okay, this makes sense. Just to be sure, are versions going to be set by hand in these cases, or will latest upstream releases be automatically tracked?

If we build from latest commit, You recommend we use: latest.${DATE_OF_latest_commit_we_built_from}.${SHORT_COMMIT_SHA} [Example: latest.241217T180221.ded6174c]

If we just fetch prebuilt-asset from GitHub etc., we use their release-version/tag as-is

All correct. FWIW I'd prefer shorter dates, but that's not critical especially of longer ones are important for you.

Answer selected by Azathothas

Azathothas Dec 17, 2024
Author

Just to be sure, are versions going to be set by hand in these cases, or will latest upstream releases be automatically tracked?

Our build script will automatically get the latest tag, and attempt to download the prebuilt binaries using this tag, if it succeeds (has all needed files), we will use this tag as version, if it doesn't succeed we abort the build/fetch for that particular pkg.
So rest assured, even if these aren't set by hand, our data will never contain fake/incorrect version.

All correct. FWIW I'd prefer shorter dates, but that's not critical especially of longer ones are important for you.

The longer ones are used by us just for precision.

Thank you, I will mark the discussion as answered.
After I have fixed all our versioning issue, I will formally create an Issue to finally add our repos to repology.
Much work to do.

AMDmi3 Dec 17, 2024
Maintainer

So rest assured, even if these aren't set by hand, our data will never contain fake/incorrect version.

Yes, but how are you going to extract the version from a tag?

Azathothas Dec 17, 2024
Author

Yes, but how are you going to extract the version from a tag?

By tag, I meant Github Release Tag, which are mostly same as version unless they aren't...

For example, we fetch curl from this repo: https://github.com/stunnel/static-curl
The release tag https://github.com/stunnel/static-curl/releases is same as curl's official versions: https://curl.se/docs/releases.html
So here, our SBUILD will have:

curl -qfsSL "https://api.gh.pkgforge.dev/repos/stunnel/static-curl/releases?per_page=100" | jq -r '[.[] | select(.draft == false)] | .[0].tag_name | gsub("\\s+"; "")' | tr -d '[:space:]'

The value returned from there, will then be used to download the binary:

soar dl "https://github.com/stunnel/static-curl@${RELEASE_TAG}"

And if we find the version & the tag differ, then we will change our x_exec.pkgver to use whatever is the correct way to fetch it

AMDmi3 Dec 17, 2024
Maintainer

So, what will happen if tag is e.g. v1.2.3 or release-1.2.3?

Azathothas · 2025-01-29T04:47:44Z

Azathothas
Jan 29, 2025
Author

Hi @AMDmi3,
We are finally done with our rewrite & would like your feedback if the data now is upto repology's standard.

Data Format

https://docs.pkgforge.dev/repositories/bincache/metadata (Prebuilts, similar to brew's regular packages)
https://docs.pkgforge.dev/repositories/pkgcache/metadata (Prebuilts, similar to brew's casks)
https://docs.pkgforge.dev/repositories/soarpkgs/metadata (Recipe only, used by our build infra, it's like the AUR)

Availability

The metadata urls are listed in the above section, the main urls are:

https://meta.pkgforge.dev/bincache/aarch64-Linux.json
https://meta.pkgforge.dev/bincache/x86_64-Linux.json
https://meta.pkgforge.dev/pkgcache/aarch64-Linux.json
https://meta.pkgforge.dev/pkgcache/x86_64-Linux.json
https://meta.pkgforge.dev/soarpkgs/INDEX.json
You can also append .xz or .zstd to any of the urls above to get a compressed file, for instance:
https://meta.pkgforge.dev/soarpkgs/INDEX.json --> https://meta.pkgforge.dev/soarpkgs/INDEX.json.xz | https://meta.pkgforge.dev/soarpkgs/INDEX.json.zstd

Versioning

This went through the biggest change.
All of our versions are real & reliable. We also added a new field version_upstream which shows the last latest version upstream released. (We use this for easy identification, as now we build the majority of our packages from source)
Packages built from source start with HEAD- (.version), and can be easily filtered
We have also added the .repology[] field to majority of the packages so you can also directly reference that.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trying to have the sanest naming schema for Pkgforge's Versions #1454

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 3 comments 8 replies

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Trying to have the sanest naming schema for Pkgforge's Versions #1454

Azathothas Dec 17, 2024

Replies: 3 comments · 8 replies

AMDmi3 Dec 17, 2024 Maintainer

Azathothas Dec 17, 2024 Author

AMDmi3 Dec 17, 2024 Maintainer

Azathothas Dec 17, 2024 Author

AMDmi3 Dec 17, 2024 Maintainer

Azathothas Dec 17, 2024 Author

AMDmi3 Dec 17, 2024 Maintainer

Azathothas Dec 17, 2024 Author

AMDmi3 Dec 17, 2024 Maintainer

Azathothas Jan 29, 2025 Author

Data Format

Availability

Versioning

Azathothas
Dec 17, 2024

Replies: 3 comments 8 replies

AMDmi3
Dec 17, 2024
Maintainer

Azathothas Dec 17, 2024
Author

AMDmi3
Dec 17, 2024
Maintainer

Azathothas Dec 17, 2024
Author

AMDmi3 Dec 17, 2024
Maintainer

Azathothas Dec 17, 2024
Author

AMDmi3 Dec 17, 2024
Maintainer

Azathothas Dec 17, 2024
Author

AMDmi3 Dec 17, 2024
Maintainer

Azathothas
Jan 29, 2025
Author