scripts/sitemaps.js is slow

[Daniel15]

This is one of the slowest parts of building the site:

> [email protected] build:sitemaps /opt/build/repo
> ./scripts/sitemaps.js
1:41:03 PM: Sitemap generated successfully
1:41:04 PM: Done in 334.40s.

@Haroenv - Could we somehow speed this up, or otherwise cache the sitemaps so they're only regenerated every so often? I wonder if we could store the sitemaps somewhere (perhaps some sort of blob storage like S3) and regenerate them via a cronjob rather than every time the site is deployed.

[Haroenv]

Definitely not a bad idea, I don't have any s3 account or whatever, so feel free. All you need is the api key which is on netlify. You can crop it daily or weekly and it will be fine

[Daniel15]

Do you know how large the files are? I can probably just throw them onto my server under https://nightly.yarnpkg.com/ somewhere.

[Haroenv]

I ran the script locally and it's:

$ ls -l sitemaps/
total 498496
-rw-r--r--  1 algolia  staff   1.4K Jun 30 09:58 sitemap-index.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:54 sitemap.0.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:55 sitemap.1.xml
-rw-r--r--  1 algolia  staff    12M Jun 30 09:58 sitemap.10.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:55 sitemap.2.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:56 sitemap.3.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:56 sitemap.4.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:56 sitemap.5.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:57 sitemap.6.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:57 sitemap.7.xml
-rw-r--r--  1 algolia  staff    23M Jun 30 09:57 sitemap.8.xml
-rw-r--r--  1 algolia  staff    24M Jun 30 09:58 sitemap.9.xml

So it's 255MB on disk

This can also be gzipped, either by the server (that's the case on netlify now IIRC), or by the script. I didn't do that now, because it would take a while, but I'm pretty sure it will get a very significant improvement by doing that, there must be lots of repetition.