OpenAPI, changelog, cache package support

.github/workflows/build.yml

@@ -1,9 +1,7 @@
 name: Build
 on:
-  push:
-    branches:
-      - master
-  pull_request:
+  release:
+    types: [ published ]
 
 jobs:
   build:

.github/workflows/pages.yml

@@ -0,0 +1,34 @@
+name: GH Pages
+on:
+  schedule:
+    - cron: '0 0 * * *'
+  workflow_dispatch:
+
+jobs:
+  build:
+    name: Build files for GH Pages
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v2
+      - name: Generate schemas
+        uses: addnab/docker-run-action@v3
+        with:
+          registry: ghcr.io
+          image: sysbot-org/tgscraper:latest
+          options: -v ${{ github.workspace }}:/out
+          run: |
+          'app:export-schema --readable botapi.json'
+          'app:export-schema --yaml --readable botapi.yaml'
+          'app:export-schema --postman --readable botapi_postman.json'
+          'app:export-schema --openapi --readable botapi_openapi.json'
+          'app:export-schema --yaml --openapi --readable botapi_openapi.yaml'
+      - name: Deploy
+        uses: peaceiris/actions-gh-pages@v3
+        with:
+          github_token: ${{ secrets.GITHUB_TOKEN }}
+          publish_dir: ${{ github.workspace }}
+          destination_dir: schemas
+          publish_branch: gh-pages
+          cname: tgscraper.sys001.ml
+          enable_jekyll: true

CHANGELOG.md

@@ -0,0 +1,139 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+
+## [Unreleased]
+### Added
+- Support for OpenAPI schema: you can now generate code in any language!
+- Support for the new [sysbot/tgscraper-cache](https://github.com/Sysbot-org/tgscraper-cache) package: if installed, TGScraper will be much faster (there is no need to always fetch the live webpages)!
+- You can now validate a schema by using the `validateSchema` method, provided by the `TgScraper` class.
+- New `Versions::STABLE` constant: it will automatically return the latest stable version instead of the live version (useful for the cache package).
+- Added the JSON schema specification for the custom format provided by TGScraper.
+- Added this changelog.
+
+### Changed
+- The `Generator` class has now been renamed `TgScraper`.
+- The `required` property in method fields has been replaced by the new `optional` property, for the sake of consistency.
+- You now need a schema in order to instantiate the `TgScraper` class (don't worry, you can use the new methods `TgScraper::fromUrl` and `TgScraper::fromVersion`).
+- The `Versions` class constants have been replaced with an actual version string. If you still need the URLs, use the new class constant `Versions::URLS`.
+- TGScraper will now only return arrays. If you still need JSON or YAML encoding, please use the new `Encoder` class.  
+
+### Fixed
+- Minor improvements to `StubCreator`.
+- Fixed an issue with the CLI where `autoload.php` couldn't be found.
+- When exporting the schema, the CLI will now make sure that the destination directory exists.
+
+### Security
+- When a custom schema is used, only use `version`, `types` and `methods` fields.
+
+## [2.1.0] - 2021-07-31
+### Added
+- New repo workflows: automatic package build (and push to the GitHub registry), and automatic notifications via Telegram.
+- New `version` field for schemas: it contains the bot API version (if possible).
+- New `extended_by` field for types: if the current type is a parent one, it will contain its child types.
+
+### Changed
+- Now all type stubs implement the base `TypeInterface` interface.
+- Children type stubs now extend their parent.
+- Optional fields are now actually optional in the Postman collection (previously, you had to manually disable optional ones).
+
+### Fixed
+- Minor improvements to the schema extractor.
+
+## [2.0.1] - 2021-07-24
+### Changed
+- The README now includes many CLI examples.
+
+### Fixed
+- The link for the Bot API 5.2.0 snapshot was broken, it's working now.
+
+## [2.0.0] - 2021-07-24
+### Added
+- Support for Postman collection: now you can generate a JSON to use in Postman!
+- New class for the URLs of various bot API snapshots: `TgScraper\Constants\Versions`. 
+
+### Changed
+- Moved `TgScraper\StubCreator` to the new namespace `TgScraper\Common\StubCreator`.
+- Moved scraping logic from `TgScraper\Generator` to the new class `TgScraper\Common\SchemaExtractor`.
+- CLI has been completely reworked: it now uses the Symfony Console and it's much more reliable!
+
+## [1.4.0] - 2021-06-23
+### Added
+- YAML format is now supported.
+- Docker support! The package is published on the GitHub registry.
+
+## [1.3.0] - 2021-06-22
+### Added
+- Badges in the README! They contain a lot of useful information about the project (such as the minimum PHP version, latest stable version, etc).
+
+### Fixed
+- CLI now catches exceptions more reliably.
+
+### Security
+- Dependency `paquettg/php-html-parser` has been upgraded to `^3.1`.
+
+## [1.2.2] - 2021-06-20
+### Fixed
+- Fixed a typo in a property name of the `Response` class stub.
+
+## [1.2.1] - 2021-06-19
+### Removed
+- The abstract constructor for the `API` trait stub has now been removed.
+
+## [1.2.0] - 2021-06-19
+### Changed
+- The `API` class stub has been converted to a trait, and the constructor and the `sendRequest` methods are now abstract.
+
+### Fixed
+- Minor improvements to the CLI.
+
+## [1.1.0] - 2021-06-18
+### Added
+- New field for types: `optional`. It tells whether a value is always present or not.
+- Class stubs now have typed properties (with related PHPDoc comments).
+
+### Changed
+- Variable names have been changed to `camelCase`.
+
+## [1.0.2] - 2021-06-18
+### Fixed
+- Improved argument parsing for the CLI.
+
+## [1.0.1] - 2021-06-17
+### Changed
+- Project license is now the GNU Lesser GPL.
+
+## [1.0.0] - 2021-06-17
+### Added
+- New CLI to easily generate JSON schema or class stubs!
+- It's now possible to parse old bot API webpages! Pass the URL and it should work just fine.
+- New API class stub, it implements all bot API methods (it's incomplete though, so you must add your custom logic).
+
+### Changed
+- Renamed project to `tgscraper`.
+- Class namespace is now `TgScraper`, for the sake of consistency.
+- `StubProvider` class is now named `StubCreator`.
+- Reworked syntax for array of objects in field types: `Object[]` has been replaced by `Array<Object>`. 
+
+### Removed
+- Method stubs will no longer be generated.
+
+### Fixed
+- The parser is now more reliable, it no longer needs to be updated at every bot API release!
+
+[Unreleased]: https://github.com/Sysbot-org/tgscraper/compare/2.1...HEAD
+[2.1.0]: https://github.com/Sysbot-org/tgscraper/compare/2.0.1...2.1
+[2.0.1]: https://github.com/Sysbot-org/tgscraper/compare/2.0...2.0.1
+[2.0.0]: https://github.com/Sysbot-org/tgscraper/compare/1.4...2.0
+[1.4.0]: https://github.com/Sysbot-org/tgscraper/compare/1.3...1.4
+[1.3.0]: https://github.com/Sysbot-org/tgscraper/compare/1.2.1...1.3
+[1.2.1]: https://github.com/Sysbot-org/tgscraper/compare/1.2...1.2.1
+[1.2.0]: https://github.com/Sysbot-org/tgscraper/compare/1.1...1.2
+[1.1.0]: https://github.com/Sysbot-org/tgscraper/compare/1.0.2...1.1
+[1.0.2]: https://github.com/Sysbot-org/tgscraper/compare/1.0.1...1.0.2
+[1.0.1]: https://github.com/Sysbot-org/tgscraper/compare/1.0...1.0.1
+[1.0.0]: https://github.com/Sysbot-org/tgscraper/releases/tag/1.0

Dockerfile

@@ -3,9 +3,8 @@ FROM composer:latest AS tgscraper
 MAINTAINER Sys <[email protected]>
 
 WORKDIR /app
-COPY . .
-RUN composer install
-WORKDIR /artifacts
-VOLUME /artifacts
+RUN composer require sysbot/tgscraper sysbot/tgscraper-cache --no-progress --no-interaction --no-ansi --prefer-stable --optimize-autoloader
+WORKDIR /out
+VOLUME /out
 
-ENTRYPOINT ["php", "/app/bin/tgscraper"]
+ENTRYPOINT ["php", "/app/vendor/bin/tgscraper"]

README.md

@@ -9,8 +9,10 @@
 A PHP library used to extract JSON data (and auto-generate PHP classes)
 from [Telegram bot API documentation page](https://core.telegram.org/bots/api).
 
-**Note: the scraper is, obviously, based on a hack and you shouldn't rely on automagically generated files from it,
-since they are prone to errors. I'll try to fix them ASAP, but manual review is always required (at least for now).**
+## Changelog
+
+Interested in recent changes? Have a look [here](CHANGELOG.md)!
+
 
 ## Installation
 
@@ -52,12 +54,23 @@ Extract the latest schema in YAML format:
   $ vendor/bin/tgscraper app:export-schema --yaml botapi.yaml
 ```
 
+### OpenAPI
+
+Work in progress.
+
 ### Stubs
 
+_Note: since Telegram may change the page format at any time, do **NOT** rely on the automagically generated
+stubs from this library, **ALWAYS** review the code!_
+
 TGScraper can also generate class stubs that you can use in your library. A sample implementation is available in the [Sysbot Telegram module](https://github.com/Sysbot-org/Sysbot-tg).
 
-Create stubs in the `out/` directory using `Sysbot\Api` as namespace prefix:
+Create stubs in the `out/` directory using `Sysbot\Telegram` as namespace prefix:
 
 ```bash 
-  $ vendor/bin/tgscraper app:create-stubs --namespace-prefix "Sysbot\Api" out
-```
+  $ vendor/bin/tgscraper app:create-stubs --namespace-prefix "Sysbot\Telegram" out
+```
+
+## Custom format
+
+If you're interested in the custom format generated by TGScraper, you can find its schema [here](docs/schema.json).

bin/tgscraper

@@ -6,7 +6,17 @@ use Symfony\Component\Console\Application;
 use TgScraper\Commands\CreateStubsCommand;
 use TgScraper\Commands\ExportSchemaCommand;
 
-require_once __DIR__ . '/../vendor/autoload.php';
+$autoloadFiles = [
+    __DIR__ . '/../vendor/autoload.php',
+    __DIR__ . '/../../../autoload.php'
+];
+
+foreach ($autoloadFiles as $autoloadFile) {
+    if (file_exists($autoloadFile)) {
+        require_once $autoloadFile;
+        break;
+    }
+}
 
 $application = new Application('TGScraper', InstalledVersions::getVersion('sysbot/tgscraper'));
 

composer.json

@@ -12,6 +12,9 @@
     "symfony/console": "^5.3",
     "symfony/yaml": "^5.3"
   },
+  "suggest": {
+    "sysbot/tgscraper-cache": "To speed up schema fetching and generation."
+  },
   "autoload": {
     "psr-4": {
       "TgScraper\\": "src/"