Removed deprecated pkg_resources API

The `pkg_resources` API has been deprecated in favor of the new
`importlib.resources` API. This commit replaces all the usages of
`pkg_resources` with the new API.

To implement this new path, the `get_schema` method of each StreamFlow
extension must now return the content of the schema instead of the path
to the JSON Schema file. Documentation has been updated accordingly.

Plus, the standard `importlib.metadata` package is now used instead of
the external `importlib_metadata` backport.

docs/source/ext/plugins.rst

@@ -39,7 +39,7 @@ The ``cls`` attribute points to the class that implements the extension point, w
 
 The ``register`` method of the ``StreamFlowPlugin`` implementation must contain all the required calls to the ``register_extension_point`` methods in order to make the extension points available to the StreamFlow control plane. Note that a single ``StreamFlowPlugin`` can register multiple extension points, and even multiple implementations for the same extension point.
 
-Each extension point class extends the ``SchemaEntity`` abstract class, whose method ``get_schema`` returns the path to the class configuration in `JSON Schema 2019-09 <https://json-schema.org/draft/2019-09/release-notes.html>`_ format. The root entity of this file must be an ``object`` defining a set of properties, which will be exposed to the user in the ``config`` section of the ``streamflow.yml`` file, type-checked by StreamFlow at the beginning of each workflow run, and passed to the class constructor at every instantiation.
+Each extension point class extends the ``SchemaEntity`` abstract class, whose method ``get_schema`` returns the class configuration in `JSON Schema 2019-09 <https://json-schema.org/draft/2019-09/release-notes.html>`_ format. The root entity of this file must be an ``object`` defining a set of properties, which will be exposed to the user in the ``config`` section of the ``streamflow.yml`` file, type-checked by StreamFlow at the beginning of each workflow run, and passed to the class constructor at every instantiation.
 
 The base class for each extension point defined by StreamFlow and the respective registration method exposed by the ``StreamFlowPlugin`` class are reported in the table below:
 
@@ -58,7 +58,7 @@ Base Class
 :ref:`streamflow.core.scheduling.Scheduler <Scheduler>`                                           ``register_scheduler``
 =============================================================================================     ===================================
 
-In addition, a ``register_schema`` method allows to register additional JSON Schema files, which are not directly referenced by any ``SchemaEntity`` class through the ``get_schema`` method. This feature is useful to, for example, define some base abstract JSON Schema that concrete entities can extend.
+In addition, a ``register_schema`` method allows to register additional JSON Schemas, which are not directly referenced by any ``SchemaEntity`` class through the ``get_schema`` method. This feature is useful to, for example, define some base abstract JSON Schema that concrete entities can extend.
 
 Note that there is no official way to make JSON Schema files inherit properties from each other, as vanilla JSON Schema format does not support inheritance. However, it is possible to extend base schemas using the combination of `allOf <https://json-schema.org/understanding-json-schema/reference/combining.html#allof>`_ and `unevaluatedProperties <https://json-schema.org/understanding-json-schema/reference/object.html#unevaluated-properties>`_ directives of JSON Schema 2019-09, as follows:
 
@@ -107,16 +107,19 @@ As an example, suppose that a class ``PostgreSQLDatabase`` implements a `Postgre
     class PostgreSQLDatabase(Database):
         @classmethod
         def get_schema(cls) -> str:
-            return pkg_resources.resource_filename(
-                __name__, os.path.join("schemas", "postgresql.json")
+            return (
+                resources.files(__spec__.parent)
+                .joinpath("schemas")
+                .joinpath("postgresql.json")
+                .read_text("utf-8")
             )
         ...
 
     class PostgreSQLStreamFlowPlugin(StreamFlowPlugin):
         def register(self) -> None:
             self.register_database("unito.postgresql", PostgresqlDatabase)
 
-Each extension point class must implement a ``get_schema`` method, pointing to a `JSON Schema <https://json-schema.org/>`_ file, which contains all the configurable parameters that can be specified by the user in the ``streamflow.yml`` file. Such parameters will be propagated to the class constructor at each invocation. For example, the ``PostgreSQLDatabase`` class specified above points to a ``schemas/postgresql.json`` schema file in the same Python module.
+Each extension point class must implement a ``get_schema`` method, which returns a `JSON Schema <https://json-schema.org/>`_ with all the configurable parameters that can be specified by the user in the ``streamflow.yml`` file. Such parameters will be propagated to the class constructor at each invocation. For example, the ``PostgreSQLDatabase`` class specified above points to a ``schemas/postgresql.json`` schema file in the same Python module.
 
 A schema file should follow the `2019-09 <https://json-schema.org/draft/2019-09/release-notes.html>`_ version of JSON Schema. StreamFlow uses schema files to validate the ``streamflow.yml`` file at runtime before executing a workflow instance. Plus, it relies on schema ``properties`` to print documentation when a user invokes the ``streamflow ext show`` CLI subcommand. An example of schema file for the ``PostreSQLDatabase`` class is the following:
 

requirements.txt

@@ -6,7 +6,6 @@ bcrypt==4.0.1
 cachetools==5.3.2
 cwltool==3.1.20231114134824
 cwl-utils==0.31
-importlib_metadata==6.8.0
 Jinja2==3.1.2
 jsonschema==4.20.0
 kubernetes_asyncio==28.2.1

streamflow/config/schema.py

@@ -1,11 +1,9 @@
 from __future__ import annotations
 
-import json
-import os
 import posixpath
+from importlib import resources
 from typing import MutableMapping
 
-import pkg_resources
 from referencing import Registry, Resource
 
 from streamflow.core.context import SchemaEntity
@@ -31,20 +29,23 @@
 
 
 ext_schemas = [
-    pkg_resources.resource_filename(
-        "streamflow.deployment.connector", os.path.join("schemas", "queue_manager.json")
-    )
+    resources.files("streamflow.deployment.connector")
+    .joinpath("schemas")
+    .joinpath("queue_manager.json")
 ]
 
 
 class SfSchema:
     def __init__(self):
         self.registry: Registry = Registry()
         for version in _CONFIGS.keys():
-            schema = pkg_resources.resource_filename(
-                __name__, os.path.join("schemas", version, "config_schema.json")
+            self.add_schema(
+                resources.files(__spec__.parent)
+                .joinpath("schemas")
+                .joinpath(version)
+                .joinpath("config_schema.json")
+                .read_text("utf-8")
             )
-            self.add_schema(schema)
         self.inject_ext(binding_filter_classes, "bindingFilter")
         self.inject_ext(checkpoint_manager_classes, "checkpointManager")
         self.inject_ext(cwl_docker_translator_classes, "cwl/docker")
@@ -56,14 +57,13 @@ def __init__(self):
         self.inject_ext(policy_classes, "policy")
         self.inject_ext(scheduler_classes, "scheduler")
         for schema in ext_schemas:
-            self.add_schema(schema)
+            self.add_schema(schema.read_text("utf-8"))
         self.registry = self.registry.crawl()
 
     def add_schema(self, schema: str) -> Resource:
-        with open(schema) as f:
-            resource = Resource.from_contents(json.load(f))
-            self.registry = resource @ self.registry
-            return resource
+        resource = Resource.from_contents(schema)
+        self.registry = resource @ self.registry
+        return resource
 
     def get_config(self, version: str) -> Resource:
         if version not in _CONFIGS:

streamflow/cwl/antlr/ECMAScriptLexer.py

@@ -1,7 +1,8 @@
 # Generated from ECMAScript.g4 by ANTLR 4.13.0
-from antlr4 import *
-from io import StringIO
 import sys
+
+from antlr4 import *
+
 if sys.version_info[1] > 5:
     from typing import TextIO
 else:

streamflow/cwl/antlr/ECMAScriptParser.py

@@ -1,8 +1,9 @@
 # Generated from ECMAScript.g4 by ANTLR 4.13.0
 # encoding: utf-8
-from antlr4 import *
-from io import StringIO
 import sys
+
+from antlr4 import *
+
 if sys.version_info[1] > 5:
 	from typing import TextIO
 else:

streamflow/cwl/requirement/docker/docker.py

@@ -2,10 +2,9 @@
 
 import os
 import posixpath
+from importlib import resources
 from typing import MutableSequence
 
-import pkg_resources
-
 from streamflow.core import utils
 from streamflow.core.deployment import DeploymentConfig, Target
 from streamflow.cwl.requirement.docker.translator import CWLDockerTranslator
@@ -207,8 +206,11 @@ def __init__(
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "docker.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("docker.json")
+            .read_text("utf-8")
         )
 
     def get_target(

streamflow/cwl/requirement/docker/kubernetes.py

@@ -1,9 +1,8 @@
 from __future__ import annotations
 
-import os
 import tempfile
+from importlib import resources
 
-import pkg_resources
 from jinja2 import Template
 
 from streamflow.core import utils
@@ -30,9 +29,16 @@ def __init__(
         wait: bool = True,
     ):
         super().__init__(config_dir=config_dir, wrapper=wrapper)
-        self.template: str | None = template or pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "kubernetes.jinja2")
-        )
+        if template is not None:
+            with open(template) as t:
+                self.template: Template = Template(t.read())
+        else:
+            self.template: Template = Template(
+                resources.files(__spec__.parent)
+                .joinpath("schemas")
+                .joinpath("kubernetes.jinja2")
+                .read_text("utf-8")
+            )
         self.debug: bool = debug
         self.inCluster: bool = inCluster
         self.kubeconfig: str | None = kubeconfig
@@ -47,8 +53,11 @@ def __init__(
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "kubernetes.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("kubernetes.json")
+            .read_text("utf-8")
         )
 
     def get_target(
@@ -59,10 +68,8 @@ def get_target(
         target: Target,
     ) -> Target:
         name = utils.random_name()
-        with open(self.template) as t:
-            template = Template(t.read())
         with tempfile.NamedTemporaryFile(mode="w", delete=False) as f:
-            template.stream(
+            self.template.stream(
                 name=name,
                 image=image,
                 network_access=network_access,

streamflow/cwl/requirement/docker/singularity.py

@@ -2,10 +2,9 @@
 
 import os
 import posixpath
+from importlib import resources
 from typing import MutableSequence
 
-import pkg_resources
-
 from streamflow.core import utils
 from streamflow.core.deployment import DeploymentConfig, Target
 from streamflow.cwl.requirement.docker.translator import CWLDockerTranslator
@@ -143,8 +142,11 @@ def __init__(
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "singularity.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("singularity.json")
+            .read_text("utf-8")
         )
 
     def get_target(

streamflow/data/manager.py

@@ -1,12 +1,10 @@
 from __future__ import annotations
 
 import asyncio
-import os
+from importlib import resources
 from pathlib import Path, PurePosixPath
 from typing import TYPE_CHECKING
 
-import pkg_resources
-
 from streamflow.core.data import DataLocation, DataManager, DataType
 from streamflow.core.deployment import Connector, Location
 from streamflow.data import remotepath
@@ -77,8 +75,11 @@ def get_data_locations(
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "data_manager.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("data_manager.json")
+            .read_text("utf-8")
         )
 
     def get_source_location(

streamflow/deployment/connector/container.py

@@ -7,10 +7,10 @@
 import posixpath
 import shlex
 from abc import ABC, abstractmethod
+from importlib import resources
 from shutil import which
 from typing import Any, MutableMapping, MutableSequence
 
-import pkg_resources
 from cachetools import Cache, TTLCache
 
 from streamflow.core import utils
@@ -684,8 +684,11 @@ async def get_available_locations(
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "docker.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("docker.json")
+            .read_text("utf-8")
         )
 
     async def undeploy(self, external: bool) -> None:
@@ -867,8 +870,11 @@ async def get_available_locations(
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "docker-compose.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("docker-compose.json")
+            .read_text("utf-8")
         )
 
     async def undeploy(self, external: bool) -> None:
@@ -1186,8 +1192,11 @@ async def get_available_locations(
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "singularity.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("singularity.json")
+            .read_text("utf-8")
         )
 
     async def undeploy(self, external: bool) -> None:

streamflow/deployment/connector/kubernetes.py

@@ -10,6 +10,7 @@
 import tarfile
 import uuid
 from abc import ABC, abstractmethod
+from importlib import resources
 from math import ceil, floor
 from pathlib import Path
 from shutil import which
@@ -22,7 +23,6 @@
     cast,
 )
 
-import pkg_resources
 import yaml
 from cachetools import Cache, TTLCache
 from kubernetes_asyncio import client
@@ -852,8 +852,11 @@ async def deploy(self, external: bool) -> None:
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "kubernetes.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("kubernetes.json")
+            .read_text("utf-8")
         )
 
     async def undeploy(self, external: bool) -> None:
@@ -1056,8 +1059,11 @@ async def deploy(self, external: bool) -> None:
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "helm3.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("helm3.json")
+            .read_text("utf-8")
         )
 
     async def undeploy(self, external: bool) -> None:

streamflow/deployment/connector/local.py

@@ -4,10 +4,10 @@
 import shutil
 import sys
 import tempfile
+from importlib import resources
 from pathlib import Path
 from typing import MutableMapping, MutableSequence
 
-import pkg_resources
 import psutil
 
 from streamflow.core.deployment import Connector, LOCAL_LOCATION, Location
@@ -109,8 +109,11 @@ async def get_available_locations(
 
     @classmethod
     def get_schema(cls) -> str:
-        return pkg_resources.resource_filename(
-            __name__, os.path.join("schemas", "local.json")
+        return (
+            resources.files(__spec__.parent)
+            .joinpath("schemas")
+            .joinpath("local.json")
+            .read_text("utf-8")
         )
 
     async def undeploy(self, external: bool) -> None: