HEPCloud decisionengine 2.0.0

Released on 4/25/2022 (RC1 on 1/18/2022)

The full HEPCloud Decision Engine includes decisionengine, the framework, and decisionengine_modules, the standard library for sources, transformation, and providers.
Please see the curated release notes on hepcloud.github.io for Notes, Warning and more details.
Here are instructions to install and configure Decision Engine.

Fermilab users can access the RPMs also on the development HEPCloud yum repository.

---

Here are the GitHub generated notes:

What's Changed

• Support latest setuptools_scm by @jcpunk in #505
• Python optimized byte code removes assert under some conditions by @jcpunk in #497
• TaskManager now inherits from ComponentManager. by @bjwhite-fnal in #501
• Fix trailing whitespace in TRUNK by @jcpunk in #513
• Run the code style/standards checks early. by @jcpunk in #510
• Setup black, isort, and pyupgrade by @jcpunk in #444
• Remove now-unnecessary Python-to-Jsonnet conversion by @knoepfel in #516
• Unconditionally execute publishers with default configured logic engine by @knoepfel in #514
• Rework tests for #454 by @jcpunk in #517
• Fix the missing declarations on create_runner and ProcessingState by @jcpunk in #519
• Possible logging simplification by @knoepfel in #515
• Various simplifications recommended by flake8-simple by @jcpunk in #518
• Add channel-combination facilities by @knoepfel in #524
• Further simplify some cases by @jcpunk in #521
• Update hooks to latest via pre-commit autoupgrade by @jcpunk in #525
• Encourage use of automatic nag hook by @jcpunk in #520
• Improve parameter and variable names by @knoepfel in #527
• Allow channels to boot in parallel by @knoepfel in #532
• Remove unnecessary conversion to Pandas dataframe by @knoepfel in #535
• Start running tests with xdist by @jcpunk in #537
• Fix out of sync devel requirements by @jcpunk in #536
• Incrase coverage in LogicEngine by @jcpunk in #530
• Add error for running server as root by @jcpunk in #534
• Remove now-unnecessary blocking by @knoepfel in #540
• Fix config utilities by @knoepfel in #539
• dummy source and test by @goodenou in #538
• Start testing python 3.10 by @jcpunk in #533
• Automatically format jsonnet files with jsonnetfmt by @jcpunk in #542
• Forbid inheritance from SourceProxy by @knoepfel in #526
• EL7 doesn't ship with a new enough golang for jsonnetfmt by @jcpunk in #543
• Source proxy tests by @knoepfel in #541
• Ignoring E203, whitespace after ':', since black is adding the whitespaces by @mambelli in #545
• Drop testing/support for PyPy by @jcpunk in #548
• Adding a tool to help parsing Decision Engine structured logs by @mambelli in #544
• Cleanups in preparation for Kombu by @knoepfel in #549
• Lower timeout for actions. by @jcpunk in #552
• Lock in specific version of pytest-postgresql. by @jcpunk in #555
• Add instructions for running the Redis container. by @jcpunk in #553
• Add metrics collection to the framework by @shreyb in #523
• Removed unnecessary comment by @shreyb in #559
• Drop obsolete raw postgresql interface by @jcpunk in #558
• Don't try to package obsolete sql file by @jcpunk in #560
• Use kombu to communicate data from sources to cycles by @knoepfel in #556
• Add a redis server to the CI testing containers by @jcpunk in #565
• Do the build of the wheel as not-root per our requirements by @jcpunk in #566
• Added install document and updated instructions by @mambelli in #567
• Changed summaries to histograms in DecisionEngine and TaskManager mod… by @shreyb in #569
• Set state to error for exceptions caught before the thread start. by @jcpunk in #570
• Cleanup before source separation by @knoepfel in #564
• Further flesh out the documentation by @jcpunk in #571
• Report PID for source process by @knoepfel in #574
• Ensure that redis is running by @knoepfel in #573
• Set Apache 2.0 license and added REUSE compliance by @mambelli in #575
• Fix faulty tests by @knoepfel in #578
• Updated developers instructions w/ licensing requirements by @mambelli in #581
• Also run checks on backports to 1.7 by @jcpunk in #579
• addition of queue logging for multiprocessing by @goodenou in #563
• Drop unneeded module by @jcpunk in #583
• Recommend using the site user pip dir instead. by @jcpunk in #584
• Segment the update for setuptools so it gets cached correctly by @jcpunk in #588
• Reduce number of fixtures by @knoepfel in #591
• Ensure de_std.libsonnet is installed by @knoepfel in #593
• Update pre-commit hook versions by @knoepfel in #592
• Use local time for structlog timestamp by @knoepfel in #590
• Use pre-commit.ci rather than local actions by @jcpunk in #594
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #595
• Allow duplicate keys if their values are the same by @knoepfel in #589
• Set max retry timeout for sqlite in unit tests. by @jcpunk in #596
• Explicitly pass .coveragerc to pytest. by @jcpunk in #597
• Add some more logging to try and trace startup state by @jcpunk in #598
• Support sources shared between channels by @knoepfel in #568
• More logging about cherrypy state by @jcpunk in #601
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #602
• Add a 'ping' method by @jcpunk in #600
• Prep the server fixture to permit the metrics webserver by @jcpunk in #604
• Log finished with DB init by @jcpunk in #605
• Move metrics fixtures to the fixtures file by @jcpunk in #603
• Ensure we are calling the cherrypy shutdown methods by @jcpunk in #606
• Leave note to remember to cleanup temp files by @jcpunk in #607
• cherrypy metrics server blocking startup by @jcpunk in #608
• Pin pytest version by @jcpunk in #609
• Metrics seems to want the channels setup to complete by @jcpunk in #610
• pin jsonnet 0.17.0 by @vitodb in #611
• Remove the unit test log directory if it got created by @jcpunk in #612
• Note lines we are not testing by @jcpunk in #614
• Add missing tests for inherited functions by @jcpunk in #615
• Add missing test to ensure we can change the channel level twice by @jcpunk in #618
• Logger tests are a bit unstable at high parallelization. by @jcpunk in #619
• A simple test to ensure the metrics can run by @jcpunk in #617
• We should have one path where we test without -v. by @jcpunk in #616
• More unit testing by @jcpunk in #621
• Remove unnecessary @pytest.mark.usefixtures decorations by @knoepfel in #620
• Flush Redis DB after DE server stops by @knoepfel in #623
• Isolate Redis DBs during unit tests by @knoepfel in #624
• Print number of pickled bytes of source-produced data by @knoepfel in #629
• [pre-commit.ci] pre-commit autoupdate by @pre-commit-ci in #631
• Fix message explosion by @knoepfel in #632
• Add a minimal container image more suited to production usage by @jcpunk in #622
• Add Countdown class to address global timeout problem by @knoepfel in #636
• Update Python 3.6-compatible pre-commit hooks. by @knoepfel in #641
• Archive unit test logs by @vitodb in #643
• Update ci workflow by @vitodb in #640
• Miscellaneous fixes for DE 2.0 by @knoepfel in #637
• Added changelog file for developers curated list of changes by @mambelli in #645

New Contributors

• @pre-commit-ci made their first contribution in #595

Full Changelog: 1.7.0...2.0.0