[WIP] pyomo config (#259)

* getting started on software to convert base_parsers to the Pyomo config system


* new version of Pyomo

* drop the use of afarmer and merge into farmer

* update the sed bash (without testing)

* add Gatherv to the mock MPI

* update the readme for the disruption

README.rst

@@ -6,6 +6,18 @@ Optimization under uncertainty for `Pyomo <https://pyomo.org>`_ models.
 `Documentation is available at readthedocs <https://mpi-sppy.readthedocs.io/en/latest/>`_ and
 a technical report is on `OOL <http://www.optimization-online.org/DB_HTML/2020/11/8088.html>`_
 
+NOTICE
+^^^^^^
+
+There was a disruptive change on August 11, 2022 concerning how
+options are accessed. See the file ``disruptions.txt`` for more
+information. The most recent release was done before the change so
+installation using pip is not yet affected. If you are a new user,
+this will not affect you, regardless of how you install. If you are an
+existing user, you should consider the disruption before updating to
+the latest mpi-sppy code on github. The documentation on readthedocs
+probably refers to the newest version.
+
 Status for internal tests
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 

disruptions.txt

@@ -0,0 +1,37 @@
+(Summer 2022) Conversion to Pyomo config will cause disruption for many users.
+
+Note: a  release was done just before this change so you can revert to the
+last release if you don't have time to deal with the disruption.
+
+See examples such as farmer.  There is a bash script called
+sed_config_help.bash that uses sed to do most of the things listed
+here for py, but after you run it, some edits are still needed.
+Also, you will have to edit slurm and bash scripts by hand.
+
+0. baseparsers should be replaced by config
+
+1. vanilla should be replaced by vanilla_cfg
+
+2. Command line arguments using "with" have had the "with_" removed
+   and the default is now False! (So you need, e.g. --xhatshuffle and --lagrangian)
+   There are no longer no_ options in config.py
+
+3. If you are using vanilla, the first element of beans should be cfg rather than args, where
+    cfg is of type Config
+
+4. For multi-stage problems using --branching-factors on the command line, the
+   branching factor list needs to be in quotes.
+
+5. zhat4xhat (and some other confidence interval code) needs --xhatpath on the command
+   line before the name of the npy file.
+
+Send an email to [email protected] if you want help converting.
+
+BTW: a release was done right before this change.
+
+========
+Documentation notes:
+
+In vanilla examples we have stopped using positional arguments for
+the number of scenarios (e.g., in Farmer and UC). Instead --num-scens
+is required.

doc/src/amalgamator.rst

@@ -23,7 +23,7 @@ problem, taking the amalgamator options as an input, and giving as an output
 additional arguments for the ``scenario_creator``. The amalgamator class
 is not flexible with respect to function names in the module.
 
-The ``options`` argument is a dictionary that specifies information 
+The ``cfg`` argument is a ``Config`` class object that specifies information 
 about the problem, and dictates the way Amalgamator runs. 
 It must contains the following attributes for use with cylinders:
 
@@ -49,10 +49,10 @@ It must contains the following attributes for use with cylinders:
 
 
 Create Amalgamator from a module and command line
--------------------------------------------------
-Given an options dictionary as above, ``amalgamator.Amalgamator_parser``
-calls the appropriate parsers from ``baseparsers.py`` and completes the options
-to add the necessary information for different modules.
+------------------------------------------------- Given an options
+``Config`` object (typically called `cfg`) as above,
+``amalgamator.Amalgamator_parser`` creates calls the appropriate
+functions to add the necessary information for different modules.
 
 The method ``amalgamator.from_module`` uses the two utilities described above.
 It takes as input a module name, and calls ``amalgamator.Amalgamator_parser``
@@ -63,7 +63,7 @@ runs an Amalgamator object.
 .. Note::
    The module must contains several methods:
    ``scenario_creator``, ``scenario_names_creator``, ``inparser_adder`` and
-   ``kw_creator``. The files ``examples.farmer.afarmer.py``, ``mpisppy.tests.examples.aircond.py`` contain
+   ``kw_creator``. The files ``examples.farmer.farmer.py``, ``mpisppy.tests.examples.aircond.py`` contain
    examples of these functions.
 
 The full options dictionary is passed through to ``kw_creator`` so keyword arguments for
@@ -72,8 +72,8 @@ scenario creation can be placed in the almalgamator options dictionary.
 Notes about ``inparser_adder``
 ------------------------------
 
-The function adds arguments unique to the instance. Note that `--branching-factors` can be added
-by this function if ``base_parsers.py`` has been used because it allows for conflict resolution.
+The function adds config arguments unique to the instance. Note that `--branching-factors` can be added
+even if something else added it, because the config.add functions bypass duplicates.
 
 
 Amalgamator with EF

doc/src/confidence_intervals.rst

@@ -54,15 +54,15 @@ The first step in computing a confidence interval is creating a ``MMWConfidenceI
 that takes as an argument an ``xhat_one`` and options.
 This object has a ``run`` method that returns a gap estimator and a confidence interval on the gap.
 
-Example
--------
+Examples
+--------
 
-An example of use with the ``farmer`` problem, can be found in the main of ``mmwci.py``.
+There are example scrips for sequential sampling in both ``farmer`` and ``aircond``.
 
 Using stand alone ``mmw_conf.py``
 ---------------------------------
 
-(Currently for use with 2-stage problem only; for multi-stage problems, instantiate an ``MMWConfidenceItervals`` object directly)
+(The stand-alone module is currently for use with 2-stage problem only; for multi-stage problems, instantiate an ``MMWConfidenceItervals`` object directly)
 
 ``mmw_conf`` uses the ``MMWConfidenceIntervals`` class from ``mmw_ci`` in order to construct a confidence interval on the optimality gap of a particular candidate solution ``xhat`` of a model instance. 
 
@@ -71,15 +71,15 @@ To use the stand along program a model compatible with ``Amalgamator`` and ``.np
 First, ensure that the model to be used is compatible with the
 ``Amalgamator`` class. This requires the model to have each of the
 following: a ``scenario_names_creator``, a ``scenario_creator``, an
-``inparser_adder``, and a ``kw_creator``. See ``afarmer.py`` in
+``inparser_adder``, and a ``kw_creator``. See ``farmer.py`` in
 ``examples`` for an example of an acceptable model. Note: the `options` dictionary
 passed to ``kw_creator`` will have the command line arguments object in `args`, which
 is not required (or used) by ``amalgamator`` but is used by confidence interval codes
 to be able to pass problem-specific args down without knowing what they are. 
 
 Once a model satisfies the requirement for amalgamator, next a ``.npy`` file should be constructed from the given model. This can be accomplished, for example, by adding the line 
 ``sputils.ef_ROOT_nonants_npy_serializer(instance, 'xhat.npy')`` after solving the ef ``instance``. When using ``Amalgamator`` to solve the program, this can be done by adding the line
-``sputils.ef_ROOT_nonants_npy_serializer(ama_object.ef, "xhat.npy")`` to your existing program (see the example in ``afarmer.py`` for an example of this).
+``sputils.ef_ROOT_nonants_npy_serializer(ama_object.ef, "xhat.npy")`` to your existing program (see the example in ``farmer.py`` for an example of this).
 
 Once this is accomplished, on the command line, run
 ``python -m mpisppy.confidence_intervals.mmw_conf my_model.py xhat.npy gurobi --num-scens n --alpha 0.95``. Note that ``xhat.npy`` is assumed to be in the same directory as ``my_model.py`` in this case. If the file is saved elsewhere then the corresponding path should be called on the command line.

doc/src/drivers.rst

@@ -20,13 +20,13 @@ The ``utils`` directory has utilities that set up command line options
 and create dictionaries used to create hubs and spokes. The main shared utilities
 are
 
-* ``baseparser.py`` that creates command line options.
-* ``vanilla.py`` that creates dictionaries used for hub and spoke
+* ``config.py`` that creates a Pyomo Config object and command line options.
+* ``cfg_vanilla.py`` (was ``vanilla.py``) that creates dictionaries used for hub and spoke
   creation. These dictionaries are ultimately fed to
   ``sputils.spin_the_wheel``.
 
 The constructors for the vanilla spokes take arguments that vary slightly depending
-on the spoke, but all want the args passed in by the args parser,
+on the spoke, but all want a configuration object,
 followed by ``scenario_creator`` function, a ``scenario_denoument`` function
 (that can be ``None``), a list of scenario names as ``all_scenario_names``,
 and ``scenario_creator_kwargs``. Other arguments can be seen in the file ``mpisppy.utils.vanilla.py``
@@ -42,7 +42,7 @@ will need to add extensions. Here are few examples:
 
 * In the ``farmer_cylinders.py`` example, there is a block of code to add a ``--crops-mult`` argument that is passed to the scenario create in the ``scenario_creator_kwargs`` dictionary.
 
-* In the ``hydro_cylinders.py`` example (which has three stages), ``baseparser.py`` is not used. The branching factors are obtained from the command line and passed to the scenario constructor via ``scenario_creator_kwargs`` and also passed to ``sputils.create_nodenames_from_BFs`` to create a node list.
+* In the ``hydro_cylinders.py`` example (which has three stages). The branching factors are obtained from the command line and passed to the scenario constructor via ``scenario_creator_kwargs`` and also passed to ``sputils.create_nodenames_from_BFs`` to create a node list.
 
 * The ``uc_cylinders.py`` example adds arguments that are used to provide data or trigger the inclusion of extensions. The  extension specifications and arguments are added to the dictionaries  (e.g., ``hub_dict``) create by ``vanilla.py``.
 
@@ -82,6 +82,11 @@ that new spokes and extensions can simply look for any options that
 they like. The disadvantage is that developers who use ``mpi-sppy``
 cannot count on it to detect spelling errors in options names.
 
+In contrast, ``Config`` objects will check spelling if you assign to
+an existing option, but functions like ``quick_assign`` are more popular and
+this function does will create a new option if the option does not
+exist.
+
 
 Contrasting ``_mpisppy_node_list`` and ``all_node_names``
 ---------------------------------------------------------

doc/src/examples.rst

@@ -403,17 +403,11 @@ unusual in that the model file, ``aircond.py``, lives in
 ``mpisppy.tests.examples`` directory. Scripts and bash files that use
 it live in ``examples.aircond``.  A good place to start is the
 ``aircond_cylinders.py`` file that starts with some functions that
-support the main program. The main program makes use of the command
-line parsers provided with the library supplemented by arguments
-provided by the aircond reference model using the line
+support the main program. The main program makes use of the 
+``Config`` object called `cfg` that creates a parser and gets arguments.
 
-::
-
-  parser = aircond.inparsers_adder(parser)
-
-
- The ``args`` obtained by the parser are passed directly to the vanilla hub
- and spoke creator which knows how to use the arguments from the ``baseparsers``.
+ The configuration data obtained by the parser are passed directly to the vanilla hub
+ and spoke creator which knows how to use the arguments from a ``Config`` object.
  The arguments unique to aircond are processed by the ``create_kwargs`` function
  in the reference model file.
 

doc/src/extensions.rst

@@ -51,7 +51,10 @@ fixer.py
 
 This extension provides methods for fixing variables (usually integers) for
 which all scenarios have agreed for some number of iterations. There
-is an example of its use in ``examples.sizes.sizes_demo.py``
+is an example of its use in ``examples.sizes.sizes_demo.py`` also
+in ``examples.sizes.uc_ama.py``. The ``uc_ama`` example illustrates
+that when ``amgalgamator`` is used ``"id_fix_list_fct"`` needs
+to be on the ``Config`` object so the amalgamator can find it.
 
 xhat
 ^^^^

doc/src/hubs.rst

@@ -9,16 +9,18 @@ and modification provided in the form of :ref:`Extensions`.  Many of
 the algorithms can be run in stand-alone mode (not as a hub with
 spokes), which is briefly described in :ref:`Drivers`.  Most hubs have
 an internal convergence metric, but the threshold option
-(``--intra-hub-conv-thresh`` in ``baseparsers.py`` or ``"convthresh"``
-in ``PHoptions``) is often set to a negative number so internal
+(``--intra-hub-conv-thresh`` on the command line, ``intra_hub_conv_thresh``
+in a ``Config`` object) is often set to a negative number so internal
 convergence is ignored in favor of the threshhold on the gap between
-upper and lower bounds as computed by the spokes (``--rel-gap`` and
-``--abs-gap`` in ``baseparsers.py``).  Most hubs can be terminated
-based on an iteration limit (``--max-iterations`` in ``baseparsers.py``).
+upper and lower bounds as computed by the spokes (``rel_gap`` and
+``abs_gap`` in ``Config`` object).  Most hubs can be terminated
+based on an iteration limit (``max_iterations`` in a ``Config`` object).
 
-An additional gap-based termination option is supported by ``baseparser.py`` and
-``vanilla.py``: ``--max-stalled-iters`` that specifies how many iterations
-can pass without an improvement to the gap between upper and lower bounds.
+An additional gap-based termination option is supported by
+``Config`` and ``cfg_vanilla.py``: ``max_stalled_iters``
+(``--max-stalled-iters`` on the command line) that specifies how many
+iterations can pass without an improvement to the gap between upper
+and lower bounds.
 
 PH
 --
@@ -34,11 +36,11 @@ Linearize proximal terms
 
 The proximal term can be approximated linearly using the PHoption
 `linearize_proximal_terms` (which is included as
-``--linearize-proximal-terms`` in ``baseparsers.py``). If this option
-is specified, then the option `proximal_linearization_tolerance`
-(which is ``--proximal-linearization-tolerance`` in
-``baseparsers.py``) is a parameter.  A cut will be added if the
-proximal term approximation is looser than this value (default 1e-1).
+``--linearize-proximal-terms``). If this option is specified, then the
+option `proximal_linearization_tolerance` (which is
+``--proximal-linearization-tolerance`` on the command line) is a parameter.
+A cut will be added if the proximal term approximation is looser than
+this value (default 1e-1).
 
 
 If only the binary terms should be 

doc/src/overview.rst

@@ -54,3 +54,14 @@ computing upper and lower bounds.
 Most developers using ``mpi-sppy`` will not need to concern themselves
 very much with the architecure because ``mpi-sppy`` can take
 care of the communcation aspects.
+
+Solver threads
+--------------
+
+To illustrate one of the issues associated with having a lot of things
+running at once, we note that you will often want to limit your
+solvers to two threads if they take essentially as many threads as
+they can get (e.g., cplex and gurobi). This is because if you have
+multiple cylinders each launching a solver and each solver wants to
+use every thread it can find, you may oversubscribe you computer
+rather badly.

doc/src/secretmenu.rst

@@ -1,7 +1,7 @@
 Secret Menu Items
 =================
 
-There are many options that are not exposed in ``mpisppy.utils.baseparsers.py`` and we list
+There are many options that are not exposed in ``mpisppy.utils.config.py`` and we list
 a few of them here.
 
 

doc/src/zhat.rst

@@ -13,7 +13,7 @@ Unless you are directly using mid-level functionality, your
 code may need to write the root node nonanticipative variable values
 (called `xhat` or `xhat_one`) to a file for later processing. This is
 typically done using ``sputils.ef_ROOT_nonants_npy_serializer``, which
-is shown in various examples, e.g., ``examples.farmer.afarmer.py``
+is shown in various examples, e.g., ``examples.farmer.farmer.py``
 
 zhat4xhat
 ---------
@@ -36,7 +36,7 @@ seedoffset
 
 Most of the confidence interval code assumes that is can pass in a
 seed, particularly to the ``scenario_creator`` function so that
-replicates can be obtained. See ``examples.farmer.afarmer.py`` for an
+replicates can be obtained. See ``examples.farmer.farmer.py`` for an
 example.
 
 When only a small number of scenarios are available, the

examples/acopf3/Readme.rst

@@ -2,7 +2,8 @@ CCOPF
 =====
 Contingency Constrained OPF
 
-This example is not fully developed as of October 2020.
+This example is not fully developed as of October 2020. As of May 2022, it executes
+but uses mid- and lower-level code.
 
 To use this, egret must be installed (also run
 egret/thirdparty/get_pglib_opf.py and then you still
@@ -12,21 +13,21 @@ might have to make sure the directories match.)
 EF
 ---
 
-- To get the EF use ``python ccopf_multistage.py bf1 bf2 iters scenperbund``
-  where bf1 and bf2 are branching factors (e.g. 2 3 for a small test).
+- To get the EF use ``python ccopf_multistage.py bf1 bf2 bf3
+  where bf1 and bf2 are branching factors (e.g. 2 3 1 for a small test).
 Use 1 and 0 for iters and bundles
 for a small test use
-python ccopf_multistage.py bf1 bf2 2 3 1 0
+python ccopf_multistage.py 2 3 1
 
   Edit the line in the py file that assigns `casename` to change the example that is run.
 
 PH
 --
   
 - To run with PH and a fixed scenario set for the upper bound use
-  ``mpiexec -np 2 python -m mpi4py ccopf2wood.py bf1 bf2 iters scenperbund``
+  ``mpiexec -np 2 python -m mpi4py ccopf2wood.py bf1 bf2 iters scenperbund solver``
   e.g.,
-  ``mpiexec -np 2 python -m mpi4py ccopf2wood.py 2 3 2 0``
+  ``mpiexec -np 2 python -m mpi4py ccopf2wood.py 2 3 2 0 cplex``
 
 The number of processors is restricted by the branching factors; basically, multiples of the
 first branching factor.

examples/acopf3/ccopf_multistage.py

@@ -115,7 +115,6 @@ def pysp2_callback(
         _egret_md (egret tuple with dict as [1]) egret model data
 
     """
-    print("Debug: convex_relaxation=",convex_relaxation)
     # pull the number off the end of the scenario name
     scen_num = sputils.extract_num(scenario_name)
 

examples/afew.py

@@ -40,23 +40,24 @@ def do_one(dirname, progname, np, argstring):
 
 # for farmer, the first arg is num_scens and is required
 do_one("farmer", "farmer_cylinders.py", 3,
-       "3 --bundles-per-rank=0 --max-iterations=50 "
-       "--default-rho=1 --with-display-convergence-detail "
-       "--solver-name={} --no-fwph --use-norm-rho-updater".format(solver_name))
+       "--num-scens=3 --bundles-per-rank=0 --max-iterations=50 "
+       "--default-rho=1 --display-convergence-detail "
+       "--solver-name={} --xhatshuffle --lagrangian --use-norm-rho-updater".format(solver_name))
 do_one("farmer", "farmer_lshapedhub.py", 2,
-       "3 --bundles-per-rank=0 --max-iterations=50 "
+       "--num-scens=3 --bundles-per-rank=0 --max-iterations=50 "
        "--solver-name={} --rel-gap=0.0 "
-       "--no-fwph --max-solver-threads=1".format(solver_name))
+       " --xhatlshaped --max-solver-threads=1".format(solver_name))
 do_one("sizes",
        "sizes_cylinders.py",
        4,
        "--num-scens=3 --bundles-per-rank=0 --max-iterations=5 "
        "--iter0-mipgap=0.01 --iterk-mipgap=0.001 --linearize-proximal-terms "
-       "--default-rho=1 --solver-name={} --with-display-progress".format(solver_name))
+       " --xhatshuffle --lagrangian --fwph "
+       "--default-rho=1 --solver-name={} --display-progress".format(solver_name))
 
 do_one("hydro", "hydro_cylinders_pysp.py", 3,
        "--bundles-per-rank=0 --max-iterations=100 "
-       "--default-rho=1 --with-xhatshuffle --with-lagrangian "
+       "--default-rho=1 --xhatshuffle --lagrangian "
        "--solver-name={}".format(solver_name))
 
 if len(badguys) > 0: