Belos: OutputManager restricts printing to process zero

[rppawlo]

From some collaborators at ORNL:

It looks like the output manager in Belos hardwires MPI_COMM_WORLD, which means that if you are running on a unique communicator you cannot get any iteration output (unless your communicator contains rank 0 in MPI_COMM_WORLD - unfortunately ours does not).

We will also need this fixed for internal Sandia projects. We will have separate belos solvers running simultaneously on separate subcommunicators. The simplest thing to do is probably allow users to set the print rank (and possibly the comm). Then it could be made to work with FancyOStream usage which also controls the print rank.

@trilinos/belos

[aprokop]

@trilinos/belos

[mhoemmen]

Alas, we've known this for years. Anasazi has the same issue. I think @hkthorn even discussed a plan for fixing it a few years ago.

[aprokop]

@trilinos/xsdk Is this considered to be compatible with xSDK?

[hkthorn]

@mhoemmen @aprokop @rppawlo Yes, this is a known issue and I did discuss an idea earlier this year (around February 8th) about using Teuchos::Comm to address this issue. Here is what I sent to Mark:

'In light of the outputting issues for Belos and Anasazi when communicators (other than WORLD are used), I was thinking of amending the solver managers to have a method that will take a Teuchos::Comm. It would be optional and not manditory, and that Teuchos::Comm will only be passed to utilities used within the solvers to ensure safe "global" synchronization.'

I am open to hear opinions about this proposed path forward. If there are no objections, then I can move forward with those modifications in the beginning of FY18.

[mhoemmen]

@hkthorn It could be nicer if we added a hidden traits class that extracts a comm from the Operator. We could do this for Epetra as well as Tpetra that way, and other Operator implementations would just return null.

[hkthorn]

@mhoemmen Thinking about this further, I don't need a comm, just a boolean to indicate that output should be sent to the provided output stream ... So, the hidden traits could really be comm-free in the interface. Of course, the implementation would use the communicators provided by Epetra/Tpetra or any other implementation. Does that sound reasonable to you?

[hkthorn]

@mhoemmen Thought 2: As Roger suggested there is always the FancyOStream class in Teuchos that combines print formatting, print rank, and the output stream. Would it be better to migrate to using that instead of the just the ostream inside of Belos/Anasazi? It could avoid adding more interface methods to the OperatorTraits ...

[mhoemmen]

@hkthorn It's a good idea to use FancyOStream; I find it useful for tests. However, this doesn't solve the problem. Users still have to know which rank is Proc 0, and tell the FancyOStream about it. This really feels like something that users shouldn't have to think about.

Here's my proposal. First, here is an interface and default implementation:

template<class OperatorType>
struct OutputStreamTraits {
  static Teuchos::RCP<Teuchos::FancyOStream>
  getOutputStream (const OperatorType& op, const Teuchos::RCP<std::ostream> out)
  {
    // In the default implementation, we assume nothing about MPI.
    return Teuchos::getFancyOStream (out);
  }
};

Here is a sample Tpetra implementation:

template<class ST, class LO, class GO, class NT>
struct OutputStreamTraits<Tpetra::Operator<ST, LO, GO, NT> > {
  typedef Tpetra::Operator<ST, LO, GO, NT> operator_type;

  static Teuchos::RCP<Teuchos::FancyOStream>
  getOutputStream (const operator_type& op, const Teuchos::RCP<std::ostream> out)
  {
    Teuchos::RCP<Teuchos::FancyOStream> fancy = Teuchos::getFancyOStream (out);
    auto comm = op.getComm ();
    const int myRank = comm.is_null () ? 0 : comm->getRank ();
    const int numProcs = comm.is_null () ? 1 : comm->getSize ();
    fancy->setProcRankAndSize (myRank, numProcs);
    constexpr int rootRank = 0;
    // This is irreversible, but that's only a problem if the input std::ostream
    // is actually a Teuchos::FancyOStream on which this method has been
    // called before, with a different root rank.
    fancy->setOutputToRootOnly (rootRank);
    return fancy;
  }
};

Epetra and Thyra implementations would be comparable. Here is how to use the class in practice, to wrap std::cout:

Teuchos::RCP<Teuchos::FancyOStream> outPtr =
  OutputStreamTraits<OP>::getOutputStream (*A, Teuchos::rcpFromRef (std::cout));
Teuchos::FancyOStream& out = *outPtr;

out << "This output will only appear on Process 0 of A's communicator." << std::endl;

What do you think? Note that I didn't abstract out getting myRank and numProcs, because I didn't want to encode an assumption of MPI.

[bartlettroscoe]

Epetra and Thyra implementations would be comparable.

All Thyra classes derive from Teuchos::VerboseObjectBase which allows clients (anyone) to set the FancyOStream objects on them (and a default global one for all objects and all classes of objects). That solves the problem for Thyra objects. And Thyra wrappers for Belos can just give the FancyOStream object they were given to the Belos objects. The Belos interfaces just need to accept a FancyOStream object (so they can use Teuchos::OSTab to help format the output) and that is it.

That is the best solution, in my opinion.

[hkthorn]

@mhoemmen @bartlettroscoe Thanks for the excellent ideas!

[github-actions]

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and remove the MARKED_FOR_CLOSURE label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE.