PG 2.19 answerHints issue with custom graders which check correctness by checking conditions

[taniwallach]

The changes made in #974 are causing trouble for me in problems where answerHints is used after a custom grader which checks answers by checking conditions (in particular in the sample below checking if the student's answer satisfies an ODE). The sample is based on a real question but with all the dependence on random parameters and earlier parts of the question dropped.

I run into two issues:

1. The answerHints is now running even on correct answers for "wrong" answers which are not "CODE", which it did not in the past.
   • That is due to a dropped condition in the changes in fix bugs with answerHints from issue #964 #974 .
   • As a result, incorrect messages are given in WW/PG 2.19 which were not given on 2.18.
2. Now that answerHints is processing the wrong answers in such a case - something is going wrong with how the "wrong" answers are tested to determine which feedback message to give.
   • It seems like this type of custom grader does not behave all that well with AnswerHints.
   • This was not obvious to me in the past (in the question where I ran into this), as many incorrect answers trigger a message with Value->Error so do not get a revised message.
   • The AnswerHints code in one of the real problems where this came up was trying to catch DNE and NONE and that apparently did not work even in the past.
   • Another issue occurred in a quite different setting using adaptive parameters in a custom grader.

Sample problem

DOCUMENT();

loadMacros('PGstandard.pl','MathObjects.pl','answerHints.pl');

Context("Numeric");

$ans6c = Compute('(-1/3)*exp(-5x)'); # Needs grader as solution of non-homogeneous ODE

Context()->texStrings;
BEGIN_TEXT
Find a solution to:
\[ y'' + 3 y' - 4 y = -2 e^{-5x} \]
\{ $ans6c->ans_rule(10) \}
END_TEXT
Context()->normalStrings;

ANS( $ans6c->cmp( 
  bypass_equivalence_test => 1,
  checker => sub {
      my ( $correct, $student, $ans ) = @_;

      # Try to see if it satisfies the ODE
      $Y0 = Formula( $student );
      $Y1 = $Y0->D('x');
      $Y2 = $Y1->D('x');

      $toCheck = $Y2 + 3 * $Y1 - 4 * $Y0 - Formula( '-2 exp(-5x)' );

      if ( Formula(0) != $toCheck ) {
        $ans->{ans_message} = 'That is not a solution of the non-homogeneous equation.' unless $ans->{isPreview};
        return 0;
      } else {
        return 1;
      }
      return 0;
  }

)->withPostFilter(AnswerHints(
  # The message on the first uncommented line below gets given on correct answers.
  Compute("DNE")  => [ 'There is a solution.', checkTypes => 0 ], 
  Formula('sin(x)') => [ 'Junk message - sin(x).' ],
  Formula('exp(x)') => [ 'Junk message - exp(x).' ],
)));


ENDDOCUMENT();

Give -exp(-5x)/3 to see these issues. With the current (PG 2.19) version of answerHints.pl an incorrect message is triggered. With the PG 2.18 version - no message is triggered.

Another correct answer is `-exp(-5x)/3 + exp(-4x)'.

[somiaj]

Not a solution to this actual issue but you have comments about $ans->{ans_message} not working, and that is because the inputs to MathObject answer checkers are just my ($correct, $student, $ans) = @_. The instructor hass an extra $self in there which is used for MultiAnswer checkers, but not for MathObject checkers.

[taniwallach]

I have what seems to be a fix in a PR I am about to submit.
I have not been writing much in the way of custom graders recently, so must have messed up when trying to get $ans->{ans_message} to work now as an alternate to the Value->Error approach.

I fixed up the code in the first message so it now uses $ans->{ans_message} properly. Thanks @somiaj

[somiaj]

@taniwallach unless $self->{isPreview}; that should be unless $ans->{isPreview}; now.

[taniwallach]

About the second issue - that this type of grader causes unexpected choices of messages in answerHints, the best fix I could come up with was to add just after the my ( $correct, $student, $ans ) = @_; an alternative "grading" option for use by the answerHints code.

      # Don't use the special grading code for answerHints, so it
      # will assign messages only when the answer matches the
      # "wrong" answer from the list.
      return( $correct == $student) if ($ans->{_filter_name} eq 'Answer Hints Post Filter');

It seems to me that the only real thing to possible do is to add to the documentation about answerHints.pl and about custom graders mention of this.

This type of fix also prevents incorrect error messages for correct answers, but does require editing each problem. As such, I think that the patch from #1171 is still the best solution to the first issue, as it will fix backwards compatibility of not giving messages for correct answers without explicitly allowing them.

[taniwallach]

@taniwallach unless $self->{isPreview}; that should be unless $ans->{isPreview}; now.

Fixed that also above. Sorry

[somiaj]

Note, if a user is already writing their own answer checker, the answer checker itself could also be generating the message hints (this is what I do in my answer checkers) and not use answerHints.pl. I do agree that maybe you should document this behavior so if someone wants to use answerHints.pl with a custom answer checker, they understand what needs to be done, but my personal suggestion is just not use that macro and instead include any messages/hints you want as part of the custom checker.

[somiaj]

  # Don't use the special grading code for answerHints, so it
  # will assign messages only when the answer matches the
  # "wrong" answer from the list.
  return( $correct == $student) if ($ans->{_filter_name} eq 'Answer Hints Post Filter');

I can confirm that this plus your fixes in #1171 gets the answerHints macro working properly.

[dpvc]

I have made an extended comment in #1171 that explains why the original checker function does not work with AnswerHints, and some ways to modify the checker function to make it work properly with AnswerHints without additional changes to AnswerHints itself.

[taniwallach]

Bottom line: The PG 2.19 changes to answerHints does change behavior, and for "poorly coded" problems can trigger undesirable behavior. The recommended resolution is to expect the problem code to be fixed.

The documentation and sample problems should be improved to try to guide people away from the pitfalls.

[drgrice1]

@lahvak: No. You are reading something wrong.