Should we change TODO tests to behave like Test::More’s TODO tests?

[drmuey]

Howdy,

If a test has this:

TODO: { local $TODO="derp"; ok(0) }

The failure is considered success:

not ok 2 # TODO derp
#   Failed (TODO) test at todo.t line 1.
ok
All tests successful.

Then if it starts passing (TODO: { local $TODO="derp"; ok(1) }) it is marked as such:

ok 2 # TODO derp
ok
All tests successful.

Test Summary Report
-------------------
todo.t (Wstat: 0 Tests: 2 Failed: 0)
  TODO passed:   2
Files=1, Tests=2,  0 wallclock secs ( 0.04 usr  0.00 sys +  0.08 cusr  0.01 csys =  0.13 CPU)
Result: PASS

That will trigger systems that something is amiss because a test started passing that shouldn’t.

it "should be tested"; behaves like the latter:

ok 2 - example should be tested # TODO (unimplemented)
1..2
ok
All tests successful.

Test Summary Report
-------------------
spec.t (Wstat: 0 Tests: 2 Failed: 0)
  TODO passed:   2
Files=1, Tests=2,  0 wallclock secs ( 0.04 usr  0.00 sys +  0.16 cusr  0.01 csys =  0.21 CPU)
Result: PASS

Flagging that a test is now passing unexpectedly.

It feels like that should instead behave like an actual TODO since its definitely not passing since its not implemented.

Thoughts?