Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add diff mode tests #513

Open
quasilyte opened this issue Jun 16, 2020 · 3 comments
Open

Add diff mode tests #513

quasilyte opened this issue Jun 16, 2020 · 3 comments

Comments

@quasilyte
Copy link
Contributor

noverify is only tested in normal mode where it reports all issues that were found.

The diff (git) mode is what may be the most important feature for the real use-cases.

We need to add at least some tests to a diff mode to make sure it works correctly and does not break. This is especially important since git_main is scheduled for refactoring.

I had several bug reports where diff was computed in an unexpected way. It's hard to fix these issues without having a foundation for verification (manual testing is meh).
@YuriyNasretdinov
Copy link
Contributor

I thought about this. The issue is that you need an external tool (git) to be able to test it. It is still possible, of course. Also it would probably be pretty hard to do a full test anyway as there are a lot of components there, e.g. a git fetch operation, or comparison to master branch. So, in short, the tests themselves would be pretty complex and susceptible to breakage as well as the code. Probably still better than relying on manual testing I guess, although I'm not entirely convinced.

@quasilyte
Copy link
Contributor Author

Isn't it a property of most integration tests?
They're not that simple, but they test the real system.
We can have a git repository in testdata.

If git is not available, this test can be skipped with t.Skip.

@quasilyte
Copy link
Contributor Author

It seems like my recent refactoring broke the diff mode. :(

I'm not used to do a lot of manual testing after CI is passing and when I do, it's usually the normal mode run + our bash script wrapper which not necessarily cover everything (it doesn't include full diff, for starters).

If diff mode will not be replaced by the baseline mode, it would be very useful to have some tests for it. :)

quasilyte added a commit that referenced this issue Jun 29, 2020
@quasilyte
cmd: fix git mode for the gitRepoComputeReportsFromCommits() case
dde6854 
Code was assigning to a wrong field so git.Zero was not replaced
by the "master" string. This led to a panic that is quite hard to
reproduce outside of the real-world environment.

To make this mistake less likely in future, I removed fields
from the struct and store these values as local values as they're
only needed inside that function scope.

gitRef looks unused. It was unused before the refactoring as well.
Maybe it was used before but we broke something a long time ago?
This calls for another investigation.

Refs #513

Signed-off-by: Iskander Sharipov <[email protected]>
@quasilyte quasilyte mentioned this issue Jun 29, 2020
Merged
quasilyte added a commit that referenced this issue Jun 29, 2020
@quasilyte
cmd: fix git mode for the gitRepoComputeReportsFromCommits() case (#532)
c1415b9 
Code was assigning to a wrong field so git.Zero was not replaced
by the "master" string. This led to a panic that is quite hard to
reproduce outside of the real-world environment.

To make this mistake less likely in future, I removed fields
from the struct and store these values as local values as they're
only needed inside that function scope.

gitRef looks unused. It was unused before the refactoring as well.
Maybe it was used before but we broke something a long time ago?
This calls for another investigation.

Refs #513

Signed-off-by: Iskander Sharipov <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@YuriyNasretdinov @quasilyte