Recreate test appdomain between each benchmark

[Aaronontheweb]

To decrease the potential for test pollution as a result of previous tests leaving behind artifacts, changing static members, failing to clean up properly, etc... I would propose that we completely destroy and recreate the test AppDomain between each spec.

1 spec can do all of its runs and warmups within the domain, but as soon as it's done reporting its results we destroy it and move on.

This would require a redesign in how the TestRunner works, but I think there's merit to it. Any thoughts?

[mattwarren]

@Aaronontheweb Disclaimer: I work on BenchmarkDotNet, another .NET Benchmarking tool (with a slightly different focus to NBench).

Here's a bit of info/justification why BenchmarkDotNet doesn't just use separate AppDomains, but instead separate processes.

Not saying that NBench has to do it this way, I just thought you might find it interesting :-)

[Aaronontheweb]

Next, recall that the MS.NET GC is self-tuning. So, the execution of the first benchmark method in the first app domain can affect GC and execution of the second benchmark method in the second app domain.

@mattwarren heyo, I did not know that. That's definitely a good reason to consider an alternative approach like separate processes.

[Aaronontheweb]

Although, that being said - our goals are a bit different. Separate AppDomains might be "enough" since a lot of our users are using NBench for things other than micro-benchmarking... Just thinking out loud :p

[mattwarren]

Although, that being said - our goals are a bit different. Separate AppDomains might be "enough" since a lot of our users are using NBench for things other than micro-benchmarking...

Yeah I really like the functionality you have for ensuring a single unit test doesn't allocate more than 'X' bytes of memory or take longer than 'Y' seconds. BenchmarkDotNet will give you than info, but it doesn't integrate with unit tests, as you said we have different goals.

It's just good to see several .NET projects taking performance testing seriously!!

BTW I just found out that AppDoamins (probably) aren't going to be support on CoreCLR, see https://github.com/dotnet/coreclr/issues/642 for a discussion about why.