

## Highlights

This release is focused mainly on bug fixes that were affecting user experience. But don't worry, we have some new features too!

* **Diagnosers**
  * ConcurrencyVisualizerProfiler (allows profiling benchmarks on Windows and exporting the data to a trace file which can be opened with Concurrency Visualizer)
* **Command-line:**
  * `--stopOnFirstError`: Stops the benchmarks execution on first error. [#947](https://github.com/dotnet/BenchmarkDotNet/pull/947)
  * `--statisticalTest`: Performs a Mann–Whitney Statistical Test for identifying regressions and improvements. [#960](https://github.com/dotnet/BenchmarkDotNet/pull/960)
* **Bug fixes:**
  * Dry mode doesn't work because of the ZeroMeasurementHelper [#943](https://github.com/dotnet/BenchmarkDotNet/issues/943)
  * MannWhitneyTest fails when comparing statistics of different sample size [#948](https://github.com/dotnet/BenchmarkDotNet/issues/948) and [#950](https://github.com/dotnet/BenchmarkDotNet/issues/950)
  * Improve the dynamic loading of Diagnostics package [#955](https://github.com/dotnet/BenchmarkDotNet/issues/955)
  * BenchmarkRunner.RunUrl throws NRE when Config is not provided [#961](https://github.com/dotnet/BenchmarkDotNet/issues/961)
  * Don't require the users to do manual installation of TraceEvent when using Diagnostics package [#962](https://github.com/dotnet/BenchmarkDotNet/issues/962)
  * Stop benchmark after closing application + Flush log after stopping benchmark [#963](https://github.com/dotnet/BenchmarkDotNet/issues/963)

---

## Diagnosers

### ConcurrencyVisualizerProfiler

`ConcurrencyVisualizerProfiler` allows to profile the benchmarked .NET code on Windows and exports the data to a CVTrace file which can be opened with [Concurrency Visualizer](https://learn.microsoft.com/visualstudio/profiling/concurrency-visualizer).

`ConcurrencyVisualizerProfiler` uses `EtwProfiler` to get a `.etl` file which still can be opened with [PerfView](https://github.com/Microsoft/perfview) or [Windows Performance Analyzer](https://learn.microsoft.com/windows-hardware/test/wpt/windows-performance-analyzer). The difference is that it also enables all Task and Thread related ETW Providers and exports a simple `xml` which can be opened with Visual Studio if you install [Concurrency Visualizer plugin](https://marketplace.visualstudio.com/items?itemName=Diagnostics.ConcurrencyVisualizer2017)

![open trace](https://user-images.githubusercontent.com/6011991/48638184-2b13fe00-e9d0-11e8-8a94-0e951e4606ae.png)

![Utilization](https://user-images.githubusercontent.com/6011991/48638108-f6a04200-e9cf-11e8-8f7c-feda0a99138c.png)

![Threads](https://user-images.githubusercontent.com/6011991/48638120-fb64f600-e9cf-11e8-9e2d-53b615bfe9a8.png)


* [#964](https://github.com/dotnet/BenchmarkDotNet/issues/964) Concurrency Visualizer Profiler Diagnoser (by [@adamsitnik](https://github.com/adamsitnik))
* [dfb3c89](https://github.com/dotnet/BenchmarkDotNet/commit/dfb3c8912505799a76b0eb5ae0c082bb44599fa7) ConcurrencyVisualizerProfiler diagnoser! (by [@adamsitnik](https://github.com/adamsitnik))

---

## Command-line

In this release, we have some new command-line arguments!

### `--stopOnFirstError`:  Stops the benchmarks execution on first error

When provided, BenchmarkDotNet is going to stop the benchmarks execution on first error.

* [#947](https://github.com/dotnet/BenchmarkDotNet/pull/947) Add option to stop running when the first benchmark fails (by [@wojtpl2](https://github.com/wojtpl2))

### `--statisticalTest`: Statistical Test

To perform a Mann–Whitney U Test and display the results in a dedicated column you need to provide the Threshold via
`--statisticalTest`. Examples: 5%, 10ms, 100ns, 1s.

Example: run Mann–Whitney U test with relative ratio of 1% for all benchmarks for .NET 4.6 (base), .NET Core 2.0  and .NET Core 2.1.

```cs
class Program
{
    static void Main(string[] args) => BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
}

public class MySample
{
    [Benchmark]
    public void Sleep()
    {
#if NETFRAMEWORK
        Thread.Sleep(50);
#elif NETCOREAPP2_0
        Thread.Sleep(45);
#elif NETCOREAPP2_1
        Thread.Sleep(55);
#endif
    }
    
    [Benchmark]
    public void Same() => Thread.Sleep(50);
}
```

```log
dotnet run -c Release -f netcoreapp2.1 --filter * --runtimes net46 netcoreapp2.0 netcoreapp2.1 --statisticalTest 1%
```

**Note:** .NET 4.6 will be our baseline because it was provided as first on the runtimes list.


| Method | Runtime |     Toolchain |     Mean |     Error |    StdDev | Ratio | MannWhitney(1%) |
|------- |-------- |-------------- |---------:|----------:|----------:|------:|---------------- |
|  Sleep |     Clr |         net46 | 50.51 ms | 0.1833 ms | 0.1714 ms |  1.00 |            Base |
|  Sleep |    Core | netcoreapp2.0 | 45.53 ms | 0.1262 ms | 0.1181 ms |  0.90 |          Faster |
|  Sleep |    Core | netcoreapp2.1 | 55.50 ms | 0.1217 ms | 0.1138 ms |  1.10 |          Slower |
|        |         |               |          |           |           |       |                 |
|   Same |     Clr |         net46 | 50.47 ms | 0.1795 ms | 0.1679 ms |  1.00 |            Base |
|   Same |    Core | netcoreapp2.0 | 50.55 ms | 0.1873 ms | 0.1752 ms |  1.00 |            Same |
|   Same |    Core | netcoreapp2.1 | 50.55 ms | 0.2162 ms | 0.2022 ms |  1.00 |            Same |
