benchmarkdotnet - powerful .net library for benchmarking

BENCHMARKDOTNET - POWERFUL .NET LIBRARY

FOR BENCHMARKINGLarry Nung

AGENDAIntroductionGetting startedJobsColumnsDiagnosersExportersParamsSetupBaselineReferenceQ & A 2

INTRODUCTION3

INTRODUCTION A powerful .NET library for benchmarking

Standard benchmarking routine: generating an isolated project per each benchmark method; auto-selection of iteration amount; warmup; overhead evaluation; statistics calculation; and so on.

Supported runtimes: Full .NET Framework, .NET Core (RTM), Mono

Supported languages: C#, F#, and Visual Basic Supported OS: Windows, Linux, MacOS Easy way to compare different environments

(x86 vs x64, LegacyJit vs RyuJit, and so on; see: Jobs) Reports: markdown, csv, html, plain text, png plots. Advanced features: Baseline, Params Powerful diagnostics based on ETW events

(see BenchmarkDotNet.Diagnostics.Windows)4

GETTING STARTED5

INSTALLATION Install-Package BenchmarkDotNet

6

INSTALLATION

7

INSTALLATION

8

INSTALLATION

9

INSTALLATION

10

WRITE CODE TO BENCHMARKusing BenchmarkDotNet.Attributes; … public class ProgramBenchmarker { protected Program m_Program { get;

set; } = new Program(); [Benchmark] public void Test() { m_Program.Test(); } } …

11

RUN THE BENCHMARKusing BenchmarkDotNet.Running; … public class Program { static void Main(string[] args) { var summary =

BenchmarkRunner.Run<ProgramBenchmarker>();

} public void Test() {… } } …}

12

VIEW RESULTS

13

ANALYZE RESULTS

14

JOBS15

JOBS Describes how to run your benchmark

Avaliable Jobs DryJob ClrJob CoreJob MonoJob LegacyJitX86Job LegacyJitX64 RyuJitX64Job SimpleJob LongRunJob MediumRunJob ShortRunJob VeryLongRunJob

16

JOBSusing BenchmarkDotNet.Attributes; using BenchmarkDotNet.Attributes.Jobs; ... [ShortRunJob] public class ProgramBenchmarker { ... }

17

JOBS

18

COLUMNS19

COLUMNS A column in the summary table

Available Columns NamespaceColumn MedianColumn MinColumn MaxColumn RankColumn e.t.c

20

COLUMNSusing BenchmarkDotNet.Attributes.Columns;…[NamespaceColumn] [MedianColumn] [MinColumn] [MaxColumn] [RankColumn][RankColumn(NumeralSystem.Roman)] [OrderProvider(SummaryOrderPolicy.FastestToSlowest)] public class ProgramBenchmarker { …} 21

COLUMNS

22

COLUMNSusing BenchmarkDotNet.Attributes; using BenchmarkDotNet.Columns; using BenchmarkDotNet.Configs; …[Config(typeof(Config))] public class ProgramBenchmarker { private class Config : ManualConfig { public Config() { Add(new TagColumn("HashCode", item =>

item.GetHashCode().ToString())); } } …}

23

COLUMNS

24

COLUMNSusing BenchmarkDotNet.Columns; using BenchmarkDotNet.Reports; using BenchmarkDotNet.Running;…public class HashCodeColumn : IColumn { public string ColumnName { get; } = "HashCode"; public HashCodeColumn() { } public bool IsDefault(Summary summary, Benchmark benchmark)

=> false; public string GetValue(Summary summary, Benchmark benchmark)

=> benchmark.Target.Method.Name.GetHashCode().ToString(); public bool IsAvailable(Summary summary) => true; public bool AlwaysShow => true; public ColumnCategory Category => ColumnCategory.Custom; public string Id { get; } = "1"; public int PriorityInCategory { get; } = 0; public override string ToString() => ColumnName; }

25

COLUMNS…[Config(typeof(Config))] public class ProgramBenchmarker { private class Config : ManualConfig { public Config() { Add(new HashCodeColumn())); } } …}

26

DIAGNOSERS27

DIAGNOSERS Can attach to your benchmark and get some

useful info

Available Diagnosers MemoryDiagnoser

28

DIAGNOSERSusing BenchmarkDotNet.Attributes; using BenchmarkDotNet.Configs; using BenchmarkDotNet.Diagnosers; ... [Config(typeof(Config))] public class ProgramBenchmarker { private class Config : ManualConfig { public Config() { Add(MemoryDiagnoser.Default); } } ... }

29

DIAGNOSERSusing BenchmarkDotNet.Attributes; using BenchmarkDotNet.Configs; using BenchmarkDotNet.Diagnosers; ... [MemoryDiagnoser] public class ProgramBenchmarker { ... }

30

DIAGNOSERS

31

EXPORTERS32

EXPORTERS Allows you to export results of your benchmark in

different formats.By default, files with results will be located in .\BenchmarkDotNet.Artifacts\results directory.

Available Exporters HtmlExporter CsvExporter MarkdownExporter AsciiDocExporter CsvMeasurementsExporter PlainExporter JsonExporter 33

CONFIGusing

BenchmarkDotNet.Attributes.Exporters;…[AsciiDocExporter] [CsvMeasurementsExporter] [PlainExporter] [JsonExporter] public class ProgramBenchmarker { …}

34

CONFIGusing BenchmarkDotNet.Configs; using BenchmarkDotNet.Exporters; using BenchmarkDotNet.Exporters.Csv; using BenchmarkDotNet.Exporters.Json; …[Config(typeof(Config))] public class ProgramBenchmarker { private class Config : ManualConfig { public Config() { Add(AsciiDocExporter.Default); Add(CsvMeasurementsExporter.Default); Add(PlainExporter.Default); Add(JsonExporter.Default); } } …}

35

HTMLEXPORTER

36

MARKDOWNEXPORTER

37

CSVEXPORTER

38

ASCIIDOCEXPORTER

39

CSVMEASUREMENTSEXPORTER

40

PLAINEXPORTER

41

JSONEXPORTER

42

PARAMS43

PARAMSusing BenchmarkDotNet.Attributes;... public class ProgramBenchmarker { [Params(100, 200)] public int Parameter { get; set; } protected Program m_Program { get; set; }

= new Program(); [Benchmark] public void Test() { m_Program.Test(); } } 44

PARAMS

45

SETUP46

SETUPusing BenchmarkDotNet.Attributes; ... public class ProgramBenchmarker { protected Program m_Program { get; set; } [Setup] public void Setup() { m_Program = new Program(); } [Benchmark] public void Test() { m_Program.Test(); } }

47

BASELINE48

BASELINEusing System.Threading; using BenchmarkDotNet.Attributes; ... public class ProgramBenchmarker { protected Program m_Program { get; set; } = new Program(); [Benchmark(Baseline = true)] public void Test1() { m_Program.Test(); Thread.Sleep(10); } [Benchmark] public void Test2() { m_Program.Test(); Thread.Sleep(20); } }

49

BASELINE

50

REFERENCE51

REFERENCE NuGet Gallery | BenchmarkDotNet 0.10.3

https://www.nuget.org/packages/BenchmarkDotNet

dotnet/BenchmarkDotNet: Powerful .NET library for benchmarking https://github.com/dotnet/BenchmarkDotNet

Home - BenchmarkDotNet Documentation http://benchmarkdotnet.org/

52

Q&A53

QUESTION & ANSWER

54