v2.4.0 – Welch's T-test statistical performance testing

I'm happy to release another minor version of Expecto with a new performance-test feature – namely Expect.isFasterThan.

Don't let the name mislead you. It's actually a more profound test that builds on Student's T-test by using Welch's T-test for validating the hypothesis that the two passed functions are equally fast. Here's a sample:

    testSequenced (testCase "matrix" <| fun _ ->
      let n = 100
      let rand = Random 123
      let a = Array2D.init n n (fun _ _ -> rand.NextDouble())
      let b = Array2D.init n n (fun _ _ -> rand.NextDouble())
      let c = Array2D.zeroCreate n n

      let reset() =
        for i = 0 to n-1 do
            for j = 0 to n-1 do
              c.[i,j] <- 0.0

      let mulIJK() =
        for i = 0 to n-1 do
          for j = 0 to n-1 do
            for k = 0 to n-1 do
              c.[i,k] <- c.[i,k] + a.[i,j] * b.[j,k]

      let mulIKJ() =
        for i = 0 to n-1 do
          for k = 0 to n-1 do
            let mutable t = 0.0
            for j = 0 to n-1 do
              t <- t + a.[i,j] * b.[j,k]
            c.[i,k] <- t
      Expect.isFasterThanSub (fun measurer -> reset(); measurer mulIKJ ())
                             (fun measurer -> reset(); measurer mulIJK ())
                             "ikj faster than ijk")

More information about this feature can be found in the docs.

A big thanks to @AnthonyLloyd for contributing this code!

Happy performance testing!