It's (not) down in the noise.

Whenever a developer makes a change to a body of software (adds a feature, makes a bugfix), that developer wants to know (and even more important, the upstream maintainers want to know) whether and how it affects system performance. Too often on the mailing lists I see the phrase 'The performance change is down in the noise' Two simple statistical tests can be used to determine whether a change is *really* down in the noise, or whether there's a real difference. Student's T test can tell you how likely it is that the mean of two sample sets is the same, but it works only on Normal (Gaussian) distributions. The Mann-Whitney U test tells you how likely it is for the median to be the same, and works on *any* underlying distribution. Kernel microbenchmarks usually show very skewed distributions, where the minimum and the mode are the same; but there is a long tail, with several small modal points in it. I've worked out the reason for this, and how to use the Mann-Whitney U test to determine whether small changes (too noisy to see by eyeballing the resutst) are real or not. Even though this talk has some statistics in it, I'm not a statistician, so it's all stuff that anyone with year 10 maths can understand.