Presented by

  • Peter Chubb

    Peter Chubb

    Peter has been using or contributing to the Unix and Linux systems since 1979. He currently works with the Trustworthy Systems group in Data61, CSIRO. Peter's expertise is in Linux kernel and low level systems software, and in measuring and improving these.


Whenever a developer makes a change to a body of software (adds a feature, makes a bugfix), that developer wants to know (and even more important, the upstream maintainers want to know) whether and how it affects system performance. Too often on the mailing lists I see the phrase 'The performance change is down in the noise' Two simple statistical tests can be used to determine whether a change is *really* down in the noise, or whether there's a real difference. Student's T test can tell you how likely it is that the mean of two sample sets is the same, but it works only on Normal (Gaussian) distributions. The Mann-Whitney U test tells you how likely it is for the median to be the same, and works on *any* underlying distribution. Kernel microbenchmarks usually show very skewed distributions, where the minimum and the mode are the same; but there is a long tail, with several small modal points in it. I've worked out the reason for this, and how to use the Mann-Whitney U test to determine whether small changes (too noisy to see by eyeballing the resutst) are real or not. Even though this talk has some statistics in it, I'm not a statistician, so it's all stuff that anyone with year 10 maths can understand.