Sep. 6th, 2012

holdthesky: (Default)
This is used in loads of places but I'm not sure what it's called (Amazon use it a lot), not even a slightest clue.

The problem is one of coming up with a succinct notion of performance for a repeating task which may well be highly non-normal (eg page requests on a site with very different kinds of pages). Even among the people who know what variance is and can calculate it, very few people grok it, and that usually has all kinds of normality assumptions built in.

The argument is that people don't pay much attention to speed (care much about it, anyway) unless something is over a certain threshold of slowness. Folk will also forgive slowness up to some percentage of experiences, but only a tiny proportion (say 1% or 0.1%).

So what people do is select a percentile (usually 99% or 99.9% depending on if a request is seen as "critical" or not) and ask how long a request takes at that percentile. And that's all that's reported, that one number.

For example, Amazon web services all negotiate performance by passing historical 99.9% response times. Aggregating services drop sub-requests (or fail back to backups) if the reported 99.9% figure is too high for them to sustain their own 99.9% contract with their own contract. This goes way deep into their code, not just at the services level. A contract might be "99.9% of pages within 1s".

You can extend this to things like memory usage, bandwidth, the latency of garbage collect pauses, and so on and, I think, even people's perceptions of brand quality generally. 99.9%, for my own personal experience, is a pretty constant figure in software terms.

I think this is a wonderful idea and I think the main reason people think Firefox is slower than Chrome: it's not because its sustained performance is slower (though it is) but because Firefox has an extremely rubbish and non-threaded javascript garbage collector which inserts massive pauses way to frequently to be ignorable at 99.9%.

So here's my problem. What's this whole idea called? I know Jeff Bezos is a great exponent of it but I very much doubt it was thats guy's original idea: it must be from some business guru or something? Or maybe some statistician who advocated it? Anyone know? When I use it I call it bezos_* at the moment, which I'd very much like to stop doing.


holdthesky: (Default)

July 2013

21222324 252627

Style Credit

Expand Cut Tags

No cut tags
Page generated Jun. 27th, 2017 07:16 pm
Powered by Dreamwidth Studios