This post summaries an interesting paper called Thinking Clearly About Performance, by Cary Millsap.
- Trial-and-error approach of algebra:
- When solving
3x + 4 = 14, try
x = 2,
x = 3 and so on.
- May work sometimes, yet still not a correct way for algebra.
- Using it only indicates one is not thinking clearly about algebra.
- Axiomatic approach of algebra (and other things):
- Each step should be documented (and proven), reliable and repeatable.
- One small, logical, provable and auditable step at a time.
- Such steps form a proof of thinking clearly
- Proving skill is vital
- equally or more important than knowing
- a must for good consultant / leader / employee
Definition of performance
task: a unit of work
#seconds per task
#tasks per second
Diagnosis of performance - The users' claim is often about the response time - simple form: "It used to take only 1s" - complex form: "It's so ... slow" - First steps - State the problem clearly - Define the goal state - worse: when user does not understand a quantative goal - worse: when user's expectation is impossible to meet - Tool: sequence diagram - Tool: profile - pitfall: skew - non-uniformity in a list of values - eliminate 1/2 calls may not eliminate 1/2 execution time - skew histogram: group calls with (range of time / counts / total time) - Estimations: - cost - return - Amdahl's law: performance improvement is proportional to how much a program uses the improved thing - political capital: credibility in group - reliability of estimations
Efficiency - inverse measure of waste - waste: how much of service time can be eliminated - without adding capacity - without sacrificing function - do not adjust the program to accommodate inefficient programs
Load - Competition for resource induced by tasks - The reason that performance testing does not catch all problems in production - measurement: utilization (usage / capacity) - High load increases response time
Impacts of Load
Testing - may not catch all problems, still necessary - keep balance between "no test" and "complete emulation of production"
Measuring for optimization - Throughtput: easier to measure - Response time: happens at client side, and is harder to measure - People tend to measure what's easy to measure - Surrogate measures: - are easy to measure - may work sometimes - know their limitation and use them wisely - e.g. subroutine call counts, samples (i.e. not all) of subroutine execution durations