While reviewing usability tools and looking at the best way to test mobile (cell phone) apps I came across this handy summary that sums up how I feel about lab testing.
“Usability researchers and practitioners have been concerned that laboratory evaluations do not simulate the context where mobile phones are used (Johnson 1998) and lack the desired ecological validity. Interruptions, movement, noise, multitasking etc. (Tamminen et al. 2004) that could affect the users’ performance are not present in laboratory tests. The surrounding environment and mobility are assumed to set special requirements for mobile applications. Usability testing should take these requirements into account.”
Although the conclusion of this paper is that when testing mobile applications you will identify all the issues. The thing you will miss is the incidental observations about user behaviour – stepping out of a crowd to do complex tasks – but nothing that would effect the identification of problems the user has your application.
Usability Testing of Mobile Applications: A Comparison between Laboratory and Field Testing