Very interesting discussion. When ranburr referred to "unbiased" in Post #1 he was clearly talking about the initial selection of the sample, not whether the reviewers themselves were neutral or unbiased as individuals. When more than one reviewer is involved, obvious differences (e.g, experience, technical knowledge, personal property, hunter, birder, naturalist, astronomer, etc.) need to be kept track of and represented in the results. It makes little sense (IMHO) to simply average over such evaluator differences, because it may well be that hunters see things differently than birders, people who own Leicas differently than those who own Swaros, etc. Moreover, since people vary greatly in how they scale observed differences, care must be taken to normalize for this factor. So, IMO, the meaningfulness of any review is largely conditional upon the populations that the reviewers represent, as well as the skill of the analyst to tease out product differences from potential observer "biases". BTW, observer biases are a fact of life. They are only "bad" when misleading (i.e., invalid) conclusions are drawn by unskilled analysts. Finally, as Ilkka implied, it would be very advantageous for a a thoroughgoing product review to include a sample of at least three binoculars of each type so that the results can be interpreted in the light of within-product as well as between-product variation.
Elkcub
