Wine judging scrutinized after study finds inconsistent results(1)
Judges had no idea that one of the wines in each of the three flights given them was identical. During the first two flights, they rejected the wine as undeserving of any kind of award. During the third flight, however, they unanimously agreed that it warranted a gold medal.
Great for the wine, not so great for the reputation of the judges. I may have been one of them. If so, how can I explain this inconsistency without sounding dismissive or defensive?
And if I were one of the oscillating judges, will I ever be asked to judge again?
This quiet test took place at the California State Fair commercial wine competition in 2003. It was part of a trial run for a proposed study to measure the reliability of wine judges.
Since then, the study was refined and then conducted over four years. It drew little fanfare until researcher Robert Hodgson announced his initial findings in a paper published earlier this year in the Journal of Wine Economics.
In short, judges who have been evaluating wines in the State Fair's commercial wine competition in recent years haven't been very reliable in identifying consistently duplicate pours. Only 10 percent of the judges gave the same wine an identical score or close to identical score, Hodgson found. What's more, it seems judges weren't always consistent.
This doesn't look good for the credibility of the State Fair wine competition. If judges can't identify the same wine if it is in three of the 30 or so glasses in front of them, how much confidence should consumers put in the awards?
Of two minds on judging
I approach this subject with conflicted sentiments. For one, as a member of the State Fair's wine advisory committee, as well as an occasional judge at the State Fair, I was aware of Hodgson's research. I welcomed his study even while recognizing that ultimately I might be found to be not as keen a student of wine as I'd like to think. (Only Hodgson and the chief judge of the State Fair competition, G.M. "Pooch" Pucilowski, know how individual judges performed during the study; eventually, judges found to be the most inconsistent likely won't be invited back.)
Secondly, while I believe wine competitions perform a helpful if nonessential service in pointing consumers to commendable wines, I have qualms about the process. As I've sniffed, sipped and spat my way through 100 or 200 wines in a day, I've found myself wondering at what point do I hit the wall in this marathon: 50 wines, 95 or 125, somewhere in between, or beyond? When does my palate become too fatigued?
There are other questions judges mull as they try to be fair to the flight arranged invitingly in front of them. Did that olive I just ate really do its job in reviving my palate or does its lingering flavor cause me to rate the next wine as worthy of only a bronze medal rather than the gold my neighbor judge gave it? What if the room is too hot, cold, too dimly lit or too bright?
Despite these concerns, we judges shoulder on. Frankly, it's fun. All we know of the wines is their varietal or style, and perhaps vintage and price niche; such blind evaluations effectively diminish or eliminate preconceived notions. As a wine writer, competitions provide me with material.
And competitions are educational. Panels often consist of this kind of mix: A wine collector, a wine writer, a winemaker and a wine dealer such as grocer, importer or sommelier. Each generally has long experience on the wine scene, each has his or her own take on what constitutes an exceptional wine, and each often brings to the table fresh insight.
Each competition has the air of a family reunion. Many of the same judges are repeatedly invited, while at the same time others show up for the first time. We mingle, we swap news, we learn from each other.
While competitions benefit by the "institutional memory" of seasoned judges, this homogeneity in the composition of many panels is another reason to question the results: Do they too much reflect the tastes of older white guys? Some competition directors are responding to concerns that panels aren't as representative of the country's demographic profile as they might be by recruiting more women, more younger judges and more members of ethnic minorities.
