I Had an AI Adversarially Audit All 415 of My WSET Level 2 Questions. It Found 3 Errors.
Every quiz app calls its questions accurate. I had an adversarial AI audit read all 415 of mine and try to break them. Here is what it found.
Every wine quiz app calls its questions accurate. Mine says it too. The difference is that I can show you what accurate cost me. I built the WSET Level 2 question bank behind WineQuiz myself, and on 18 July I pointed an adversarial AI audit at all 415 active questions and told it to try to break them. It sustained three real errors. I fixed all three the same day.
The First Audit Was About Position, Not Content
The bank had already been through one audit, back in April. That one did not check facts. It checked where the correct answers sat.
Eighty-nine percent of my correct answers were parked on option A or B. A candidate who knew nothing about wine and guessed B on every question would have scored around 45 percent, which is not far off the pass line. That is not a wine test. That is a pattern a test-wise student can game without opening the book.
So the whole bank was rebalanced until each of the four positions holds close to 25 percent. No answer changed. Only its seat did. That fix mattered, but it was mechanical. It did not tell me whether the questions themselves were right.
What an Adversarial Audit Actually Does
The July audit was a different kind of check. The instruction was not "confirm these look fine." It was "assume each question is wrong and prove it."
Seventeen independent reviewers each took a batch of the 415 questions and read them against a correctness rubric. Between them they raised nine flags. Then every flag was attacked. Each of the nine went to three independent refuters whose only job was to knock it down.
Six flags died there. They looked like errors and were not, or the objection did not survive contact with the actual WSET Level 2 syllabus. Three survived every refutation. Those were the real ones.
The Three That Survived
The first was a serving temperature. I had keyed a dry Sercial Madeira into the well-chilled band, alongside sparkling and Fino. A medium-bodied fortified wine like that belongs in the lightly chilled band, next to Amontillado. Wrong key, corrected.
The second asked which region is known for Ribera del Duero, then offered four grape varieties as the options. The stem and the choices were not answering the same question. I reworded the stem to ask for the grape. Tempranillo was always the answer. Now it matches what the question asks.
The third contained its own answer. "Which grape dominates South Africa's Chenin Blanc" hands you the grape in the prompt. That is not a question, it is a gift. I reworded it to ask for South Africa's most-planted signature white and left the answer alone.
Why "Written by an Expert" and "Verified" Are Different Claims
I hold a WSET Level 2 Merit. I wrote these questions from the syllabus, built around the reasoning I made the bank to train rather than pure recall. That is the "written by an expert" claim, and it is worth something. It is also not the same claim as "verified."
"Written by an expert" means one person applied their knowledge once. "Verified" means the work was handed to people whose only job was to find what the author missed, and the survivors were fixed. An expert who writes 415 questions will get some of them wrong. I got at least three wrong, and I am the person who cares most about this bank.
The point is not that the author is fallible. Everyone knows that already. The point is whether anyone checked, adversarially, and then told you the number. Most of the gap between a Merit and a Distinction is reasoning, and reasoning practice is worthless if the key is quietly wrong.
What Checking Looks Like as a Number
Three errors in 415 is a rate I can live with, and it is a rate I can only quote because someone counted. Nine flags raised, six refuted, three sustained, all fixed the same day. That is the whole story, and I would rather publish the number than the adjective.
If you want to judge the questions before you take my word for any of this, there is a free 25-question mock at winequiz.gjdrinker.com/mini-mock, no signup. And if you want the full audit trail behind that number, I wrote up how the bank is checked at winequiz.gjdrinker.com/how-its-certified.