AI Open-End Coding: How to Test Accuracy Before You Buy

Every coding vendor now seems to offer automation, and most demos look impressive. The trouble is that a demo uses clean data chosen by the seller. If you are buying support for a live study, you need to know how ai open-end coding performs on your verbatims, in your languages, against your code frame.

This article gives research agency project managers and insight managers a simple protocol for testing that before you sign anything. It does not require a data science team. It needs a small sample, a few human coders and some discipline about what you measure.

What AI open-end coding does well and where it breaks

Automated tools are good at volume and consistency. Given a clear code frame and a large set of open-ended responses, they apply the same rule to the last verbatim as they did to the first. They are fast, they do not tire, and they are useful for first-pass grouping and for spotting themes you had not expected. Background reading on the basics of open-ended verbatim coding and on survey coding can help if your team is new to the topic.

They tend to break in predictable places:

• Sarcasm, irony and double meaning, where the words and the intent disagree.
• Answers that mention two or three ideas at once, where only some are coded.
• Brand, product and category slang that the tool has not seen before.
• Very short answers with almost no context, such as "ok" or "price".
• Languages and dialects with less training material behind them.

A vendor's average accuracy claim hides all of this. Your test should be built to expose it.

Build a gold-standard sample coded by humans

The foundation of any fair comparison is a gold-standard sample: a set of verbatims coded by experienced human coders against the code frame you will actually use. Without it, you are comparing a tool with a sales pitch.

Some practical rules:

• Use real data. Take verbatims from a recent study of the same type, with identifying details removed. Check your data protection obligations before sharing anything with a supplier.
• Sample on purpose. Include long answers, short answers, off-topic answers and the awkward ones. A random pull of easy responses flatters every tool.
• Freeze the code frame. Write short definitions and an example or two for each code, including what the code does not cover. Give the same document to the humans and to the supplier.
• Use two independent coders. Have them code the sample separately, then resolve differences through discussion or a senior reviewer. The agreed result is your reference.

The disagreements between your two human coders are useful in their own right. They show you where the code frame is vague, and no tool can be more consistent than the frame it is given.

Measure agreement: match rates, code-level checks and edge cases

Once the tool or supplier has coded the same sample blind, compare the output with your reference. Do not stop at one headline figure. AI coding accuracy looks different depending on how you slice it.

Check	What it tells you
Overall match rate	How often the tool's codes agree with the reference at response level. A useful start, but easily inflated by common codes.
Code-level agreement	Whether each code is applied correctly. Check how often the tool misses a code that applies and how often it adds one that does not.
Multi-code responses	Whether the tool captures every idea in an answer or only the most obvious one.
Edge cases	How it treats sarcasm, negation, mixed sentiment and "other" answers.
Stability	Whether running the same data twice gives the same result.

Code-level checks matter most. A tool can post a strong overall match rate while getting a small but commercially important code wrong, such as a complaint theme or a competitor mention. Look at the codes that drive your headline findings and check those first.

Also compare the tool with your humans fairly. If two human coders agree with each other only to a certain level, expect a tool to land in a similar range. Asking for perfect agreement is not realistic, and it is not what you get from people either.

Finally, review the errors by hand. Read a batch of misses and decide whether they are random noise or a pattern. A pattern, such as one code consistently confused with another, can often be fixed by tightening definitions. Random errors are harder to live with.

Test multilingual and short-answer verbatims separately

Do not pool everything into one score. Performance in English says little about performance elsewhere, and a strong overall result can hide weak segments.

For multilingual coding, build a separate gold-standard sample for each language in your study, coded by native-level coders who understand the market. Ask the supplier to explain how the tool handles each language: does it code in the original language, or translate first and code the translation? Translation first can lose nuance, idiom and tone, and it adds a second source of error. Mixed-language answers and local spellings deserve their own test cases. If your project spans several markets, a partner who also runs your global fieldwork coordination will already know where local wording causes trouble.

Short answers need the same separation. One-word and two-word verbatims are common in tracking studies and mobile surveys, and they give a tool little to work with. Test them as their own group and look at two things: how often the tool assigns a code confidently, and how often that code is right. A good system should be able to say "unclear" rather than guess. If it forces a code onto every response, your data will look tidier than it really is.

Hybrid workflow: where human review must stay

For most studies the practical answer is a hybrid workflow, not a choice between tool and people. The question is where each one sits.

• Code frame design. Humans should build or approve the frame, using the research objectives. A tool can suggest themes, but it does not know what the client needs to decide.
• Low-confidence responses. Route anything the tool is unsure about to a person.
• Quality sampling. Review a sample of the automated output in every wave or batch, not only at the start.
• New and emerging answers. Humans should decide when a new code is needed, for example after a product launch or news event.
• Sensitive topics. Healthcare, financial and other regulated studies should keep a person responsible for the final code.
• Sign-off. A named person should approve the coded file before it goes to tabulation.

Coded data feeds directly into your tables, so errors here travel downstream. It helps to have coding and data processing and tabulation handled by teams who talk to each other. If you are also setting rules for how AI is used across your projects, our piece on AI governance in fieldwork covers the wider questions.

Questions to put to any coding supplier

Whether the supplier is a software vendor or a service provider using one, ask the same things:

1. Will you run a blind test on a sample of our own data before we commit, and share the full results, not only a summary?
2. How do you measure agreement, and at what level: response, code or both?
3. Which languages have you tested, and who checked the output?
4. How does the tool handle very short answers, and can it flag low confidence?
5. Who reviews the output, and what is their experience in our category?
6. How do you handle changes to the code frame partway through a study?
7. Where is our data processed and stored, and is it used to train any model?
8. Which standards and regulations do you follow for data protection and quality?
9. Can we see the audit trail for a coded file?

The answers matter, but so does the manner of answering. A supplier confident in its quality will welcome a blind test. One that resists it has told you something useful. For a wider view of the tool market, see this overview of AI tool providers, and if you would like to talk through a test design for your own project, you can speak to an expert.

FAQ

How big should a gold-standard sample be?

Large enough to cover every code in your frame, every language and the awkward answer types, and small enough that two humans can code it carefully. Make sure rarer codes appear often enough to judge, even if that means sampling for them deliberately.

Is a high overall match rate enough to approve a tool?

No. Overall figures are driven by common codes and hide weak languages, short answers and important minor codes. Check agreement at code level, and by language and answer length, before deciding.

Can AI replace human coders entirely?

For simple, high-volume questions with a stable code frame, automation can do most of the work. For multilingual, sensitive or changing studies, keep human review in the workflow and a named person accountable for the final file.