Human evaluation

How to design an audio rating study your reviewers can score consistently

Use a paired listening task to separate naturalness from instruction following and turn disagreements into useful evidence.

By the Annotvia AI team

Ask one question at a time

Imagine two voices answering a navigation prompt. One sounds smoother but reads the wrong street name. If the only question is “Which is better?”, two careful reviewers may disagree while both follow reasonable rules. Give content correctness and audio naturalness separate questions.

Prompt: “Read the address exactly as written.” Candidate A reads the correct address with a long pause. Candidate B sounds conversational but changes the building number. Score exactness first, naturalness second. If you also need an overall preference, state whether a wrong address overrides a smoother delivery.

Make the listening conditions explicit

Decide if headphones are required, whether replay is allowed and what to do if the player fails. Include “cannot assess” for genuinely unplayable items. Counting a broken URL as poor model quality hides a fixable study problem. For paired judgments, vary which candidate appears first and record that order.

Calibrate on disagreements

Use a short shared set with clear wins, near ties and at least one broken item. Collect each reviewer’s decision before group discussion. Ask for a short reason on disputed items: “The final word cuts off” gives the study owner something to inspect. Repeat the exercise with fresh items after any rubric change.

Report uncertainty alongside preference

Show how many items were presented, how many received valid judgments and how many were ties or unassessable. If you report a preference percentage, say exactly which responses are in the denominator. Keep reviewer disagreement visible; a narrow win on ambiguous pairs means something different from a consistent win on clear pairs.

What to send when asking for a quote

Share a public or synthetic sample pair, the intended audience, target language, clip length, number of judgments per item and proposed rating questions. If the study runs on your own endpoint, describe how a participant enters, how completion is recorded and what happens when someone exits early.