Arabic AI evaluation

How to test whether an AI assistant understands Egyptian Arabic

A practical rubric for testing dialect, intent and factual accuracy, with a worked Egyptian Arabic example and a pilot brief.

By the Annotvia AI team

Start with a decision the test can change

A support assistant may write fluent Arabic and still make a promise the business cannot keep. Before collecting ratings, write down what would prevent you from shipping: a wrong date, an invented refund, or a reply in a register that feels wrong for your customers. Those failures need different labels.

Try this item with two reviewers

The user writes in Egyptian Arabic: «ممكن نأجل المكالمة لبكرة؟ لسه مش عارف الساعة.» They want to move a call to tomorrow and have not picked a time. Compare two replies:

A: «أكيد، تحب نحدد الساعة بعدين؟»
B: «تمام، حجزتلك المكالمة بكرة الساعة تلاتة.»

Score what actually happened

Reply B sounds natural in Egyptian Arabic, but it invents a time and claims a booking. Reply A asks a useful follow-up, although it does not explicitly confirm that tomorrow is the requested day. If the product cannot change bookings, the reviewer should never reward a reply for claiming it already did so.

Use three separate questions

1. Intent: did the reply preserve the request and its constraints? 2. Grounding: did it introduce an unsupported fact or action? 3. Dialect: would the wording sound natural to the intended users in this setting? Have each reviewer quote the words behind their judgment. Allow “cannot judge” when the prompt lacks context.

Calibrate before increasing volume

Give two reviewers the same small set independently before showing them each other’s ratings. Include an MSA reply that is factually correct and a dialect-perfect reply that is factually wrong. Inspect disagreements by question. If one reviewer penalizes every MSA word while another accepts ordinary code-switching, revise the language guideline and re-rate the disputed items.

What to put in the pilot brief

Specify the product, target dialect and audience, the tasks the model can perform, and examples of critical errors. Request item-level scores, reviewer rationales, a disagreement log and a recommendation on whether the rubric is ready to scale. The example above is illustrative; it is not a client benchmark.

For a Saudi Arabic comparison, inspect our illustrative evaluation sample (JSON).