Build a representative evaluation set
Collect realistic user tasks with consent and remove unnecessary personal information. Include common cases, rare cases and examples that historically cause confusion.
Practical takeaway: A diverse set is more useful than a large but repetitive one.
Separate dimensions of quality
Evaluate factual grounding, completeness, instruction adherence, user experience, latency and cost separately. A fluent answer is not necessarily a correct answer.
Practical takeaway: Record definitions and review rubrics before comparing versions.
Test the system around the model
Many failures happen in retrieval, permissions or tool orchestration rather than model generation. Evaluation should include how the whole application handles stale records and blocked actions.
Practical takeaway: Assess end-to-end workflows, not isolated chat completions.
Stress the boundary cases
Include adversarial prompts, prompt injection inside retrieved content, mismatched permissions, ambiguous requests and attempts to trigger unauthorized actions.
Practical takeaway: A safe rejection is sometimes the correct result.
Evaluate multilingual use thoughtfully
Translation fluency and task correctness are not the same. Bangla, Banglish and English workflows may need their own examples and review by people familiar with the language and domain.
Practical takeaway: Check numbers, units, culturally relevant phrases and code switching.
Make evaluation continuous
After release, users discover scenarios the test set missed. Maintain a controlled process to convert failures into regression cases and monitor the impact of model or prompt updates.
Practical takeaway: Version datasets, prompts, provider choices and acceptance thresholds.
Where to go from here
A useful next step is to translate these principles into a written product decision: who owns the workflow, what data may be used, how success will be evaluated and what happens when the system is unsure.