Evaluation¶
Measure whether a model meets your goals using explicit tasks and scoring criteria.
Use evaluation to compare versions, identify failure patterns, and decide whether to deploy a new model. Meaningful comparisons hold the dataset version, scoring method, and inference configuration constant.
Choose an evaluation type¶
| Type | Purpose | Main configuration |
|---|---|---|
| Harbor Agent | Evaluate an agent completing tasks in its environment | Benchmark, agent, model, and execution settings |
| QA (HLE/FSR) | Evaluate answers using a judge model | Benchmark, tested model, judge model, and scoring configuration |
Choose a type and benchmark on Evaluation (评测). Harbor can use an available deployment; the form also offers a custom model endpoint. Available benchmarks are listed in the current console.
Prepare a fair comparison¶
Use the same test set for the base and trained model. Keep training examples out of the evaluation set to avoid measuring memorization. For agent tasks, also keep the agent configuration and environment consistent.
Before submission, confirm model access, benchmark selection, and optional execution settings. Track status, trial progress, and available results in My evaluations (我的评测).
Interpret the results¶
Execution completion describes the run; a score or pass rate describes model performance. Missing scores, abnormal exits, and incomplete trials need investigation before they are included in a quality comparison.
Read individual failures to distinguish answer quality and tool-use problems from model-service or task-environment failures.
Improve the next version¶
Use failure patterns to revise examples in your external data preparation workflow, then upload a new training dataset version. The current console does not run question-bank solving or automatic SFT generation. See Question Bank.
Related: Training · Deployment · Troubleshooting