JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks

Evaluating agentic AI on open-ended professional tasks faces a fundamental dilemma between rigor and flexibility. Static rubrics provide rigorous, reproducible assessment but fail to accommodate diverse valid response strategies, while LLM-as-a-judge approaches adapt to individual responses yet suffer from instability and bias. Human experts address this dilemma by combining domain-grounded principles with dynamic, claim-level assessment. Inspired by this process, we propose JADE, a two-layer evaluation framework. Layer 1 encodes expert knowledge as a predefined set of evaluation skills, providing stable evaluation criteria. Layer 2 performs report-specific, claim-level evaluation to flexibly assess diverse reasoning strategies, with evidence-dependency gating to invalidate conclusions built on refuted claims. Experiments on BizBench show that JADE improves evaluation stability and reveals critical agent failure modes missed by holistic LLM-based evaluators. We further demonstrate strong alignment with expert-authored rubrics and effective transfer to HealthBench and DR.BENCH, covering medical and 10-domain professional evaluation settings. Code and data are available at https://github.com/smiling-world/JADE.

AI agents are increasingly used to write long reports for tasks like market research and supplier selection. However, it is hard to tell whether these reports are actually trustworthy: a report may sound convincing while relying on weak evidence, wrong facts, or shaky reasoning. Existing methods either follow rigid fixed checklists that ignore valid alternative answers, or ask another AI to give one overall score, which is inconsistent and can be fooled by fluent writing. We propose JADE, which evaluates reports the way a two-person review team would: expert-written guidelines decide what a good answer should cover for the given task, while a fact-checker verifies the specific claims in the report. If a conclusion depends on a claim that turns out to be wrong, that conclusion receives no credit. To test JADE, we also built BizBench, a collection of 150 real business research questions drawn from genuine professional workflows. JADE matches human expert judgments more closely than existing methods, and exposes hidden problems, such as unreliable sources and reasoning built on made-up “facts”, that simple overall scores miss. This helps users decide when an AI report can actually be trusted for high-stakes business decisions.