Unearth-AI: Human-in-the-Loop Cohort Extraction from Natural Language for Health AI

Defining and extracting clinical prediction tasks from electronic health record (EHR) data is technically demanding and time-consuming, requiring substantial domain expertise and dataset-specific tooling. We present Unearth-AI, a proof-of- concept interactive, agentic large language model (LLM) system that translates natural language task descriptions into formal task specifications in the ACES framework (Xu et al., 2025), combining LLM generation with automated validation, feasibility detection, and optional human-in-the-loop refinement while requiring only dataset schema information rather than patient-level data. In a feasibility study spanning 14 tasks over the INSPIRE perioperative EHR dataset, Unearth-AI generated specifications judged consistent with the provided free-text inputs for all 8 feasible tasks and correctly identified 4 of 6 infeasible tasks. For feasible tasks, end-to-end generation took under 4 minutes and cost under $0.25 on average. These results suggest the practical feasibility of an interactive LLM-mediated task extraction workflow and motivate future work on systematic evaluation and user-centered tooling for clinical AI development.