LLM-Based Detection of Policy-Regulation Conflicts in Philippine Financial Compliance
Antonio A Yamzon ⋅ Justin R Garcia ⋅ Abien Fred Agarap ⋅ Inigo M Benavides ⋅ Sara A Venturina
Abstract
Financial institutions operating under regulatory frameworks must ensure that their internal policies align with applicable regulations. The manual review of such alignment does not scale to the volume and length of modern regulatory texts. We study the task of automatically detecting conflicts between internal policy documents and long-form regulatory manuals using large language models (LLMs). We framed this task as a retrieval-augmented generation (RAG) problem, wherein an LLM identifies conflict pairs given a policy document and retrieved regulatory context. We evaluate two retrieval strategies: standard RAG and Recursive Abstractive Processing for Tree-Organized Retrieval (RAPTOR). On a corpus of 323 internal policy documents paired with four regulatory manuals (ranging from 59K to 1.5M tokens), both approaches achieve comparable precision (62.07\% for RAG, 63.33\% for RAPTOR; $p = 0.92$). RAPTOR produces significantly more faithful outputs ($p = 0.037$) and reduces prompt token consumption by approximately 23\% while RAG outputs are preferred by domain evaluators for readability. These results inform the design of practical regulatory compliance systems and highlight open challenges in cross-document legal conflict detection.
Chat is not available.
Successful Page Load