RLIE: Rule Generation with Logistic Regression, Iterative Refinement, and Evaluation for Large Language Models

Large Language Models (LLMs) can propose natural-language rules, circumventing the reliance on a predefined predicate space in traditional rule learning. However, existing LLM-based methods often neglect the global interactions among rules, and the potential of using fine-grained rule importance scores to calibrate neuro-symbolic reasoning remains underexplored. To address this gap, we introduce RLIE, a framework that integrates LLMs with probabilistic modeling to learn weighted rule sets in four stages: (1) Rule generation: proposing and filtering candidate rules via LLMs; (2) Logistic regression: learning sparse, calibrated weights for global rule selection; (3) Iterative refinement: revising the rule set with error-driven hard examples; and (4) Evaluation: validating the learned system via comparative inference paradigms. Across multiple real-world datasets and LLM backbones, our learned weighted rules achieve superior stability and accuracy, whereas rule-injection prompting yields mixed results and often degrades performance. These results suggest LLMs excel at semantic rule discovery but are less reliable at controlled probabilistic aggregation. Our findings highlight both the promise and the limits of LLMs for inductive reasoning, motivating a principled integration with classic probabilistic rule combination for reliable neuro-symbolic reasoning.

Large language models can write useful “if-then” rules in ordinary language, such as clues that help decide whether a review is positive or a message is harmful. However, having many rules is not enough: some rules are more reliable than others, and rules can agree or conflict. Our work studies how to turn rules proposed by language models into a dependable decision-making system. We introduce RLIE, a method that asks a language model to generate candidate rules and then learns how much each rule should matter using a simple statistical model. It also improves the rule set by revising rules based on examples it previously got wrong. We compare RLIE with directly putting rules into a language model prompt and asking the model to reason with them. Across several real-world datasets and language models, learned rule weights give more stable and accurate predictions, while direct rule prompting is less consistent and can even hurt performance. Our results suggest that language models are good at discovering meaningful rules, but less reliable at combining many rules in a controlled way. Pairing language models with statistical rule weighting offers a more interpretable and robust path toward AI systems that reason with human-readable knowledge.