Hybrid LLM/Rule-based Approaches to Business Insights Generation from Structured Data

The paper addresses how to effectively extract actionable business insights from increasingly complex, diverse datasets by combining rule-based systems with LLMs. Traditional rule-based methods provide precision and interpretability but struggle with flexibility and scalability. Stand-alone AI models, including LLMs, offer adaptability and rich pattern recognition but may lack precision and transparency in certain business contexts. The research explores hybrid approaches that integrate the strengths of both systems to navigate data complexity and improve insight generation for business intelligence.

The work situates hybrid LLM/rule-based systems within broader research on interpretable AI and data-driven insight extraction. It references interpretable techniques (e.g., LIME), supervised document classification, and logic learning machine methods, as well as studies integrating structured information and knowledge graphs with LLMs for enterprise SQL QA, graph-to-sequence models for SQL-to-text, multi-phase LLM pipelines for document retrieval, improved chain-of-thought prompting, and HCI considerations in data work. It also cites patents on semantic graphs and AI-driven actionable insights for business contexts. The literature collectively points toward using LLMs not as isolated black boxes but as components in structured, rule-augmented pipelines that incorporate algorithmic curation and structured data to improve reliability and utility.

The methodology combines rule-based processing with LLM capabilities within business insight generation pipelines and evaluates them via benchmarking. Key components include: 1) Data preprocessing: cleaning, normalization, integration, transformation, and reduction. Two strategies are contrasted: rule-based preprocessing (deterministic rules for imputation, outlier handling, encoding, etc.) and LLM-based preprocessing (using LLMs to infer or correct values in complex/unstructured text), along with an experimental approach where an LLM generates preprocessing code from input-output dataset examples with iterative validation and automation. 2) Business insights extraction: identification of patterns, anomalies, spikes, all-time highs, top-performing dimensions, and comparative performance across dimensions, using rule-based detection and LLM analysis for deeper context. 3) Natural language narrative generation: rule-based templating for precision and consistency vs. LLM-generated narratives for richness and adaptability, and hybrid strategies that structure LLM outputs around key metrics and goals. 4) Hybrid pipeline architectures: a) LLM-based insight generation from chunked data with prompt engineering per chunk to overcome token limits; b) Sequential data processing where rule-based fragment extraction and expert prompts guide LLMs to produce atomic insights, later summarized; c) Hybrid rule-based atomic insight generation followed by LLM summarization into final reports. 5) Benchmarking setup: efficiency and quality of insight extraction are compared across three approaches—pure rule-based, pure LLM, and hybrids—using data from 30 corporate Google Analytics 4 and Google Ads accounts collected via APIs over approximately two years. GPT-4 (API) is used for the LLM conditions. Metrics include precision of mathematical operations (processing efficiency), proper name hallucinations, recall of important insights, and user satisfaction (likes-to-dislikes ratio).

Quantitative benchmarking results across rule-based, LLM, and hybrid approaches: 1) Precision of mathematical operations (processing efficiency): Rule-based 100%; LLM 63%; Hybrid (rule-based precalculation + LLM analysis) 87%. 2) Proper name hallucinations (number of errors): Rule-based 0%; LLM 12%; Hybrid (name hashing + LLM analysis + hash decoding) 3%. 3) Recall of important business insights (processing efficiency): Rule-based 71%; LLM 67%; Hybrid (source-specific data chunking + LLM analysis + LLM summarization) 82%. 4) Overall user satisfaction on weekly/monthly reports (likes-to-dislikes ratio): Rule-based 1.79; LLM 3.82; Hybrid 4.60. These results indicate hybrids generally balance precision, reduced hallucinations, improved recall, and superior user satisfaction compared to single-method approaches.

The findings demonstrate that hybrid LLM/rule-based pipelines effectively address the core challenge of extracting accurate and comprehensive business insights from structured data. Rule-based elements ensure deterministic precision, proper metric handling (e.g., weighted averages), and control over preprocessing, which reduces LLM-induced mathematical and naming errors. LLM components add adaptability, richer context, and broader pattern recognition, improving recall and narrative quality. The empirical results show hybrids improve precision over pure LLMs and reduce hallucinations substantially, while also achieving higher recall than either method alone and the highest user satisfaction. These outcomes support the hypothesis that combining deterministic rules and LLM reasoning/narration yields a more robust, scalable, and user-friendly BI insight pipeline than standalone approaches. The trade-offs include increased integration complexity and prompt engineering effort, and potential computational overhead when chunking and parallelizing LLM analysis.

The paper concludes that integrating rule-based systems with LLMs forms a compelling strategy for generating accurate, nuanced, and actionable business insights from structured data. Hybrid pipelines leverage rule-based precision and interpretability alongside LLM adaptability and narrative generation to improve mathematical reliability, reduce hallucinations, enhance recall, and raise user satisfaction. Future work includes refining hybrid architectures, optimizing computational efficiency, advancing prompt engineering and integration methods, and extending applicability across broader data types and business domains.

Identified and discussed limitations include: potential bias and misinformation propagation from LLM training data; interpretability challenges when combining complex LLM behavior with rules; significant computational resource demands for training, fine-tuning, and inference (especially with chunking and parallel processing); complexity and maintenance overhead for rule sets and prompt engineering; risk of reduced predictability in purely LLM-driven preprocessing; and integration challenges when synthesizing insights from multiple chunks or atomic outputs. Benchmarking was limited to 30 Google Analytics 4 and Google Ads accounts over about two years and used a single LLM (GPT-4), which may constrain generalizability across industries, data sources, and models.