A GenAI rules system that knows when to ask
Strategy for a GenAI system that finds duplicate and conflicting decision logic across thousands of product configurations, with confidence-based routing between automation and expert review.
Context
Across 180+ product definitions and 3,000+ rule configurations, rule authors in different business units were creating overlapping or contradictory logic without visibility into the whole catalog.
Manual audits happened quarterly at best and covered a fraction of the rules. Conflicts that slipped through caused processing errors, onboarding delays and occasional compliance issues.
Constraints
- The rules govern client onboarding, so a wrong change has real cost.
- Rule owners are accountable for their rules and need to understand any proposed change.
- Engineering capacity is scarce, so the concept had to be proven before asking for it.
Options on the table
More reviewers or better checklists would not scale to thousands of rules, and would still miss logic written two different ways.
Pattern detection across a large corpus is a strong fit for LLMs. Applying their suggestions automatically to onboarding rules without validation is not.
The system proposes, with a confidence score and its reasoning. High-confidence findings move to a lightweight approval path; low-confidence ones go to the rule owner with the evidence attached. Every expert decision feeds back.
The decision
The system looks for three things: duplicated rules (the same logic expressed differently), conflicting rules (contradictory outcomes for the same inputs), and consolidation opportunities. Each recommendation carries the model's reasoning, its confidence, the affected products and the likely impact.
I prototyped the concept with Claude Code and n8n before asking for engineering investment, which shortened the path from idea to a validated proof of concept from weeks to days.
My role
Defined the product strategy and the human-in-the-loop design, including how confidence routes work between automation and expert review.
Built the early prototypes myself to validate the approach and build confidence with stakeholders before committing engineering time.
Outcome
The proof of concept surfaced rule conflicts that manual review had missed, which validated the core hypothesis. The system targets a 40% reduction in manual rules maintenance; that is a target, and this page will say so until it is a result.
What I would change
Set thresholds per finding type from a labeled set early on. A single global threshold is too strict for obvious duplicates and too loose for subtle conflicts.
Pair model confidence with a replay of historical decisions from day one, so "high confidence" also means "provably changes nothing unexpected".