Duplicate Detection Status Tracker
Problem
Citizens were submitting multiple applications for the same benefits—sometimes intentionally to “game” the system, more often accidentally due to confusion or uncertainty about whether their first application was received. Workers spent hours manually comparing applications, investigating potential duplicates, and consolidating records. Citizens called constantly asking “what’s the status of my application?” because they had no visibility after submission.
Context
Georgia processes hundreds of thousands of benefit applications annually. Even a 2% duplicate rate represents thousands of hours of wasted worker time. Meanwhile, status inquiries consumed significant call center capacity. Both problems stemmed from lack of automation and lack of transparency.
Challenges
Defining “duplicate.” Exact SSN matches are obvious, but what about typos, name changes, partial SSN matches, similar birth dates? Performance at scale. Comparing every incoming application against hundreds of thousands of existing records in real-time required efficient algorithms and database-level optimization. False positives. Flag too aggressively and workers waste time investigating non-duplicates. Too conservatively and real duplicates slip through. Designing worker UI for rapid comparison and consolidation decisions.
Solution
Designed and implemented duplicate detection framework using Oracle-based similarity scoring function. Developed multi-dimensional matching: exact SSN match (100 points), phonetic name match using Soundex (20 points), Levenshtein distance for typo detection (15 points), birth date proximity (10 points), gender match (5 points). Scores 0-100, with threshold at 70 for “likely duplicate.”
Automated primary/duplicate designation based on submission timestamp. Built Linked Application panel for workers showing all related applications, similarity scores, and key field comparisons side-by-side. Launched Application Status Tracker in CP providing citizens real-time visibility into application progress, next steps, and estimated decision timeline.
My Contribution
Designed the scoring algorithm and database-level implementation. Architected the data model for efficient retrieval of potential matches. Implemented Oracle PL/SQL similarity function (phonetic matching, Levenshtein distance calculation). Designed worker UI for duplicate review. Coordinated implementation across Customer Portal, Worker Portal, and backend teams. Validated accuracy through testing against historical duplicate data.
Results
Reduced administrative workload as automated detection flagged duplicates immediately, eliminating hours of manual investigation. Improved data integrity by catching duplicates at submission rather than after processing began. Decreased citizen status inquiries by 30% as status tracker provided transparency. Increased trust through clearer communication of next steps and realistic timelines.
Lessons Learned
Database-level processing beats application-level for this use case. Initial prototype compared applications in Java—too slow. Moving similarity scoring into Oracle PL/SQL functions leveraged database optimization and reduced network overhead. Performance improved 10x.
Thresholds require tuning. Initial 70-point threshold produced acceptable false positive rates, but different application types (renewal vs. new application) had different duplicate patterns. We added context-aware thresholds.
Citizens want transparency more than speed. Status tracker didn’t make applications process faster, but reduced anxiety and call volume by showing progress. “Your application is in review, estimated decision in 5-7 days” is more valuable than silence.