Fuzzy Dedupe
Near-duplicate detection for names and organizations.
Scales matching and record linkage with Python and Rust.
- For
- Data operations and master data teams
- Engine
- Rust · Python
- Source code
- Open · Apache-2.0
- Buying
- Written, fixed-price pilot
What is Fuzzy Dedupe?
Fuzzy Dedupe is an open-source library that finds near-duplicate records in person and organization names. Its Rust core is called from Python, and it understands Turkish company names with abbreviations such as 'SAN. VE TİC. LTD. ŞTİ.'. A pure-Python reference implementation ships in the package and every matching strategy is property-tested against it.
Six names, one cluster and two funds that must stay apart.
6 records · 15 possible pairs
'İSTANBUL'.lower() → 'i̇stanbul' (9 characters). Turkish mode fixes it.
Visualisation. Names come from the README examples; filter counts are representative for these six records. Results always equal comparing every pair.
How it works
From the product's own documentation. Each number says where and on which version it was measured.
It normalizes
Case, punctuation, legal suffixes like Ltd. or Inc., and Turkish letters.
It filters by proof
Filters skip only pairs that provably cannot match. Nothing is guessed.
It measures the rest exactly
Every remaining pair gets a full comparison, so results equal an exhaustive scan.
Three modes
Duplicates in one list, linking two lists, or live queries against a persistent index.
- 115×
- Faster than RapidFuzz cdist on 100,000 UK company names, same 21,513 pairs. fuzzy-dedupe 0.6.0, 4 October 2026, one laptop.
- ~0.1 ms
- Median query against an 800,000-name index; RapidFuzz takes ~90 ms for the same question. 0.6.0, same machine.
- Turkish-aware
- 'ABC TEKSTİL SAN. VE TİC. LTD. ŞTİ.' matches the full legal name.
- Tested against itself
- A pure-Python reference ships in the package; every strategy is property-tested against it. Source
Stack
- Rust
- Python (PyO3)
- CLI
Try it
from fuzzy_dedupe import find_duplicates
find_duplicates(["ABC TEKSTİL SANAYİ VE TİCARET LİMİTED ŞİRKETİ",
"Abc Tekstil San. ve Tic. Ltd. Şti."],
threshold=0.0, strip_suffixes=True, turkish=True)
# [(0, 1, 0.0)]What it does not claim
Thresholds and false-positive costs are tuned on your data. Speed figures were measured on one machine; yours will differ.
Current boundary
Thresholds and false-positive costs are tuned on customer data.
What a pilot covers
A sample starting scope. Final deliverables and acceptance criteria are fixed in the written proposal.
- Question
- Which similar records refer to the same entity?
- Likely input
- Permitted name and organization records with known matches
- Deliverables
- Candidate duplicate list
- Threshold and false positive review
- Human approval flow for merges
Before you decide
Can I buy this product today?
A paid pilot and setup proposal can be prepared. Scope, data access, deliverables, timeline and a fixed price are agreed in writing after a discovery call.
Why is there no price?
Data volume, integration and success criteria differ from job to job. After the call you get a fixed-price written proposal; no paid work starts without your approval.
Do I need to upload live data here?
No. This site has no file upload or accounts. Data access, retention and permissions are set only in the pilot agreement.
What is the first step?
Choose Fuzzy Dedupe in the form and describe the problem you need to solve. The data and pilot boundaries are then set together.
More in this discipline
- CRM Cleaning APIRust (Axum) · Python
Prepares messy lead lists for CRM import.
- ProofFrameRust · Python · WASM
Data contracts with verifiable quality evidence.
- Scraping PipelinePython · asyncio
Turns public web data into validated output.