Skip to content

Fuzzy Dedupe

Near-duplicate detection for names and organizations.

Scales matching and record linkage with Python and Rust.

For
Data operations and master data teams
Engine
Rust · Python
Source code
Open · Apache-2.0
Buying
Written, fixed-price pilot

What is Fuzzy Dedupe?

Fuzzy Dedupe is an open-source library that finds near-duplicate records in person and organization names. Its Rust core is called from Python, and it understands Turkish company names with abbreviations such as 'SAN. VE TİC. LTD. ŞTİ.'. A pure-Python reference implementation ships in the package and every matching strategy is property-tested against it.

Six names, one cluster and two funds that must stay apart.

6 records · 15 possible pairs

ABC TEKSTİL SANAYİ VE TİCARET LİMİTED ŞİRKETİ
Abc Tekstil San. ve Tic. Ltd. Şti.
ABC TEKSTIL SAN.VE TIC.LTD.STI.
NEO PORTFÖY BİRİNCİ SERBEST FON
NEO PORTFÖY İKİNCİ SERBEST FON
Globex Lojistik A.Ş.
turkish=True · strip_suffixes=True

'İSTANBUL'.lower() → 'i̇stanbul' (9 characters). Turkish mode fixes it.

Visualisation. Names come from the README examples; filter counts are representative for these six records. Results always equal comparing every pair.

How it works

From the product's own documentation. Each number says where and on which version it was measured.

  1. It normalizes

    Case, punctuation, legal suffixes like Ltd. or Inc., and Turkish letters.

  2. It filters by proof

    Filters skip only pairs that provably cannot match. Nothing is guessed.

  3. It measures the rest exactly

    Every remaining pair gets a full comparison, so results equal an exhaustive scan.

  4. Three modes

    Duplicates in one list, linking two lists, or live queries against a persistent index.

115×
Faster than RapidFuzz cdist on 100,000 UK company names, same 21,513 pairs. fuzzy-dedupe 0.6.0, 4 October 2026, one laptop.
~0.1 ms
Median query against an 800,000-name index; RapidFuzz takes ~90 ms for the same question. 0.6.0, same machine.
Turkish-aware
'ABC TEKSTİL SAN. VE TİC. LTD. ŞTİ.' matches the full legal name.
Tested against itself
A pure-Python reference ships in the package; every strategy is property-tested against it. Source

Stack

  • Rust
  • Python (PyO3)
  • CLI

Try it

from fuzzy_dedupe import find_duplicates

find_duplicates(["ABC TEKSTİL SANAYİ VE TİCARET LİMİTED ŞİRKETİ",
                 "Abc Tekstil San. ve Tic. Ltd. Şti."],
                threshold=0.0, strip_suffixes=True, turkish=True)
# [(0, 1, 0.0)]

What it does not claim

Thresholds and false-positive costs are tuned on your data. Speed figures were measured on one machine; yours will differ.

Current boundary

Thresholds and false-positive costs are tuned on customer data.

What a pilot covers

A sample starting scope. Final deliverables and acceptance criteria are fixed in the written proposal.

Question
Which similar records refer to the same entity?
Likely input
Permitted name and organization records with known matches
Deliverables
  • Candidate duplicate list
  • Threshold and false positive review
  • Human approval flow for merges
Send your problem
Article (Turkish) · 6 minTürkçe şirket unvanlarında tekrar kayıt bulma: 'Ltd. Şti.' ile 'Limited Şirketi' aynı mı?

Before you decide

Can I buy this product today?

A paid pilot and setup proposal can be prepared. Scope, data access, deliverables, timeline and a fixed price are agreed in writing after a discovery call.

Why is there no price?

Data volume, integration and success criteria differ from job to job. After the call you get a fixed-price written proposal; no paid work starts without your approval.

Do I need to upload live data here?

No. This site has no file upload or accounts. Data access, retention and permissions are set only in the pilot agreement.

What is the first step?

Choose Fuzzy Dedupe in the form and describe the problem you need to solve. The data and pilot boundaries are then set together.

  • CRM Cleaning API

    Prepares messy lead lists for CRM import.

    Rust (Axum) · Python
  • ProofFrame

    Data contracts with verifiable quality evidence.

    Rust · Python · WASM
  • Scraping Pipeline

    Turns public web data into validated output.

    Python · asyncio