Hotel Deduplication: How to Stop Showing the Same Hotel Five Times

Ask any OTA product manager what their users complain about and "the same hotel listed several times" is near the top. It looks like a cosmetic issue. It is not. Hotel deduplication decides whether your price comparison works, whether your content is complete, and whether your conversion rate is what it should be.

This article is the playbook we use at Mappin to keep a master catalogue of a million hotels clean across five suppliers. It applies whether you are deduplicating your own database or evaluating a mapping provider.

What duplicates cost you

• Broken price comparison. If Supplier A's "Grand Hyatt Mumbai" and Supplier B's "Grand Hyatt Mumbai Hotel & Residences" are two listings, the traveller never sees that B is 12% cheaper. You lose the booking or the margin.
• Split content. Images from one supplier, descriptions from another, reviews from a third - each attached to a different listing. Every listing looks thinner than the property deserves.
• Wasted caching and search capacity. Five index entries, five cache keys, five sets of availability calls for one hotel.
• Misleading analytics. Demand for a property is divided across duplicates, so popularity ranking and inventory decisions are made on wrong numbers.

Where duplicates come from

Duplicates arrive in three ways, and each needs a slightly different response.

Cross-supplier duplicates

The classic case: five suppliers, five IDs, one building. This is solved by hotel mapping - resolving every supplier record to a master hotel ID.

Intra-supplier duplicates

One supplier lists the same property twice, often after a rebrand or a contract change (the old code is kept, a new one is added). These are harder to spot because the names can be quite different: "Le Meridien Pune" and "Marriott Pune Hinjewadi" may be the same hotel before and after a flag change.

Master-level duplicates

Your own master table drifts. Two masters were created months apart from two suppliers with poor coordinates, and nobody noticed until a third supplier matched both. These need a periodic scan over the master set itself.

Signals that catch duplicates

No single field is trustworthy. Robust deduplication combines several independent signals, each contributing to a score:

1. Normalised name similarity. Lowercase, strip accents, remove stop-words ("hotel", "the", "resort", "&"), then compare with a token-set ratio so word order and extra words do not matter. "Hilton Dubai Jumeirah" vs "Jumeirah Hilton Dubai Resort" should score high.
2. Geo-distance. Two 5-star properties 40 metres apart with 80%+ name similarity are almost certainly the same. The same name 3 km apart is almost certainly two properties (chains cluster in cities). Treat missing or obviously-rounded coordinates as unknown, not as a match.
3. Address tokens. Street name and number, postal code, PO box. Weak on its own in markets with inconsistent addressing, but very strong as a tie-breaker.
4. Resolved city. Compare after mapping the supplier city to a master city. Dubai, Jumeirah and Dubai Marina must resolve to the same place before the hotel comparison starts.
5. Structural fields. Star rating, chain code, property type and room count. Mismatches here lower confidence; matches raise it slightly.

A scoring approach that scales to a million records

Pairwise comparison of a million hotels is a trillion comparisons - impossible. Blocking fixes that:

for each supplier hotel:
    candidates = masters in the same master city
                 AND within 1.5 km (if coordinates exist)
    for each candidate:
        score = 0.45 * name_similarity
              + 0.30 * geo_score
              + 0.15 * address_overlap
              + 0.10 * structural_match
    best = highest-scoring candidate

    if best.score >= 0.90  -> auto-link to best master
    elif best.score >= 0.70 -> send to review queue
    else                  -> create new master

The thresholds are the important part. Set the auto-link bar high: a false merge (two real hotels combined) is far more damaging than a missed one, because partners start receiving the wrong property's rates. Everything in the grey band goes to a review queue where a person spends ten seconds per case.

Operating it: the review queue and the duplicate scanner

In production, two tools do most of the work:

• A review queue for medium-confidence matches, with both records side by side, a map, and one-click merge / keep separate / flag. At Mappin this sits inside the master panel and is worked daily.
• A duplicate scanner that periodically re-scans the master tables themselves - countries, cities and hotels - and groups suspected duplicates for review. The latest Mappin scan grouped tens of thousands of hotel candidates for review; most are confirmed distinct, the rest are merged.

When two masters are merged, every supplier mapping from the loser is moved to the winner and the old master ID is retired. Partners that keyed on the old ID learn about it through the change feed rather than through a broken page.

Keeping it clean

Deduplication is a process, not a project. Three habits make the difference:

• Map geography first. Most hotel duplicates we investigate trace back to a city mismatch. A clean master city table is the foundation.
• Publish changes. Expose merges, renames and deactivations as a dated change feed so downstream systems can sync deltas. Mappin's /api/changes/?since=YYYY-MM-DD does exactly this, and partners also receive a daily digest e-mail.
• Measure the duplicate rate. Track supplier records per master over time. If it drifts down, your matcher is missing links; if two masters suddenly share a supplier code, something merged upstream.

See your own duplicate rate

The quickest diagnosis is to run a busy city through a mapping API and count how many supplier records collapse into one master. With a free Mappin account you can do that in the Playground in a few minutes - create an account, then query /api/city/?city=Dubai and inspect the providers array on each master hotel. The API docs cover every endpoint.