How to Spot Duplicate Records Using Column Uniqueness Stats

"How many customers do we actually have?"

Someone in marketing is segmenting the customer base for an outreach campaign and asks a question that should have a simple answer: how many distinct customers are actually in this dataset? The row count says one thing. But a quick scroll turns up "Meridian Retail" and what might be the same company entered slightly differently somewhere else — and now the honest answer is "somewhere between the row count and a smaller number I can't pin down without checking every row."

This is a narrower, more specific problem than "the data is messy" — it's one particular question (how many distinct things are really here) that a full manual audit is a lot of effort to answer precisely.

Why this keeps happening

Small text variations creep in for boring reasons — one order entered by hand, one imported from a different system, one abbreviated slightly differently. None of it looks like an error at a glance. It only becomes a problem when something depends on an accurate count of distinct entities, like a customer segment size or a per-account rollup, and a handful of near-duplicates quietly inflate the number.

The manual approach

1. Sort the column alphabetically and scroll, looking for entries that look almost, but not quite, identical.
2. Cross-reference against another list if one exists, hoping it's more accurate.
3. Manually merge anything that looks like the same entity, using judgment call by judgment call.
4. Move on, without much confidence in the final number.

Why the manual version breaks

• It doesn't scale past a small dataset. Scrolling and eyeballing works on a few hundred rows; it doesn't work on ten thousand.
• "Looks almost the same" is subjective. Two people doing this manually would flag a different set of near-duplicates.
• There's no number to point to afterward — just a cleaned list, with no record of how many things were actually merged or why.

Reading the count instead of scrolling for it

DataQloo's column profile already computes exactly the number this question needs: the count of distinct values in a column, alongside the total row count — no separate step required, since it's part of the same profile covered in a field guide to reading a data profile report.

In the canonical Sales Orders dataset, the customer column profiles at 25 distinct values across 10,000 rows — a small, stable number consistent with a customer base of real repeat buyers, not one inflated by near-duplicate entries. That's the number to check first, before scrolling a single row: does the distinct count roughly match what the business should have? If a dataset that should have a few dozen real customers profiles at several hundred distinct values, that gap — not any individual row — is the signal that something's being counted more than once.

SELECT count(DISTINCT customer) AS distinct_customers, count(*) AS total_rows
FROM sales_orders

What this does and doesn't tell you

A healthy distinct count is reassuring, but it's a starting signal, not a guarantee — it tells you the scale of the mismatch is small, not that there's zero mismatch at all. "Meridian Retail" and "Meridian Retail Inc." would still count as two distinct values here, even though a human would recognize them as the same company. What the stat reliably catches is the big-picture question ("is this dataset roughly the shape I expect") without requiring a full manual audit to answer it.

Key takeaways

• The distinct-count stat, already part of every column profile, directly answers "how many different things are actually in this column" without a manual scroll-and-compare.
• Compare it against what you'd expect for that specific column — a small, stable category should have a small, stable count; a big gap is the signal worth investigating.
• This catches scale mismatches, not every individual near-duplicate — it's a fast first check, not a replacement for judgment on genuinely ambiguous cases.

Next

With the data trustworthy, filtered, and checked for duplicates, the next step is building a full report on top of it — see Visual ETL for PostgreSQL for the complete filter-summarize-sort pattern. For the rest of what a column profile tells you beyond uniqueness, see a field guide to reading a data profile report.