Remove Duplicate Lines
Remove repeated lines from a list, export, or log while preserving the order of first appearance. Optionally ignore case and surrounding whitespace, or flip the mode to keep only the lines that appeared more than once.
How Remove Duplicate Lines works
Paste a list — one item per line.
Choose whether case and surrounding spaces should matter.
Read the summary of how many duplicates were removed.
Copy the deduplicated list.
When to use Remove Duplicate Lines
- Collapsing a pasted email or subscriber list down to one entry per address before an import, without shuffling the order you already have.
- Cleaning a CSV column exported with repeated rows while leaving the rest of the ordering exactly as it was.
- Merging two lists into one pane and stripping the overlap that now appears twice.
- Auditing an export by switching to “Keep only duplicates” to see precisely which values were entered more than once.
- Deduplicating tags, SKUs, or slugs where “Blue”, “blue”, and “ blue ” should be treated as the same entry.
- Reducing a raw log or dump to its distinct lines before you count or diff it.
Examples
Input
apple banana apple cherry banana
Output
apple banana cherry
The first apple and banana stay put; only the later repeats are dropped, so your existing order is untouched. The summary reports two lines removed.
Input
invoice-01 invoice-02 invoice-01 invoice-03
Output
invoice-01
“Keep only duplicates” returns every line whose value occurred more than once — here only invoice-01 was entered twice, so it is the sole result.
How Remove Duplicate Lines works under the hood
Before anything is compared, the text is split into lines with line endings normalised, so a file that arrived with Windows CRLF breaks is treated the same as one from macOS or Linux. Each line is then reduced to a comparison key: if “Ignore surrounding spaces” is on it is trimmed of leading and trailing whitespace, and if it is not case sensitive it is lowercased. The line you actually keep is always the original, unmodified text — the key only decides what counts as the same value.
The tool makes one pass to count how many times each key appears, then a second pass to build the output. In the default unique mode, the first time a key is seen the line is kept and the key is remembered; every later line with that key is skipped. Because the walk goes top to bottom and only ever drops later repeats, the first occurrence of each value stays exactly where it started, and a list you have already sorted or ranked is never reshuffled.
The alternative mode inverts the question. Instead of the distinct list, it returns every line whose key was counted more than once — including each repeat — which is the fastest way to see which entries in an export were recorded twice rather than which are unique. The stats show lines in, lines removed, and how many distinct values had duplicates behind them.
All of this runs on your device in JavaScript. The counting builds a lookup of every distinct key in memory, so lists in the tens of thousands of rows stay comfortable; a multi-hundred-megabyte file is the strain case, because the whole document has to fit in the page at once. Nothing is uploaded, so a list of customer emails or internal identifiers never leaves the browser.
Common mistakes
- Expecting the output to come back alphabetised. Deduplication preserves the order of first appearance rather than sorting; run Sort Lines first, or after, if you want ordered output.
- Leaving “Ignore surrounding spaces” off when your list has stray indentation. With it off, “ apple” and “apple” are different values and both survive, which is rarely what you want for a pasted column.
- Forgetting that blank lines are compared like any other line. Several empty lines collapse to one in unique mode; if the vertical spacing matters, handle it with Remove Empty Lines rather than relying on the deduper.
- Reading “Keep only duplicates” as a preview of what unique mode would remove. It returns the repeated lines including every copy, so a value seen three times appears three times — it is an audit view, not the discard pile.