Recipient lists, keyword sets, error messages pulled from logs, URLs collected across files — this kind of data almost always contains duplicates, and entries that differ only in case or surrounding whitespace slip past a naive comparison.
This tool removes duplicate lines and reorders the result however you need. Combine case-insensitive matching, trimming and blank-line removal, or invert the operation to show only the duplicated entries or only the ones appearing exactly once. Sort alphabetically, by length, or with number-aware natural ordering, then add a prefix and suffix or join everything onto one line to build a SQL IN clause or an array literal directly.
How to use
- Paste your list — One entry per line on the left. Pasting a column copied from a spreadsheet works — it arrives as lines.
- Set normalisation options — Combine trim whitespace and case sensitive to define what counts as the same value. For an email list, ignoring case is usually correct.
- Choose a sort order — Alphabetical suits general lists; natural sort suits entries containing numbers, placing
item 2beforeitem 10the way a person would. - Shape the output — Set the prefix to
'and the suffix to',, then join into one line, and you have a ready-made SQLINclause. Copy the result or save it as a text file.
Frequently asked questions
Entries differing only in case are not being deduplicated.
Check whether case sensitive is enabled — with it on, Apple and apple are distinct values.
For email addresses, ignoring case matches reality: the domain part is case-insensitive by specification, and in practice every major provider treats the local part that way too. For user IDs or file paths, case distinction may genuinely matter.
Lines that look identical survive deduplication.
Almost certainly invisible characters differ.
Text copied from web pages or word processors often contains a non-breaking space (U+00A0) or an ideographic space (U+3000) instead of a normal space, and files created on Windows may leave a carriage return at the end of each line.
Trimming handles ordinary whitespace at the edges but does not normalise special spaces inside the string. Use the regex tester to find and replace \u00a0 and similar, then run the list through again.
What is the difference between keeping the first and the last occurrence?
It decides which position survives when a value repeats.
Since identical lines carry identical content, the choice mostly affects the order of the remaining entries: keeping the first preserves original ordering, keeping the last moves each value to its final position in the source. That matters when the list is chronological and you want the most recent context.
Concepts worth knowing
Why natural sort exists
Lexicographic sorting places item 10 before item 2, because comparing character by character puts 1 before 2. Filenames and version numbers make that look particularly wrong.
Natural sort recognises runs of digits as numbers, so v1.9 precedes v1.10 as a person expects. In JavaScript, Intl.Collator with numeric: true provides this, and it handles locale-specific ordering for non-Latin scripts at the same time.
The data structure behind deduplication
The naive approach compares each entry against everything before it, which costs time proportional to n² and slows down quickly.
In practice you use a hash set — JavaScript's Set, Python's set — which answers 'have I seen this?' in constant time, making the whole pass linear. That is what runs here, so tens of thousands of lines process instantly. What counts as 'seen' is the normalised string, so the case and whitespace options determine the outcome.
Turning a list into code
Moving a long list into a SQL IN clause or an array literal is routine work, and prefix, suffix and join together produce 'a','b','c' in one step.
Watch for values containing quotes, which need escaping, and for scale: IN clauses with thousands of entries perform badly and databases impose parameter limits. Past a certain size, loading the values into a temporary table and joining is the correct approach.