Delimiter.live
How-to guide

How to Remove Duplicate Lines

To remove duplicate lines, paste the list into a duplicate line remover and run it. The first occurrence of each line is kept in its original position and later repeats are dropped. Decide first whether matching should ignore case and surrounding whitespace, because those two settings determine what counts as a duplicate at all.

Remove Duplicate Lines

Strip repeated lines from a list and keep the order.

Open the tool

De-duplicating a list sounds trivial until you look at what "the same" means. Are "Admin" and "admin" the same entry? Is a line with a trailing space the same as one without? Is a blank line a duplicate of another blank line?

Those answers change with the job. Email addresses are case-insensitive in practice, so ignoring case is usually right. A list of case-sensitive identifiers or passwords is the opposite. Trailing whitespace is almost always noise from copy-paste and worth ignoring.

Step by step

  1. Paste the list

    One entry per line. Lists copied out of spreadsheets, log files and email clients frequently arrive with trailing spaces or stray carriage returns, which the next two settings handle.

  2. Decide on case sensitivity

    Turn on case-insensitive matching for email addresses, domains and URLs, where case carries no meaning. Leave it off for identifiers, tokens and anything where Admin and admin are genuinely different.

  3. Decide on whitespace

    Ignoring leading and trailing whitespace catches duplicates that differ only by an invisible trailing space. This is nearly always what you want unless the spacing is meaningful, as it can be in indented data.

  4. Run it and check the count

    Compare the input and output line counts. A surprising drop usually means matching was looser than intended; almost no drop usually means it was stricter.

  5. Switch to duplicates-only if you are auditing

    The duplicates-only mode inverts the output and shows just the repeated entries. That is what you want when the question is "what was duplicated?" rather than "give me the clean list".

Example

Case-insensitive matching with whitespace ignored. Order is preserved and the first occurrence wins.

Input

ada@example.com
bob@example.com
ADA@example.com
bob@example.com
carol@example.com

Output

ada@example.com
bob@example.com
carol@example.com

Frequently asked questions

Is the original order preserved?
Yes. The first occurrence of each line stays where it was and later repeats are removed. If you want alphabetical or numerical order instead, sort the result afterwards with the line sorter.
Which occurrence is kept?
The first. When lines differ only by case or whitespace and those differences are being ignored, the first version encountered is the one that survives into the output.
Can I see only the duplicates?
Yes. The duplicates-only mode returns just the entries that appeared more than once, which is what you need when auditing a list rather than cleaning it.
Are blank lines treated as duplicates?
Yes, once whitespace-insensitive matching is on: every blank line matches every other, so all but the first are removed. Turn the setting off if the blank lines are structurally meaningful.
How large a list can I process?
Lists of several hundred thousand lines are fine on a modern machine. Everything runs in your browser, so there is no upload limit — only your device memory.

Tools used in this guide

All tools →
All guides →

Last reviewed .