How to remove duplicate lines
- Paste your list into Input text, one item per line. A sample list is loaded and is selected the first time you click, so your paste replaces it.
- Switch options on or off: Case-sensitive, Trim whitespace, Remove empty lines, Show only duplicates and Sort A to Z. The result updates at once, so there is no button to press.
- Check the three counts, then press Copy result.
How it works
Each line becomes a comparison key: trimmed of spaces at both ends when Trim whitespace is on, and lowercased when Case-sensitive is off. The first line with each key is kept in its original position and every later match is dropped. The tool stores keys in a JavaScript Map, so the work grows in step with the number of lines: 100,000 lines take a fraction of a second. Show only duplicates keeps one copy of each line that appears more than once. Sort A to Z uses natural order with case ignored, so "2" comes before "10". The standard for this is localeCompare with numeric collation.
Blank lines count as lines too, so several empty lines collapse to one unless Remove empty lines is on. A single newline at the very end of the text is ignored.
Worked example
The sample list has 10 lines: apple, banana, cherry, apple, banana, date, Apple, cherry, elderberry, apple. With the defaults (Case-sensitive on, Trim whitespace on, the rest off) the result is apple, banana, cherry, date, Apple, elderberry. The counts read Original lines 10, Unique lines 6 and Lines removed 4, because "Apple" differs from "apple". Turn Case-sensitive off and "Apple" joins the first "apple": 5 unique lines and 5 removed. Turn on Show only duplicates and the result is apple, banana, cherry, with Repeated lines 3 and Appear once (hidden) 3.
Limits and edge cases
Lines are compared whole, so "apple" and "apple pie" are different lines, and near matches that differ in punctuation or accents are not merged. Trim whitespace also strips indentation from the lines you keep. Invisible characters such as zero-width spaces are not trimmed and can make look-alike lines differ. To sort without removing anything, use the Line Sorter. To tidy stray spaces first, try the Whitespace Cleaner, and to rewrite text before comparing, use Find and Replace.
Frequently asked questions
How do I remove duplicate lines and keep the original order?
Paste your text and leave Sort A to Z off. The tool keeps the first copy of every line where it first appeared and drops the later copies, so the order of your list does not change.
How do I find duplicate lines without deleting them?
Turn on Show only duplicates. The result then lists one copy of each line that appears more than once, in order of first appearance, and the counts show how many lines repeat and how many appear only once.
Why are two identical-looking lines not treated as duplicates?
Usually a difference you cannot see: a trailing space, a different capital letter, or an invisible character. Turn on Trim whitespace and turn off Case-sensitive to ignore spaces and capitals. Zero-width characters are not removed by trimming.
Can I remove duplicates from an Excel or Google Sheets column?
Yes. Copy the column, paste it into Input text so each cell is one line, copy the result and paste it back. If you copy several columns, each row is compared as one whole line, so a row is only removed when every cell matches.