Remove duplicates in Excel, including the near-duplicates
Excel's built-in Remove Duplicates only catches rows that match character for character, so "Acme Ltd" and "ACME Limited " both survive. This prompt has your todo.is agent find exact and near duplicates by the columns you choose, decide which row to keep by your rule, merge missing details into it and list every removed row so you can undo anything.
The prompt
- Remove duplicates in the attached file [FILE NAME]. Treat rows as duplicates when [MATCH RULE]. Ignore case, extra spaces and punctuation when comparing, and treat emails as the same regardless of capitals. Also look for near duplicates (like "Jon Smith" vs "John Smith" at the same company) but only flag those, do not remove them. When duplicates are found, keep the row that [WHICH ROW TO KEEP], and copy any details from the removed rows that the kept row is missing. Give me an .xlsx with: "Deduplicated" (final list), "Removed" (every removed row with the ID of the row it was merged into), "Possible duplicates" (flagged pairs with a similarity note) and a short summary at the top.
What to change
- [FILE NAME]: Attach the Excel or CSV and write its name, e.g. "contacts_all.xlsx".
- [MATCH RULE]: e.g. "the email is the same", "first name, last name and postcode match" or "the order ID repeats".
- [WHICH ROW TO KEEP]: e.g. "was updated most recently", "has the most filled-in fields" or "comes first".
Example result
- contacts_deduplicated.xlsx (example)
- Summary
- • Rows in: 5,212
- • Duplicates removed: 344 (matched by email, ignoring case and spaces)
- • Rows out: 4,868
- • Details merged: 96 kept rows got a phone or job title from a removed copy
- • Possible duplicates flagged, not removed: 41 pairs
- How it decided
- • "ANNA.KOVAC@NORDFIELD.IO " and "anna.kovac@nordfield.io" were treated as the same email
- • The row updated most recently was kept
- • If the kept row had an empty Phone and a removed copy had one, the phone was copied over
- Removed sheet (sample)
- • Row 2,019 Anna Kovac · nordfield.io · updated 2025-11-02 → merged into row 88 (updated 2026-08-14)
- • Row 3,301 Leo Ferrari · ferrari.design · no phone → merged into row 1,207
- Possible duplicates (sample)
- • Jon Smith (jsmith@harbor.co) and John Smith (john.smith@harbor.co): same company and phone, different emails. Likely one person
- • Priya N. and Priya Nair at Quillstone: same city and job title
- • info@baylight.com appears for 3 different names, a shared inbox, so they were kept
- Excel tip for next time
- To do a simple exact check yourself: Data > Remove Duplicates, then pick the Email column. To spot repeats first without deleting, use =COUNTIF(C:C, C2)>1 in a helper column.
- Reply "merge the Jon/John Smith pairs too" to apply any flagged pair.
How to do it with todo.is
- Attach the spreadsheet and copy the prompt with your match rule.
- Paste it into todo.is, or send the file and prompt to your agent on Telegram.
- Your agent compares rows, merges details, and lists removed and possible duplicates separately.
- Check the possible duplicates, then tell your agent which ones to merge.
Tips for a better result
- Pick the most reliable match column. Email or customer ID beats name, because names repeat and get misspelled.
- Decide which row wins before you start: newest, most complete or first. It changes the result.
- Never delete without a "Removed" list. It is your undo button.
- For lists from several sources, ask your agent to merge the files and remove duplicates in one go.
remove duplicates in Excel: FAQ
- How do I remove duplicates in Excel? Select your data, go to Data > Remove Duplicates and tick the columns that must match. It only removes exact matches, so clean spaces and case first.
- How do I find duplicates without deleting them? Use Conditional Formatting > Highlight Cells Rules > Duplicate Values, or a COUNTIF helper column. Your agent can also just flag them.
- Can it remove duplicates based on two columns? Yes. Say for example "first name and last name and postcode must all match", and only rows matching on all of them are treated as duplicates.
- Does Excel's Remove Duplicates ignore case? Yes, it treats "ANNA" and "anna" as the same, but extra spaces or punctuation make values different.
JavaScript is required to use the todo.is app.