Delimiter.live
How-to guide

How to Fix Broken Characters in Text

Broken characters almost always mean text encoded as UTF-8 was read as something else. The pattern tells you which: é in place of é means UTF-8 read as Windows-1252, and a black diamond question mark means bytes that were not valid in the encoding used. Fix the declared encoding first, and only clean the text if the original is unrecoverable.

Text Cleaner

Strip formatting, smart quotes and invisible characters.

Open the tool

Garbled text has a small number of causes and each leaves a recognisable fingerprint. Learning to read the fingerprint is faster than guessing, because it tells you where in the pipeline the damage happened — and whether it can still be undone.

The important distinction is between text that is merely displayed wrongly and text that has been irreversibly damaged. Mojibake usually means the bytes are intact and only the interpretation is wrong, which is fixable. A replacement character means information was already thrown away.

Step by step

  1. Read the fingerprint

    é, è or ’ means UTF-8 was read as Windows-1252 — the bytes survive and the fix is the declared encoding. A replacement character or a bare question mark means bytes were dropped during a conversion and the original is gone.

  2. Fix the declaration, not the text

    If the bytes are intact, the repair is upstream: the charset in a Content-Type header, a meta tag, a database connection charset, or the encoding argument when the file was read. Editing the visible characters treats the symptom.

  3. Check for invisible characters

    Zero-width spaces, non-breaking spaces and byte-order marks are invisible but break comparisons, searches and imports. Text pasted from a web page or a word processor routinely carries them.

  4. Normalise smart punctuation

    Word processors substitute curly quotes, en and em dashes and ellipses. They are valid characters, but they break code, CSV parsing and exact-match lookups. Convert them to their plain equivalents when the text is destined for a machine.

  5. Inspect the actual code points

    When two strings look identical but do not match, escape both to see the code points. That exposes a non-breaking space masquerading as a space, or an accented letter written as a base letter plus a combining mark.

Example

The first is recoverable — the bytes are intact. The second is not.

Symptom

Café résumé

Caf? r?sum?

Cause and fix

UTF-8 read as Windows-1252.
Bytes intact — fix the declared
encoding and it reads correctly.

Converted to an encoding that had
no room for those characters.
The original is unrecoverable.

Frequently asked questions

Why does é show as é?
In UTF-8 é is two bytes, C3 A9. Read as Windows-1252, those two bytes are the characters à and ©. Nothing is lost — the text is simply being interpreted with the wrong encoding, and correcting the declaration fixes it.
Can I recover text showing question marks?
Usually not. A question mark or replacement character means the conversion had no way to represent the original character and substituted a placeholder. The information is gone from that copy; you need the source.
What is a byte-order mark?
An invisible marker some editors put at the start of a UTF-8 file. It is harmless when expected and a nuisance when not — it can appear as , break a JSON parser on the first character, or turn the first CSV column header into something no lookup matches.
Why do two identical-looking strings not match?
Usually an invisible difference: a non-breaking space instead of a space, a zero-width character, or an accented letter written as a base letter plus a combining mark rather than a single code point. Escaping both to code points shows it immediately.
How do I stop this happening again?
Use UTF-8 everywhere and declare it everywhere — HTTP headers, HTML meta tags, database and connection charsets, and the encoding argument wherever files are read or written. Almost all mojibake comes from one link in that chain disagreeing.

Tools used in this guide

All tools →
All guides →

Last reviewed .