How to Detect and Remove Non-UTF-8 Characters From Text Files

Text files look calm. They sit there like tiny paper squares. Then one day, your app screams. Your CSV import fails. Your log parser chokes. A strange black diamond appears. Welcome to the small haunted house called non-UTF-8 characters.

What are non-UTF-8 characters?

First, a tiny truth bomb. Computers do not store “letters.” They store bytes. An encoding tells the computer how to turn those bytes into readable text.

UTF-8 is one encoding. It can represent English letters, emojis, accents, symbols, and many world languages. It is popular because it works almost everywhere.

A “non-UTF-8 character” usually means one of two things:

• The file is saved in another encoding, like Windows-1252, ISO-8859-1, or Shift JIS.
• The file has broken byte sequences that are not valid UTF-8 at all.

These bytes may show up as weird symbols. You may see �, question marks, boxes, or random goblin text. That � symbol is called the replacement character. It is like your computer saying, “I tried my best.”

Why should you care?

Bad encoding can cause sneaky trouble. It may not break everything at once. It may only break one row in a huge file. Very rude.

Common problems include:

• CSV imports fail.
• JSON files become invalid.
• Database loads stop halfway.
• Search tools miss words.
• Names with accents become messy.
• APIs reject the request.

If you handle data, logs, translations, or user uploads, UTF-8 cleanup is a very useful broom.

Step 1: Make a backup

Before you touch the file, copy it. Always. Encoding fixes can be destructive. If you remove bytes, they are gone.

On macOS or Linux, run:

cp messy.txt messy.txt.bak

On Windows PowerShell, run:

Copy-Item messy.txt messy.txt.bak

Now you have a safety net. Tiny dance optional.

Step 2: Detect the file encoding

Start with the simple tools.

Use the file command

On macOS or Linux:

file messy.txt

You may see something like:

messy.txt: UTF-8 Unicode text

Great. Or you may see:

messy.txt: ISO-8859 text

That means the file is probably not UTF-8. The word “probably” matters. Encoding detection is not magic. It is detective work.

Use iconv to test UTF-8

iconv can try to read the file as UTF-8. If it finds bad bytes, it complains.

iconv -f UTF-8 -t UTF-8 messy.txt > /dev/null

If the file is valid, nothing exciting happens. If not, you may see an error like:

illegal input sequence at position 1234

That position tells you where the gremlin lives.

Use Python for a clear check

Python is great for this job. Try this script:

from pathlib import Path

path = Path("messy.txt")

try:
    path.read_text(encoding="utf-8")
    print("Valid UTF-8")
except UnicodeDecodeError as e:
    print("Invalid UTF-8")
    print("Problem at byte:", e.start)
    print("Reason:", e.reason)

This is simple and friendly. It tells you if the file is valid UTF-8. It also points to the problem byte.

Step 3: Find the bad bytes

Sometimes you want to see the exact area. Use xxd or hexdump. These tools show raw bytes.

xxd messy.txt | less

This looks scary at first. It is just bytes in costume. If iconv says the error is near byte 1234, you can inspect that area.

You can also use Python to print a small chunk around the problem:

from pathlib import Path

data = Path("messy.txt").read_bytes()

try:
    data.decode("utf-8")
except UnicodeDecodeError as e:
    start = max(e.start - 20, 0)
    end = e.start + 20
    print(data[start:end])

This shows the raw bytes near the error. It may help you guess the original encoding.

Step 4: Convert the file to UTF-8

If the file is not broken, but just saved in another encoding, convert it. This is the cleanest fix.

For example, many old Windows files use Windows-1252. Convert like this:

iconv -f WINDOWS-1252 -t UTF-8 messy.txt > clean.txt

For ISO-8859-1, use:

iconv -f ISO-8859-1 -t UTF-8 messy.txt > clean.txt

Then test the new file:

iconv -f UTF-8 -t UTF-8 clean.txt > /dev/null

If there is no error, nice work. The text dragon has been brushed.

Step 5: Remove invalid UTF-8 bytes

Sometimes you do not know the original encoding. Sometimes the file is a data swamp. You just need invalid bytes gone.

With iconv, you can ignore invalid sequences:

iconv -f UTF-8 -t UTF-8 -c messy.txt > clean.txt

The -c option removes invalid characters. It is fast. It is also a little dangerous. It deletes data silently.

Another option is to replace bad bytes with a marker. Python can do this:

from pathlib import Path

raw = Path("messy.txt").read_bytes()
text = raw.decode("utf-8", errors="replace")
Path("clean.txt").write_text(text, encoding="utf-8")

This keeps the file readable. Bad parts become �. That can be useful when you want to review the damage.

If you want to remove bad bytes with Python:

from pathlib import Path

raw = Path("messy.txt").read_bytes()
text = raw.decode("utf-8", errors="ignore")
Path("clean.txt").write_text(text, encoding="utf-8")

Use ignore only when losing those bytes is acceptable.

What about weird but valid characters?

Here is a fun twist. Some strange characters are valid UTF-8. Emojis are valid. Accented letters are valid. Smart quotes are valid. The problem may not be UTF-8 at all.

You may want to remove control characters. These are invisible characters that can break files. Tabs and newlines are usually fine. Others can be trouble.

This Python script removes most control characters but keeps tabs and line breaks:

from pathlib import Path
import unicodedata

text = Path("clean.txt").read_text(encoding="utf-8")

filtered = "".join(
    ch for ch in text
    if ch in "\n\r\t" or unicodedata.category(ch)[0] != "C"
)

Path("cleaner.txt").write_text(filtered, encoding="utf-8")

Quick checklist

• Back up the file. Never skip this.
• Test if it is valid UTF-8. Use iconv or Python.
• Guess the original encoding. Try file and your data source.
• Convert when possible. This is better than deleting bytes.
• Remove or replace only if needed. Keep data loss small.
• Check the result. Validate the final file as UTF-8.

Final thoughts

Encoding issues feel spooky. But they follow rules. A text file is just bytes. UTF-8 is just one way to read those bytes. Once you know that, the monster gets much smaller.

Use detection first. Convert second. Delete only as a last resort. With a backup, a few commands, and a calm face, you can turn a messy file into clean UTF-8 text. No wizard hat required.