Problem: UnicodeDecodeError reading files — file encoding vs default encoding mismatch —— Trove problem library

What you will run into

open(path).read, pandas.read_csv(path), configparser etc. throw when reading files: on Windows Chinese locale 'gbk' codec can't decode byte 0xad ...; on Linux/mac 'utf-8' codec can't decode byte 0xd6 ....

The file came from someone else / was copied from an old system / saved via Excel. The file isn't broken, and the code logic isn't wrong — it's a mismatch between the file's encoding and the reading environment.

Why this happens

open without an explicit encoding uses the system default encoding:

When the file's actual encoding ≠ default, decoding throws UnicodeDecodeError at the first non-ASCII byte.

open without encoding uses the "system default encoding" (decided by locale), not the "file's actual encoding" — Python cannot know what encoding a file is, so it decodes under the default assumption.

Chinese Windows defaults to GBK (cp936), Linux/mac to UTF-8; when the file's encoding differs from the default, decoding throws UnicodeDecodeError at the first non-ASCII byte.

Deterministic takeaway: the codec name in the error is the current environment's default encoding (gbk codec can't decode = the file is UTF-8 while the environment defaults to GBK) — the file is not "broken";

the fix is explicit encoding: open(f, encoding="utf-8") for UTF-8 files (or utf-8-sig for BOM compat), encoding="gbk" for legacy GBK files;

scripts that run on both Windows and Linux must specify encoding explicitly, otherwise the same script behaves differently per machine; fixed diagnostic: file <name> reveals the real encoding, then compare with the codec name in the error.

Step by step

1. Check the file's real encoding first — Required

file xxx.txt

Output shows UTF-8 / ISO-8859 (GBK) / GB18030 etc.; or xxd xxx.txt | head to look at bytes.

2. Specify encoding explicitly when reading (the right fix) — Required

Reading a GBK file:

open(path, encoding='gbk')

GB18030 is more complete: encoding='gb18030' is GBK-compatible and doesn't lose rare characters. Reading a UTF-8 file: open(path, encoding='utf-8') (stable on Windows too); pandas: pd.read_csv(path, encoding='gbk').

3. Transcode once (root fix, good for batches) — Required

iconv -f gbk -t utf-8 old.txt > new.txt

Or save-as UTF-8 in Excel; standardize all text files as UTF-8 in the project.

4. Tolerant fallback — Required

open(path, errors='replace') or errors='ignore' won't throw — but bad bytes become �, irreversibly losing content; only for logs/non-critical parsing.

5. When stdout is also garbled — Required

Printing Chinese under a GBK locale garbles output; add PYTHONIOENCODING=utf-8 (or sys.stdout.reconfigure(encoding='utf-8')) — that's an output-encoding issue, debug separately from reading.

How to confirm it works

Explicit-encoding reads work (measured output shape):

encoding='gbk' reading a GBK file → normal output 「中文测试行」 (exit 0, measured)
Under GBK locale, encoding='utf-8' reading a UTF-8 file → normal (exit 0, measured)

Reproduction baseline: default read of a GBK file → 'utf-8' codec can't decode byte 0xd6 (exit 1, measured).

Other environments

Symptom

open(path).read、pandas.read_csv(path)、configparser etc. raise while reading:

UnicodeDecodeError: 'gbk' codec can't decode byte 0xad in position 2: illegal multibyte sequence
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd6 in position 0: invalid continuation byte

Key traits: Chinese Windows (default GBK/cp936) reading a UTF-8 file reports 'gbk' codec can't decode — most common; Linux/mac (default UTF-8) reading GBK/legacy files reports 'utf-8' codec can't decode; the file came from someone else / an old system / an Excel save-as (GBK or GB18030); file xxx shows ISO-8859-like (not UTF-8).

Cause

open without encoding uses the system default encoding: GBK (cp936) on Chinese Windows, UTF-8 on Linux/mac. When the file's actual encoding differs, decoding fails at the first non-ASCII byte. This is an encoding mismatch, not a corrupt file and not a logic bug.

Solution

1. Check the real encoding first: file xxx.txt (UTF-8 / ISO-8859 (GBK) / GB18030), or xxd xxx.txt | head to inspect bytes.
2. Specify encoding explicitly when reading (the fix):

• GBK file: open(path, encoding='gbk') (or encoding='gb18030', a superset of GBK that also covers rare characters);
• UTF-8 file: open(path, encoding='utf-8') (stable on Windows too);
• pandas: pd.read_csv(path, encoding='gbk').

1. Transcode once (root fix for batches): iconv -f gbk -t utf-8 old.txt > new.txt (or save-as UTF-8 in Excel); standardize all text files to UTF-8 in the project.
2. Tolerate as a fallback: open(path, errors='replace') or errors='ignore' stops the exception — but bad bytes become �, content is irreversibly lost; only for logs/non-critical parsing.
3. If stdout is also garbled: printing Chinese under a GBK locale garbles output — set PYTHONIOENCODING=utf-8 (or sys.stdout.reconfigure(encoding='utf-8')); that's an output-encoding issue, separate from reading.