Saving a web page as an MHT file is a convenient way to give an AI assistant a complete offline snapshot. The archive can contain the page HTML, images, stylesheets and other resources in one file. But there is a subtle parsing trap: a perfectly healthy page can appear to contain hundreds of broken Unicode characters when the MHT is decoded incorrectly.
We encountered this while reviewing several DigiStoreSG listings with Claude and Codex. One extraction method reported hundreds of U+FFFD replacement characters—the Unicode replacement marker identified as U+FFFD. It looked like every emoji and symbols such as the subscript in Li-MnO₂ had been damaged.
The listings were not corrupted. The extraction method was.
What caused the false Unicode warning?
An MHT file is not an ordinary HTML document. It is a MIME archive containing multiple parts. Each part can have its own content type, transfer encoding and charset metadata.
In our case, the main HTML part used Content-Transfer-Encoding: binary but did not declare a charset. Calling Python's email library method part.get_content() forced the library to guess how the bytes should become text. That guess was wrong, and valid UTF-8 byte sequences were replaced with U+FFFD.
Once the same raw payload was decoded explicitly as UTF-8, the replacement-character count fell from 327 to zero. The section emoji, bullet points, flag emoji and Li-MnO₂ all appeared correctly.
The safe way to extract HTML from an MHT file
Parse the archive as MIME, retrieve the decoded payload bytes, and then decode those bytes explicitly. Do not decode the entire MHT file as though it were a single HTML document.
from email import policy
from email.parser import BytesParser
from pathlib import Path
mht_path = Path("saved-page.mht")
message = BytesParser(policy=policy.default).parsebytes(mht_path.read_bytes())
html_part = next(
part for part in message.walk()
if part.get_content_type() == "text/html"
)
raw_html = html_part.get_payload(decode=True)
html = raw_html.decode("utf-8")
replacement_count = html.count("\uFFFD")
print(f"Replacement characters: {replacement_count}")
get_payload(decode=True) first handles the MIME transfer encoding and returns bytes. The explicit decode("utf-8") step then converts those bytes into text without relying on an undeclared or guessed charset.
Why errors="replace" can hide the real problem
A common workaround is:
text = raw_html.decode("utf-8", errors="replace")This prevents an exception, but it can silently insert U+FFFD wherever decoding fails. If your next step is to count replacement characters, you may report damage that your own extraction code introduced.
During diagnosis, use strict UTF-8 decoding first:
text = raw_html.decode("utf-8")If strict decoding raises UnicodeDecodeError, inspect the part's headers and raw bytes before choosing another charset. Do not automatically conclude that the live page is corrupted.
How to verify whether corruption is real
Use several checks before recommending that someone rewrite or reupload their content:
- Check the source document. If the listing came from an Excel workbook, inspect the cell value and count U+FFFD there.
-
Check the raw MHT bytes. UTF-8-encoded U+FFFD is the byte sequence
EF BF BD. Its absence does not prove everything is correct, but it is useful evidence. - Decode the HTML MIME part correctly. Use the transfer-decoded payload bytes, followed by explicit UTF-8 decoding.
-
Inspect representative characters. Search for expected emoji, bullet points, accented letters and technical symbols such as
₂. - Compare with the live page. A browser view or correctly decoded page source can settle whether the problem exists only in the extraction pipeline.
If the workbook, raw payload and correctly decoded HTML all contain zero U+FFFD characters, rewriting the listing will not fix anything. It only creates unnecessary work and another opportunity for accidental changes.
A prompt to give Claude or Codex
When attaching an MHT file for analysis, include an instruction like this:
Parse this file as a MIME archive. For the main text/html part, use the transfer-decoded raw payload bytes and decode them explicitly as UTF-8. Do not rely on automatic charset guessing. Before reporting Unicode corruption, confirm that U+FFFD exists in the correctly decoded HTML rather than being introduced by the extraction method.
This does not guarantee that every MHT file uses UTF-8, but it makes the assumption explicit and prevents a silent charset guess from masquerading as evidence.
What Claude and Codex users should remember
- MHT is a multipart MIME archive, not plain HTML.
- Transfer encoding and character encoding are separate layers.
- A missing charset can make convenience methods guess incorrectly.
- U+FFFD may have been introduced during extraction.
- Verify the raw source before asking someone to repair live content.
The broader lesson applies beyond MHT files. Whenever an AI reports widespread encoding damage, confirm which stage converted bytes into text. The confidence of the diagnosis does not compensate for decoding the source incorrectly.
We use AI tools extensively for ecommerce and content workflows. You can also read how we connected Codex to Shopify and what we learned, plus our practical case study on repairing and completing a broken Shopee mass-upload workbook with Claude.