- What a character encoding decides
- How UTF-8 and Shift_JIS differ
- Why text gets garbled, and what to check when you receive such a file
The short answer
A character encoding is the rule for which number (sequence of data) each character is stored as. Most garbled text happens when the encoding used to save a file differs from the one used to open it. The file is not necessarily broken, so start by opening it again with the right encoding specified, without saving over it.
What an encoding decides
Computers do not store characters as they are. They store them as numbers, and which number a given character becomes depends on the encoding.
So the same data can show up as different characters if the reader interprets it with a different encoding. When a file does not clearly state which encoding it was written in, the program opening it either guesses or uses a fixed encoding. If that guess is wrong, you see a jumble of meaningless symbols or characters. This is garbled text, called mojibake in Japanese.
The file is not necessarily broken. Check that the encoding used to save it matches the one used to open it.
UTF-8 and Shift_JIS
In Japan, two encodings come up again and again. Shift_JIS is an encoding designed for Japanese, still common in files exchanged with Japanese businesses and older Japanese systems.
| UTF-8 | Shift_JIS | |
|---|---|---|
| Characters covered | Most of the world's scripts | Mainly Japanese plus English letters and digits |
| Where it is common | Web pages, newer systems and services | Older business software, files made on Japanese-language Windows |
| Size of one Japanese character | Usually 3 bytes | Usually 2 bytes |
| Watch out for | Older software may garble it | Some characters cannot be represented and may be replaced when converting |
A byte is a unit of data size. Neither encoding is "right"; what matters is that both sides use the same one.
What to check when text is garbled
- Keep the original file: If you save over the garbled version, you may no longer be able to recover the original.
- Ask the sender: Asking which encoding they saved it in is the most reliable step.
- Open it with the encoding specified: For Excel, Microsoft's support page describes importing via From Text/CSV in the Get & Transform Data group on the Data tab. Its page on the Text Import Wizard says that File origin is where you select the character set used in the text file. It also says the wizard is a legacy feature that you might need to enable in newer versions. Menu names and locations may differ depending on your version of Excel.
- Try a text editor: Many text editors let you reopen or save a file with a chosen encoding. Steps differ by program, so check its help pages.
When you send a file
When you are the sender, add a short note about the encoding you used. For example:
The attached file is saved in UTF-8. Please let me know if any text looks garbled.If the recipient's system can only read Shift_JIS, converting from UTF-8 may replace some kanji in people's names, or some symbols, with "?" or similar. After converting, check by eye that nothing was replaced. Problems specific to CSV, such as leading zeros disappearing, are covered in our related article on CSV.
In summary
Checklist
- Did you keep the garbled file as it is, without saving over it?
- Did you ask the sender which encoding they used?
- Have you checked how to specify the encoding when opening a file?
- When sending, did you say which encoding you saved in?
- After converting, did you check that no characters were replaced?
One thing to check today
Decide now that when you receive a garbled file, you will first ask the sender which encoding they saved it in, and take a moment to check how to open a file with a chosen encoding in the software you use every day.