I learned a new word today: mojibake!
Some of my old blog posts have random gibberish in them like ñ or ’. I always assumed this was the result of some character encoding mismatch but I didn’t know exactly what was going on. While I was fixing this issue via find/replace technology, I did some reading to learn more.
The short version is a mismatch between UTF-8 and Latin-1 (ISO-8859-1) encodings. As an example, this is how ñ becomes ñ.
First, when I type ñ in my CMS it is represented in UTF-8:
>> "ñ".codepoints
=> [241]
>> "ñ".bytes
=> [195, 177]In UTF-8, codepoints above 127 are represented with more than one byte. You can see here the codepoint 241 (U+00F1) corresponds to two bytes, 195 and 177.
Then, when these bytes get written to the DB, they are not converted from UTF-8 to Latin-1. We store [195, 177] but believe they are Latin-1. Later, we re-encode these bytes as UTF-8 to convert from not-actually-Latin-1:
bytes = "ñ".bytes # => [195, 177]
latin1_str = bytes.pack("C*").force_encoding("ISO-8859-1")
puts latin1_str.encode("UTF-8") # => ñI was fixing these bugs in old posts and one AI result said, “Oh yeah, this is a classic mojibake problem.” My first reaction was, “I don’t know man–How ‘classic’ can it be if I’m hearing it for the first time after working with strings and encodings for years?” But then again, you have to learn everything sometime. Maybe it is classic!
Mojibake or 文字化け is a Japanese word for exactly this kind of garbled text. Before UTF-8, the dominant character encoding in Japan was Shift JIS, and this kind of character mangling was probably common when text interacted with systems that did not convert encodings properly.
The “moji” in mojibake (文字) means text and, yes, it’s the same moji found in emoji (絵文字)! I knew that emoji originated in Japan but I did not know that the name was actually Japanese. I assumed that emo was related to emotion since the first emoji were all faces showing different emotions. Nope! E (絵) means pictures, moji (文字) means text.
Now that UTF-8 is the dominant text encoding, I’m pretty confident you’ll see all the text here correctly and not as a “classic” mojibake issue.