UNSTABLE Releases: [0.3.1.652] Encoding (UTF-8 / ANSI) problems

BrankGnol: Your english looks pretty nice, take a look at mine: :)

Well, there are few things to know about ANSI encoding and the detector (metacrawler plugin). ANSI encoding is a generic encoding for 7-bit encodings. ANSI files therefore contains only character codes under 128 (decimal). When you open a file in Intype, the encoding detector scans first 32kB of the file. If all characters in this first block have lower code than 128, then it returns ANSI. So even if you are saving file using UTF-8 Plain (do not confuse with UTF-8), the detector still fallbacks to the ANSI. We will be working more on this, so it will fallback to UTF-8 Plain if you will set it in the preferences. If you select ANSI and you will use characters with codes greater or equal to 128, Intype will save the file with your current system code-page (we are working on supporting various codepages).

The difference between UTF-8 and UTF-8 Plain is that the plain version is not storing the BOM bytes to the beginning of the file. So another opening of file in UTF-8 Plain may sometimes result in detecting ANSI. However, using UTF-8 saves the BOM to the beginning of the file and the detector should never return ANSI at next open.

Another thing is that the conversion of the encoding is not made immediately after selecting different encoding, but is done at save. Intype keeps all files in UTF-16 LE in the memory, and the conversion is made only at the save time.

Using UTF-8 (not the UTF-8 Plain), should work without problems. If you have problem with UTF-8, it’s a bug. Can you, please, confirm that you are using UTF-8 (not the UTF-8 Plain), so we can declare this as a bug? Or am I completely wrong, and the problem is somewhere else? :)

This will require the use of fall back fonts for missing glyphs – for example if you’re using Consolas (which is, I agree, beautiful) and trying to display these characters:

埨仝佲텇漊㌞㌶ – (see Note 1, below)

you’ll get little ‘missing glyph’ blocks instead, because Consolas doesn’t have glyphs for these characters – not many western fonts do.

What intype would need to do is to use a different font (or fonts) to display these characters – ones which do have glyphs for them – such as ‘Arial Unicode MS’ – which has pretty much the whole Unicode range (it’s a 25Mb font file!)

This is what (properly setup) modern web browsers do – no font can be expected to contain glyphs for every single Unicode character, so you need to use the user’s chosen font when you can, but fall back to other fonts for any (usually international) characters which might be missing.

This might be more complex for an editor like intype though, as it’s not just read-only like a web browser and it would also almost certainly mean sometimes mixing mono-spaced and proportional fonts in the same document, line or even word.

Note 1: These are just randomly chosen Chinese/Japanese/funny looking Unicode characters – I’ve no idea what they mean. If you’re seeing little boxes, blocks or question marks instead, you’re either using an old web browser, or it’s not setup correctly. Try using ‘Arial Unicode MS’ for the international fonts in your browsers preference settings.

No, you’re right, you do need to save to have then encoding change take effect. If I recall correctly Intype uses CR+LF and UTF-16 internally, and converts to the chosen settings on save. Infact, this brings me onto a “bug” I thought might exist the other day, but forgot to check:

If the file is set to “LF” endings then one would expect searching for “blah\n” might work, but as the file is stored internally as CR+LF you have to search for “blah\r\n”.