I think we’ve solved most of the issues with special characters (such as umlauts) in Western, single-byte languages for Cerb4.

Cerb4 Build: 563

Due to a few lingering obstacles for full internationalization, we’re still using ISO-8859-1 encoding. That should completely cover English, German, Spanish, Italian, Norwegian, Icelandic, Swedish and others. It should mostly cover Dutch and French (with the exception of a few rare characters).

Warning! Dear Non-Techie friends, the rest of this post gets a bit nerdy.

The recent issues were caused by a couple things:

• META/Content-Type: We were a bit over-eager and had the browser using Unicode (UTF-8) encoding while the data was still Western Latin (ISO-8859-1). Display-wise this is entirely fine because of backwards compatibility — but when input, like forms, was sent from the browser to PHP, it could include multi-byte characters that the database would split up into seemingly random single byte characters. This affected things like umlauts and the British pound sign, not just fully multi-byte languages like Japanese, Chinese and Russian.
• Ajax/XHR: Modern browsers default their XHR requests (how Ajax happens) to UTF-8 encoding when no Content-Type header is provided. This is why even with ISO-8859-1 encoding, umlauts would break on dynamic functionality in the helpdesk (ticket peek, reply, templates).
• UTF-8 vs Unicode: Unicode is a variable byte encoding which overlaps with ISO-8859-1 completely on the first byte. UTF-8 overlaps with ISO-8859-1 over the first 7 bits (which is basically all the characters printed on the keys of a standard US Keyboard, including the SHIFT characters). Extended characters, such as umlauts, are expected in a second UTF-8 byte (0xC3BC vs 0xFC).

We’d love to switch over to UTF-8 immediately and support almost every language, but here’s what’s in the way:

• PHP6 is on the way with native Unicode support for all strings. This is much cleaner than anything we can do with the mbstring extension.
• mbstring function overloading (e.g. substr() -> mb_substr()) can simulate native UTF-8 strings with PHP5, but it requires php.ini changes. These changes will usually affect all PHP scripts on your webserver. Under ideal conditions we can control this with Cerb4 and only affect our script. However, real-world conditions are rarely ideal and this would make for a terrible minimum requirement.
• Our database abstraction (MySQL, PostgreSQL, etc.) makes UTF-8 support in the database layer more complicated. This would be much easier if we knew we were always dealing with MySQL 4.0+. We may need to make some tough decisions here (though they’d allow us a lot more efficiency on things like fulltext index searching versus our current watered-down alternatives).
• Converting existing databases to UTF-8 isn’t a one-click process. Ideally we’ll only have to help people convert their existing database if they need UTF-8 support. It can also be the new default. Our current goal is to make sure we can support UTF-8 optionally without breaking every existing Cerb4 helpdesk to get it. Data-wise we’re probably fine here, but we’d run into the same issues as described above until the database was converted to UTF-8. The MySQL default is ISO-8859-1 (which it dubs “latin1”).

UTF-8 hangups aside, this should fix the main issues with Cerberus Helpdesk for our friends in Germany, Austria, France, Belgium, Italy, Sweden, Norway, Spain and the Netherlands (who have been rightfully banging down our door about it!)