UTF-8 support for Palm OS: How hard would it be?

[ipsi]

Having been using CJKOS for a while, I finally realised that having it only support GB2312, GBK, and Big5 directly is a big nasty disadvantage, in my opinion... While it's not really noticeable when using my Treo normally, it has become apparent.

e.g. The developer was obviously a bit lazy, so that when I view a UTF-8 webpage with traditional characters, but the display charset is set to GB2312, it will use a code-page conversion to convert from UTF-8 to GB2312, but it won't think about trying to convert to Big5, so it won't display the traditional characters, but only those whose simplified form is the same as their traditional. Now, that's not cool. It's also obvious in other contexts (e.g. I need to use AppLocal to load Palm Desktop with GB2312 in order to view my Chinese Contacts, Chinese Memos, etc).

But I suspect that it gets worse. If you're studying, say, Japanese and Chinese, then you're going to have a lot of fun trying to text properly in both languages (not that any cellphone can display both at the same time).

It also causes problems with other applications. Certain accented characters (é) have the same byte-code as CJK characters, meaning that I need to turn CJKOS off to read my eBooks.

I'm not sure if changing the encoding would make a difference, but it might.

The one other thing I've noticed is that you can't enter Kanji for Japanese with CJKOS. Now, this isn't a problem for me, but I imagine it would be for Japanese speakers...

So, does anyone have any idea how hard this would be to do for Palm OS as a whole? Mike, I believe you're using UTF-8 internally for Pleco? Any comments on this?

It'd also be nice to have only one piece of software for any given language.

I'm really just thinking out loud with this, but it does annoy me that CJKOS thinks encoding everything in non-unicode formats is a good idea. I'm also well aware this would cause several problems (Text Messaging being the biggest), but I'm sure there are ways around that.

 

[mikelove]

Changing the encoding wouldn't help with viewing regular ASCII documents, unfortunately; an é in regular 8-bit ASCII would not be a valid UTF-8 encoding, UTF-8 is robust enough that the system would probably recognize that fact and display it as a blank or a question mark rather than spewing out a stream of garbage characters but you'd still want to disable UTF-8 support when viewing non-UTF-8 documents.

I'm not exactly sure what mechanism CJKOS uses to hook into Palm OS' text handling functions, but I don't know of any technical reason why it couldn't be expanded to work with UTF-8. The main question is whether or not anyone would bother developing such a system at this point, with the new and much more Unicode-friendly Linux-based Palm OS versions coming out in the next year; by mid-'09 you likely won't even be able to buy a Palm without built-in Unicode support anymore. There's still a market for regular Palm OS software, since both of those Linux-based systems will supposedly run older Palm applications quite well, but for a system hack like CJKOS it would be really hard to justify investing any more time or money in development at this point.

We actually use UTF-16 internally for Pleco; it's a lot easier to work with (particularly on Windows CE, where it's the OS's native character encoding) and since the dictionary data is all compressed anyway it doesn't really take up any extra storage space. Really that's the way things are moving in general, they made the switch on desktops years ago and now PDAs have enough memory that there's really no reason for anyone to keep using 8-bit characters.

 

[ipsi]

I've been hearing about Linux-based PDAs for a while now, but has anyone actually seen anything concrete? I've also just bought a new 680 and have no desire to go buying another phone for quite some time yet.

But yeah. It'd be a lot of work. There are a couple of reasons for doing so: 1) As a learning experience. 2) So that people can have a free way to deal with foreign languages, assuming you'd make it open source. There don't seem to be many Palm Apps that are. Why is that? 3) because you hate the old one-encoding-per-language style of doing things *that much*.

Apart from that, no actual, sensible ones that I can think of, except maybe to protect against the fact that Linux-based PDAs may take rather more than a year to come out...

 

[mikelove]

They're definitely coming soon, ACCESS showed a pretty-much-final version of theirs at LinuxWorld recently along with some third-party software to go with it and Palm's version has been making the rounds with developers too.

I think the lack of open-source on Palm may have to do with a combination of difficult-to-find development tools (see my other post on the merits of CodeWarrior) and the fact that it's so thoroughly unpleasant to write code for that you're not likely to find that many people willing to do it without getting paid - the memory management alone could drive a person insane.

 

[ipsi]

Oh? That's sounding good then.

Hehe. I guess it's not fun developing in regular C then? I've never tried developing a Palm App, but you make it sound like it's going to be harder than I had thought. And isn't memory management always the killer for hand-held devices?

The other thing dissauding OSS Palm Developement would be that CodeWarrior isn't free. While it's not exactly essential that there are free IDEs and Compilers for OSS to work, it does tend to help. And if PODS does have enough issues that it isn't good for large programs, then yeah, I can kinda understand why there isn't so much. That, and most OSS fans are probably working on slightly (significantly?) easer desktop apps.

Gah. Just had a look at the price - $400 is a bit on the high side if you're not getting paid for your development work.

 

[Ma Qinming]

I'm entering this conversation a bit late, so my apologies. So, if I want to use the memo function on my Tungsten E2 to read simple, and relatively short Chinese text (and not lose my ability to use Pleco or work in English everywhere else on my Palm),I have no choice but CJKOS? Sounds as if it is a bit flawed. Are there no other simple options out there? Thanks.

 

[ipsi]

As far as I'm aware, yes. There may be others out there, but CJKOS is the only one I know. It's not that bad, for the most part.