Damien Katz

Why not extend the OTP VM to add decent globalization features to the language?

I dive right in and start reading about Unicode's collation algorithm,

http://www.unicode.org/reports/tr10/tr10-18.html#S1.1

and ... I turn into a pillar of salt (or maybe a pillar of MSG). I haven't found where ICU uses generated code except maybe for tables read from binary files and translated to C array decls/defs. AFAICT, there's no place where you can just start changing a table so that tools spit out Erlang instead of C/C++ or Java. You've still got to re-implement in Erlang what's been done in Java and C++ in Erlang. That looks like a lot of code, though it might be mostly belaboring the obvious in in-line comments combined with further obfuscating the obscure in the case of gratuitous abstractification.

I started feeling better after reading this PPT,

http://icu-project.org/docs/papers/normalization_iuc21.ppt

but maybe only because it feels SO good when you stop reading Unicode standards.

Since Unicode processing is pretty much just one big case analysis anyway, special-casing the simplest thing that would do the most good is quite in the spirit of the system. As the above PPT points out, most strings are already in normal form. Quickcheck (to see if a string is already in normal form) is very fast, and it's a fairly short bit of code. Erlang is used mostly in the West at the moment, so for most purposes, it's probably enough to just generate Erlang code for Anglo/Euro collation orders, if that simplifies things at all (n.b. not saying it does).

So the ticket here seems to be: incrementally implement algorithms that hit the eject button and pop the ICU parachute upon encountering anything too exotic or scary in mid-flight. Get the basic algorithms working, then, if they seem to need more speed, make them incremental. Most collation decisions are made within the first few characters -- if you can see where and how it's OK to do those comparisons in Erlang, you're probaby halfway to optimal.

(I hope pointing to resources where people talk about what's been done counts toward "getting it done", and not just as "talking about it".)

Unfortunately, I don't think this is going to be efficient enough. I tried implementing ASCII collation is Erlang, and it was still slower than the port callout to ICU. ICU in C is pretty much ideal, as it is highly optimized in a way that's impossible in Erlang.

Yeah, that's unfortunate. Could you please elaborate a little on "is highly optimized in a way that's impossible in Erlang"?

What about creating a linked in driver around ICU?

I know this is sort of frowned upon... But if you need speed it can work fine. Also with some good defensive coding the Linked in library should be plenty stable. And, if it does crash... Well CouchDB was designed around that problem.

Vlad, "Impossible is erlang" is perhaps overstating it a bit, but the truth is the ICU library is low level optimized and getting similar levels of performance from Erlang just isn't possible with the current VM.

Ray, our ICU integration already is a linked-in driver, which currently must be exposed to Erlang as a port. And creating and sending binaries to it is more expensive than doing the equivalent in an Erlang BIF. I also don't worry about crashes from the ICU, both for the reason you gave and the fact that the ICU is a very reliable and well tested library and is unlikely to crash, ever.

It could be used Haskell to solve the performance problems in Erlang, which are both IO and string manipulation.

http://www.haskell.org/haskellwiki/Applications_and_libraries/Interfacing_other_languages/Erlang

http://github.com/esessoms/haskell-interface

I understand that you are using a linked in driver to wrap the ICU library.

Are you sure you are using the most efficient
way of calling the driver?

To avoid copying of binaries in calls to the driver you should implement and use the outputv
call back of the driver.

Does your driver allow per port or per driver
parallelism (i.e locking per port or per the whole driver)?
If the driver is reentrant and allows several ports you can utilize multicore much better (if you are calling the driver from several Erlang processes in parallell)

How do you create and return the result from the driver?

If you have all this as optimal as possible the difference compared with a BIF is not as big as you think.
But I agree that dynamically loadable BIF's would be a good idea since you will get rid of some locking in the SMP case and the need to pass around a port or use a named port dissapears.

Hi damien, I just stumbled across this post.

If I remember right from my work at a native json implementation ( eep0018), which paul took over, there are *two* binary encodings between Erlang OTP and native drivers. The official one - which is suppossedly stable between OTP releases - and the "internal" one, which is not even properly documented, but can be accessed in a not-too-low level way via Erlang OTP headers and such.

This was the key in Paul's final success in getting blazing speed out of the eep0018 driver, and this might improve the throupghput between CouchDB/Erlang and the ICU driver.

After announcing NIFs in Erlang it could be possible to write bindings for ICU C functions.