Lisp as Blub
There's a problem in the server software. When the load gets high, it fails catastrophically instead of gradually. Robert and Patrick Collison are investigating, but they're still not sure what the problem is. My guess from the external evidence is that it's related to garbage collection.
Killing the server process fixes the problem, at least for a day or two.
And there's the problem with Lisp for writing server software. Long lived processes, shared state threading, and garbage collection make it extremely difficult to fail gracefully. Even if your code is completely correct and bug free, it can still crash, hang or just run unacceptably slow and there is nothing you can do to correct it without completely restarting.
There is no macro or meta programming technique to fix this problem. There are things you can do to mitigate it (mostly by generating less garbage), but once you reach a certain level of activity in the system where the garbage collector can no longer keep up (and it will happen), then every line of code in your system is now a potential failure point that can leave the whole program in a bad state. Lisp has this problem. Java has this problem. Erlang does not.
Posted April 14, 2008 8:22 AM