Me and my oldest. I'm under blankie.

Me and my littlest, about 2 minutes ago.

At ETech '08 I met Pete Forde and some other people involved with RubyFringe. We hung out a bit and had some crappy barbeque with people like Anil Dash and Dan Grigsby. It was a really good time and I'm looking forward to being around the types of geeks who would come to RubyFringe.

RubyFringe will be pretty informal. So I'm planning on wearing pajama pants the whole conference, and I'm even going to give my talks in pajama pants. Unless it's too hot, in which case I'll wear my cut-off pajama pants. I might have to buy some new ones, gotta look fresh yo.

I'll be talking about CouchDB, and maybe about what it's like to move your family somewhere cheap and live off savings to build the next great open source database. If you are going, send me feedback what you want to hear about and I'll make sure to cover it.

pg - What happened?

And there's the problem with Lisp for writing server software. Long lived processes, shared state threading, and garbage collection make it extremely difficult to fail gracefully. Even if your code is completely correct and bug free, it can still crash, hang or just run unacceptably slow and there is nothing you can do to correct it without completely restarting.

There is no macro or meta programming technique to fix this problem. There are things you can do to mitigate it (mostly by generating less garbage), but once you reach a certain level of activity in the system where the garbage collector can no longer keep up (and it will happen), then every line of code in your system is now a potential failure point that can leave the whole program in a bad state. Lisp has this problem. Java has this problem. Erlang does not.

CouchDB databases grow with every document update, even if the update is a document deletion. So file compaction needs to be run occasionally to recover wasted disk space.

The compaction process will purge all old revisions and pack together the documents on disk to make sequential document access faster, like during view rebuilds and replication. Compaction is copy style and happens live or hot, while the database server is actively running. Normal database operations, reads, view index refreshes, document updates, replication, etc can all happen while the database is actively being compacted.

Also it is incremental and restartable, so if the server is shut down or there is a power failure in the middle of compaction, the next time you restart the compaction it will start back near the last spot where it left off.

So that's now checked in with a unit test working, though like most of the code it needs more testing, etc.

And here are still some other enhancements I'd like to see to storage compaction. One is compaction queuing, to make sure only one database at a time is compacting since it's a very disk IO heavy operation. That's fairly easy to implement.

Another enhancement is dealing with long transactions better that overlap the compaction file transition. Currently when a compaction completes, any read or write that started before the compaction completed will have at least 5 seconds to finish before it will be forcibly terminated with an error. That can be fixed to allow any unlimited amounts of time for transactions to complete, and to do so actually ties into some larger changes I'd like to see in the code. But until then clients can just retry the operation and things will be fine, so for now these are low priority things and I'm ok with it if they don't get done before 1.0.

Right now view indexes still do not compact, but that will be fixed later. For now as a stopgap, just delete the index files and the views will rebuild from scratch and hence "compacted". But views definitely will have proper compaction before 1.0.

Right now we are working on getting the Mochiweb branch finished and integrated back into trunk (Mochiweb is a replacement library for the current Inets HTTP library). Christopher Lenz has been doing most of the work, and I'm now going to help out finishing it up and hopefully get it checked in this week.

Once that's done plus a few more small tweaks, we might consider CouchDB 0.8 done. Then the release after that I think we'll target the incremental reduce and security, and then CouchDB will be in beta.

Lazyweb help: Anyone know how to "fix" this? (fyi I don't care if it's working as designed, the design sucks. It's slows me down and the effect is to train me to quickly enter my password to anything on OS X that asks.) If I have to I'm willing to hack things to get this annoying behavior fixed.

Of course if this is actually a bug, then is there a version where it doesn't happen? I'm running Notes 7.0.3 on OS X 10.5.2

And now they also have a version you can buy and deploy to use for your own projects. It's built on Lotus Domino, but maybe I can convince them to port over to CouchDB :)

I guess as penance to the April Fools haters, my wife and I just found out in two months we have to move out of our rental house, the owner's are moving back in. Dammit.

So what this does is accelerates our plans, which was to move to the Raleigh NC area or Silicon Valley or some other tech friendly place after the summer. We are in Charlotte NC, mostly for family reasons. We were waiting until the kids were a little older before moving, we have a 7 month old and Laura's mom is a huge help with the kids.

Man I hate moving.

Anyway, suggestions and input about moving welcome.

Now while I say I am somewhat sad, but that doesn't mean I oppose this, not at all. I like Erlang, it's approach to software concurrency and reliability is brilliant. It's found a programming model that just works and built it's whole environment optimized around supporting that model.

But the truth is you can integrate most of that into existing environments if you design your software properly. Programming is programming, whatever your language can compute, mine can compute too. This naturally means the details of a language aren't nearly as relevant as the community and number of libraries available.

As Christopher Lenz points out here, Apache is dominated by Java projects. Having CouchDB be an Erlang based project at Apache just doesn't make sense. The more things at Apache that are Java, the better they can interoperate.

Noah Slater, another project contributor, started to fork the project, partially out of dissatisfaction with Erlang.

At IBM, all the internal groups I've been talking with all are using Java for core development. The Erlang codebase of CouchDB is a constant obstacle to reusing their code, and vice-versa.

One major problem with Erlang is a general lack of libraries. Erlang lacks OO programming support, where the main focus is code reuse. Without OO you can't build software like Lego bricks, mixing and matching and swapping out the parts freely. This is a big reason why there aren't as many Erlang libraries as Java libraries, without OO it's just harder to make software reusable.

But perhaps the biggest problem with Erlang is the relative lack of Erlang developers. Developers in the US who've worked on a successful Erlang project probably number less than 10. But the number of developers who've worked on a successful Java project are easily 100X that amount. With a much wider talent pool to work from the quality of the contributions should increase dramatically.

So I've seen the writing on the wall. Erlang was great for a single developer working alone, but it's more important that CouchDB be able to take contributions from more developers with more backgrounds. Database internals aren't rocket science, it's simply a bunch of algorithmic transformations and actions combined to store and report on data with certain reliability and performance guarantees. On modern PC hardware, this stuff is a breeze. And now with Java it will get even easier for people to contribute to it.

I've already started rewriting the core in Java. Unfortunately finding libraries to support the indexing and storage features I need don't exist or have serious design shortcomings, so I'm writing most of that from scratch again. But the code completion in Java is so damn nice it really makes it easy to produce gobs of code quickly. With the amount of code I'm writing right now this thing will be done in a few months tops.

The upshot is the interaction between storage and view engine is getting simpler (and the code smaller) and will allow each to be coded and developed independently. The simplification will also allow for different query and indexing engines to be integrated into CouchDB.

Once compaction is done then it's on to incremental reduce, which is something I'm really excited about. By storing the intermediate reductions directly in the BTree inner nodes, the index update and query costs are Olog(N), allowing real-time map/reduce queries of huge databases.

And with this structure it is possible to do a "range reduce", which allows not only to get reduce results of a single key, but also a range of keys. So for example, if you created a map/reduce view of all purchases keyed by date, you can then query for the sum, average, min and max spent for all purchases between any 2 dates (the range), and CouchDB will calculate them nearly instantly. Not a reduced value for each unique key in the range (that's possible too), but a single reduced value of all the keys in the range.

BTW, I upgraded this blogs Movable Type installation from 3.x to 4.1, mostly to fix a bug that's been irritating the hell out of me, and to get better spam filtering. But now it seems I borked my installation but good. Since the upgrade the spam filter stopped working completely, letting in a new spam message every few minutes. So I turned off all comments to stop the flow. And now I can't get them to turn back on. It says they are on. But they aren't on. Doh.

Update: I've figured out why Movable Type comments weren't working, I didn't specify anonymous comments (or any other kind) are allowed. Fixed. Now if I can just get the comment spam filter working to slow down the spam deluge I'll be happy.

http://twitter.com/damienkatz