CouchDB Progress

For the past few weeks I've been working on the guts of the next release of CouchDB. Right now I'm nearly done with a big refactoring of the core code, completely separating out the view engine from the storage engine, so the storage engine doesn't know at all about the view engine. Partially this is necessary for live compaction (reclaiming wasted disk space while the database is running), and partially it's just to make the code more manageable.

The upshot is the interaction between storage and view engine is getting simpler (and the code smaller) and will allow each to be coded and developed independently. The simplification will also allow for different query and indexing engines to be integrated into CouchDB.

Once compaction is done then it's on to incremental reduce, which is something I'm really excited about. By storing the intermediate reductions directly in the BTree inner nodes, the index update and query costs are Olog(N), allowing real-time map/reduce queries of huge databases.

And with this structure it is possible to do a "range reduce", which allows not only to get reduce results of a single key, but also a range of keys. So for example, if you created a map/reduce view of all purchases keyed by date, you can then query for the sum, average, min and max spent for all purchases between any 2 dates (the range), and CouchDB will calculate them nearly instantly. Not a reduced value for each unique key in the range (that's possible too), but a single reduced value of all the keys in the range.

BTW, I upgraded this blogs Movable Type installation from 3.x to 4.1, mostly to fix a bug that's been irritating the hell out of me, and to get better spam filtering. But now it seems I borked my installation but good. Since the upgrade the spam filter stopped working completely, letting in a new spam message every few minutes. So I turned off all comments to stop the flow. And now I can't get them to turn back on. It says they are on. But they aren't on. Doh.

Update: I've figured out why Movable Type comments weren't working, I didn't specify anonymous comments (or any other kind) are allowed. Fixed. Now if I can just get the comment spam filter working to slow down the spam deluge I'll be happy.

Posted March 23, 2008 9:54 AM