June 29, 2006

Nihilist Job Resume

McSweeney's Internet Tendency: Nihilist Job Resume

Link

June 20, 2006

Coin in the Couch

So I've been thinking lately about how I can someday make a living with CouchDb, and one option is to try to build a real software company behind this thing. I'm not at all sure this is the path I want to take, but I'm also thinking about taking some external funding. That of course means I'll need some sort of a business plan, or at least, a revenue model.

Original Business Plan:
1. Build CouchDb.
2. Annouce to world "Look what I did".
3. People send checks for my total awesomeness.

I've been informed many people embark on a similar plan and it hardly ever works. So instead this is what I've come up with:

Give Away the Core
The CouchDb application server with simplified web based IDE. The idea is to make it extremely easy to build form and document based applications, with little development and zero database knowledge necessary.

This part will be open source, probably GPL or some compatible variant. Free to download, free to use, free to host.

Charge for Add-ons
Offline application client host. This is essentially a locally installed Couch application server that integrate with browsers. It allows users to bi-directionally replicate any Couch applications and data for offline use.

Advance server services. Server clustering and/or partioning can be enterprise grade add-ons. Management tools tend to also be a revenue maker.

The Hook
The idea is to let people freely install CouchDb server and create and share custom applications. CouchDb development will be very simple with little coding or planning required. The database is specifically designed for document oriented applications, making document application creation and customization simple enough for average office workers.

Once custom applications are built and put into use, people will have increasing needs for add-ons, like the ability take it offline. Sometimes a user will only be somewhere for a few days where they won't have reliable access, but that alone is often enough to justify the cost of a software license.

And then I got you. Mwahahaha!

Ok, this is just the roughest of business outlines. But I'd like to know what you think. Is this at all a good approach? What will I be in for I decide to look for funding? Should I take on a business partner first? Should I just not go the external funding route at all?

Link

June 15, 2006

Test Formulas

For those curious about Fabric, I just posted on the CouchDb Wiki the actual test formulas that get run and validated by the Fabric test suite. They give a pretty good idea of how language works in general, as it is everything that works right now, minus some stuff I haven't written tests for yet.

Link to the tests

Link

Most Excellent

I discovered on vowe.net that Thomas Gumz took the liberty of registering CouchDb domains to help with the project: couchdb.com, couchdb.net, couchdb.org.

I'm going to use these to set up a proper CouchDb project site before the first public release of CouchDb (within the next two months). For now I'm really digging the placeholder graphic. Thanks Thomas!

Link

Good Responses

I've gotten some really good responses to my previous post about Erlang.

Wesner Moise has interesting things to say. I didn't know file corruption was big issue in Office, particularly with incomplete writes. I just assumed that problem they solved long ago.

Also interesting is this comment left by Judah:

I think that's a great improvement, but what about getting fewer bugs in software? At work, the bugs in our software -- probably 99% of the time -- are coding errors. When our software fails, it's almost always due to programming error. In the Erlang way, it would just retry my component that failed, then the parent component, and so on until the whole thing restarts. That's a bad system for us! With 99% of the time being programming error, restarting components isn't going to help in the least. The places where non-programming errors occur (such as network timeouts), well, a simply try/catch/error message is simple enough. We've designed our software to retry in such scenarios as well before issuing any errors. And even the error messages provide an easy way for users to retry such operations.

This is almost exactly my thinking too as I began to read about Erlang. The things they were talking about seemed to be solving problems I didn't have. I need solutions to MY problems, and here they're talking about imaginary restartable components!

But once you really get in to it, you realize the restartable components are just one part of a system. And actually, the restartable-component stuff not really in the basic Erlang language design, it's just a capability that "falls out" of the language design and execution model.

Because Erlang is different in several subtle and not-so-subtle ways from "regular" languages, it's really hard to appreciate it all without being immersed in it. Erlang will change the way you think about and structure your code. It should anyway, because it's different. And it's different in ways you probably won't expect.

As an example, a huge thing that's gone away is the need for complex boolean expressions. You know:

((foo == 3 && bar == fah) || !(foo == 2 && (baz == 1 || !snoz)))

I've hardly had to use any logical ORs or ANDs, it's mostly handled by pattern matching, which just seems to do a better job. I didn't expect that, and I'm not completely sure why it is.

And for some reason, the if expression is broken in Erlang. Don't ask me how it's possible to screw up ifs but Erlang does it in a way that almost makes it useful, but not quite. But it doesn't even matter, because case statements with pattern matching is so natural and powerful you'll barely miss ifs (but you will miss them occasionally).

And it's the same pattern matching that makes it easy for a process to extract the right message out of its message queue, greatly simplifying messaging code. It also makes it easy to detect invariant states and error conditions, meaning problems are identified early before they can propagate and cause further damage.

The point I'm trying to make is there is so much that works differently in Erlang it's hard to appreciate how any one part of it could help you in your situation. You have to try it out, see how the pieces fit, wrap you brain around it, then you'll see how it can be applied to your problems. And how you're problems will change. Some things you can't read about to understand, you have to experience them.

Also, contrary to what I said in my last post, I actually am of the opinion Erlang generally makes for fewer actual bugs, but my own coding efforts haven't been sufficiently tested for me to really judge. But so far it feels solid. We'll see.

Another great comment left by Jeremy Bowers:

The natural temptation is to skirt around the problem, try to avoid the domain, and just sort of hack around and get by. Hence, the dominant threading model of today with shared memory being the primary paradigm, and thread-level memory being the exception, and all the fun locking and race conditions and such that ensue. And you feel like you've accomplished something.

It's funny because it's true. Almost any time in an application you add some concurrency it really does feel like a big accomplishment.

But sometimes, if you gird up your loins, bravely attack the problem head-on, take no prisoners, no holds barred, and let the problem domain guide you to another locally-maximal solution, you can end up in a local maximum much higher than the first one.

I've been playing with Erlang myself lately, and I get the feeling that that is where Erlang is; by embracing multithreadedness instead of cowering from it, and following it where it goes instead of trying to drive it, it ends up with an exceptionally powerful approach to software. I regret to say I don't think Erlang is going to ride its way up to the top in its present form (short form: too foreign for most people to cotton too due to its functional heritage, but I think that's the programmer's problem, not Erlang's), but I am fairly certain that in 10 years, something very like Erlang is going to be one of the dominant languages of the day because of how we'll all be running 128-core machines on our laptops and how naturally Erlang extends into that environment. (In fact I'm almost not certain how this statement could be wrong...)

Yup yup yup. I also think Erlang has just enough weirdness that it won't ever become "mainstream", but will influence a lot of languages in the years to come (the Frank Zappa of programming languages).

Link

June 13, 2006

One reason I chose Erlang

Meanwhile, back at Ericsson, some Erlang-based products that were already in progress when the "ban" went into effect came to market, including the AXD 301, an ATM switch with 99.9999999 percent reliability (9 nines, or 31 ms. downtime a year!), which has captured 11% of the world market. The AXD 301 system includes 1.7 million lines of Erlang: This isn't just some academic language.

Erlang in BYTE.com

I really should write more about Erlang (as I promised), but I've been busy actually writing CouchDb in it. Anyway, the above fact about the extreme reliability of a 1.7 LOC switch was one big reason I chose Erlang for CouchDb. I did it sort of on faith, not sure why I should use this language, only that it claimed to address problems that I couldn't figure out how to solve with regular software tools. And it seemed to walk the walk with real shipping products.

So I had high hopes about Erlang, and I must say I haven't been disappointed. Honestly, I was expecting something different. I'm not sure what I was expecting, but what I found surprised me big time, the Erlang OTP platform has completely changed the way I view reliable software construction. One thing I've found surprising was it's just as easy to create bugs in Erlang code as it is in other languages, it does not have a silver bullet to eliminate software defects. But its reliability doesn't come from being so much better about creating bug free code, but how well it tolerates bugs and faults.

In properly designed Erlang software, when a component has a problem, -- it hit an unexpected condition, runs out of memory, times-out, etc -- the component is restarted. If the problem continues, then the parent can be restarted. If necessary, the system can shut down the faulty component permanently, or continue restarting components until the entire application is restarted, essentially a fresh restart.

Basically it has built into it a notion that shit will happen, so let's not allow small problems to wreck our whole application. It goes a long long way towards addressing otherwise intractable problems like memory leaks, memory fragmentation, deadlocks and infinite recursion. Kill first, ask questions later might as well be the Erlang motto. And why not? You just respawn what you kill. And while it may sound hard to write software this way, it's actually far easier than you think, because there is very little error handling code cluttering up everything.

This incremental restart strategy is not possible in languages with mutable data and shared state threading models (such as Java, C# and just about every other mainstream programming language).

Link

June 12, 2006

A Thank You

Jake thanks someone important.

Link

June 8, 2006

Hear me talk about CouchDb

Earlier tonight I was interviewed about CouchDb by Bruce Elgort and Julian Robichaux on their popular Lotus oriented The Taking Notes Podcast show. He just now put it up so I haven't listened to it yet, but I'm betting I'll be cringing and wincing when I listen to myself speak. Wince, wince, wince....cringe...wince, wince. You know how it goes.

Anyway, I talk a lot about CouchDb and its goals, and also how it compares with Notes. Plus I think I had a rant about how Exchange sucks, and how things would be better if I were running things and you're not the boss of me. Should be good!

Link to the Podcast

Update
I just heard it and I think it turned out quite good. Have a listen!

Link

June 5, 2006

It lives

alive.jpg

Today I finally got the complete CouchDb server working.

There is still much work to do before a public beta (biggest items: Fabric isn't hooked in, REST API, UTF8 collation support), but the hardest stuff is all there and working.

I had already gotten the database code working for a while now, but last week when I was trying to complete the entire concurrent server architecture I realized I had made a bad design decision earlier. I was really stressed for a few days as I puzzled over my design and the flaw, worried I was going to have to make serious structural changes to the code. After much tortured thinking, I finally wrapped my head around it all and found the solution, which turned out not to be such a big change after all.

So today I created some basic tests to put the database server through its paces, running a lots of test client processes concurrently. Each test client reads and writes documents and creates computed tables.

The first time I ran the tests I saw tons of error messages about update conflicts. Odd, that error means there already existed in the database a document with that same id. Since each document was new, maybe there was some sort of corruption bug?

It didn't take much debugging before I realized they were indeed real update conflicts. It turns out the random number generator in Erlang is seeded per process, meaning the "random" UUIDs generated by each test process were getting randomly generated exactly the same way each time.

So the first process was succeeding with its updates and all the rest, trying to create the same documents, were getting update conflicts. And the database correctly identified them as such (even though I hadn't actually tested out that code yet, cool).

So I fixed the UUID generator and reran the tests. Then it all just worked.

It's alive.

Link