What is Couch?

Ok, I'm going to briefly describe my vision of what Couch will be. I've been a little shy about discussing it as of late, mostly because I have changed it drastically in the past and I wanted to wait until felt things had solidifed more. And so now that I feel pretty good about its direction and the likelyhood for success, I'd like to say more about what I'm building.

What is Couch? Concise version:

Couch is Lotus Notes built from the ground up for the Web.

What is Couch? Long version:

Couch is a simplified database application platform that is document centric and non-relational. It is a distributed database system, with bi-directional replication built in, enabling high scalability, failover and offline access.

Couch Components


CouchDb:

The core database storage and reporting facilities. CouchDb is not a relational database, but rather a schema-free document database.

Documents are objects that have any number of named field values. Since there is no fixed schema, documents can have any number of fields and those fields can have any arbitrary name and value. Unlike a relational database, where records are inserted and deleted out of named tables with a fixed schema, CouchDb has a flat object space and objects do not follow a defined schema.

CouchDb is a bi-directionally replicatable database, with the ability to incrementally replicate changes and resolve conflicts between two replica instances of the same database.

Reporting on the database is accomplished with "Computed Tables" that use Fabric (the CouchDb query language) to select documents and generate column values. Computed tables are very similar to Lotus Notes views. Each document that is selected by Fabric corresponds to a row in the table, and the column values are the result of Fabric expressions that reference the document fields. The same document can appear in many computed tables, or no computed tables, depending on the computed table Fabric selections.

Fabric:
This is the CouchDb query, processing and validation language. It's closely related to Notes Formula Language. It's simple list processing language, designed to be easy to learn and use, and tightly integrated into CouchDb and Couch Web Application Server.

Web Application Server:
This is a piece that will integrate into a web server (most likely Apache) and serve Couch applications and database content out of CouchDb. This layer will interface tightly with one or more popular scripting languages (PHP, Python, Ruby, etc). Design elements (forms, views, tables, etc) are simply documents in the database that store the client side HTML and server side markup and application scripts. These design documents are evaluated and translated into HTML by the Couch Web Application Server and its components.

The web application server also provides a REST based XML replication facility for clients and peer web servers to bidirectionally replicate CouchDb database.

Offline Personal Web Application Server:
This client installable version of the web application server that can replicate server based Couch applications for your local web browser. This provides the disconnected user with access to Couch applications.

Couch Web IDE:
A web browser based IDE hosted by Couch Web Application Server that allows end users to create simple Couch applications (think Lotus Notes V3). With DHTML and Ajax, it will provide a simplified and intuitive UI for creating forms, views and simple scripts (in other words: build Couch applications). People who can create Word documents and Excel spreadsheets should be productive in this environment very quickly.


So that's the grand vision for Couch. Yes, there is much work to be done. And no, I don't plan on doing all of it alone.

So what sorts of applications will Couch be good for? Email, bug databases, time sheets, RSS feeds, CRM, content management, blogs, forums.... Basically anything document-ish that you might also want to access while you're offline.

And developing those sorts of Couch applications will make Ruby on Rails look clumsy and heavyweight by comparison (that's right ROR, I'm talking smack). Plus it can do something no other mainstream web development platform can do, replicate the whole thing offline. Notes people know it's not a pipe dream.

I'm going to need a lot of help and feedback from the Notes community to make this project successful. If you're a Notes geek, I'd love to hear your questions and thoughts about Couch. That goes for anybody actually, but double for the Notes crowd. ;)

Posted December 21, 2005 8:58 PM

Comments

Sounds too good to be true...
Keep coding man, I can't wait!
- T

Thomas Gumz, December 21, 2005 11:49 PM

That's very well written and very suscinct. It's good that you end with practical sample applications that Couch would likely support well. Other than that, even though Notes wasn't built for the web from the ground up, it was extended to the web (and before a widely adopted open standard like php came), so is the Couch paradigm going to be that much better?

David Boudreau, December 22, 2005 1:44 AM

Sounds cool, but isn't this very much like Zope? Maybe Zope is a good starting point, lets add offline and a good IDE (I think Zope supports WebDAV that might be useble as well) to it and we're good to go.

Gert Jan van Halem, December 22, 2005 7:39 AM

Hi Damien

It sounds great!

But i'm a little surprise that your project has turned into a scaled down version (or a reinvention) of Domino. Besides the great challenge for you ..what does Couch bring ..that is not Domino in a simular way?

brgds
Jesper Kiaer
http://www.jezzper.com

PS Don't get wrong ..I am very impressed of what you are doing! :-) (again)

Jesper Kiaer, December 22, 2005 8:17 AM

This look really cool Damien!

I'll be very interested in seeing where this goes! Please keep us apprised of it's status and maybe even give us some screenshots and demos and such so we can contribute!

Phil Randolph, December 22, 2005 8:59 AM

Do you have a financial backer? How do you plan to sel this (great) idea to developers who have open source products that do a lot of what this does? (albiet not as good...) Can you describe the UI? Will you be using an existing platform? (Eclipse, NetBeans, XULRunner?)

jake, December 22, 2005 9:03 AM

As much as I love python, please don't use Zope. Zope 3 is a huge improvement but zodb likje most odbms is tightly coupled to python.

I second xulrunner as the local platform and would love an xpcom binding for couch. This would expose couch to javascript in mozilla. Since mozilla 1.9 will support python for xull apps (not web apps), I'll settle for python bindings.

One thing I miss in Notes that was in several flat databases bofre Notes is a previous row reference in view columns. A parent ref would be nice too.

Dan Sickles, December 22, 2005 10:30 AM

David - How will this be better than Notes/Domino? Well, it won't be burdened with 20 years of cruft that comes from being designed as a fat client. I'm trying to throw out the old Notes stuff that doesn't make sense, and keep the things makes Notes and Domino powerful at its core.

Gert - I've looked at Zope in the past, and it misses the mark for web development IMO. Zope uses a OODB, meaning its a DB is meant to provide a seamless persistance layer for Python. CouchDb is different, its meant to be a document database with robust reporting capabilities. I think this is a better match for the web.

Jesper - Yes, its very similar to in concept to Domino. I'd consider it to be Domino done right.

Jake - No financial backers and I don't have any business plans for it. Yet. But I do know one thing people will pay for: Offline. I can probably give everything else away and just charge for the offline component. But like I said, I have no plans yet. I just want to build this thing.

Damien, December 22, 2005 10:33 AM

IMHO, If you build a db repository that one can embed that handles online & offline access smoothly, that would be marketable. Sorta like the MySQL of replicated datastores... Charge for the ancillary services. Everything else would be gravy :-)

jake, December 22, 2005 10:47 AM

Damien, sounds like the lightweight nature of this and the focus on offline capability makes it a perfect candidate for mobile porting... given that CRM is one of the areas you're foreseeing this being useful for, remote and offline access via PDA might be something to specifically account for in the framework design; this is an area that Domino often gets slammed on, since this type of functionality wasn't even on the horizon until long after Domino had already become bulky.

Tim Tripcony, December 22, 2005 10:51 AM

I like the idea of building your IDE as a web app hosted your own app server. I love self-referential systems like that (C compilers written in C, etc.).

I think it also makes a lot of sense to have one UI experience. Using a browser even for off-line use might seem counter-intuitive, but the two client schism is still a problem for Notes and Domino. Writing an application for both Notes and the Web can be a real headache.

The browser has a lot of shortcomings, but now that Ajax "best practices" have matured, you can probably get a lot of mileage out of the browser for both your IDE and the end-user experience. I am a big fan of rich clients, but in this case, you are probably right to introduce a rich client as late as possible. When you do, you should consider Eclipse RCP. It's a great foundation for an IDE.

Dave Delay, December 22, 2005 10:54 AM

Nice job Damien. I think the idea of a browser based IDE is interesting. I recently was looking at a project called Wheat that has a great tag line:

Every object has a URI, the virtual machine is a web server, and the development environment is a Wiki!

The idea of a collaborative development environment that could be used either online or offline using a Couch based Wiki sounds like a cool idea to me.

Pete Lyons, December 22, 2005 1:29 PM

wow! i've been following this blog for a while, since you were talking about Couch as some sort of network / server infrastructure that sounded kinda like Google's back end.

then i noticed you talking about some kind of formula language.

then, pow! very interesting. i'd love to beta test when you get around to that.

i already have an application prototyped in Notes that i have been thinking about porting to another platform, but it needs offline capabilities desperately. i was even thinking about selling it (i've been back and forth whether to open source it).

when you get far enough along let me know, i'd be more than happy to try to develop the same app for Couch. no better way to test your dev tools, i think, than to have ppl start to develop for it. ping me, i'll fill you in if you are interested.

jonvon, December 22, 2005 4:33 PM

You mean to tell me that I could build my applications in Couch and not spend the majority of my time undoing/hacking around what it's trying to do *for* me so it can tell my v6.5 customers that the application would still be compliant with v4.6?! ;-)

First time/long time, and I (and I'm certain that I echo the majority here) can't wait to get crackin'!

Chris Toohey, December 22, 2005 9:08 PM

It is a little bit funny that your search for what Couch should be ...ended back at something Domino like :-)

I guess there is an attraction..a beauty in what Notes/Domino is. Not the actual implementation but the idea that everything is built around a note/document etc.

I have been looking for a simple Java web framework but no one seems to get near the ideas behind Domino.

I like the Notes/Domino concept so much that I actually started building a Java web framework myself built around some of the ideas from it. I want it to be really really simple so I actually chose the name SimpleD for it :-)

If it ever gets finished ..that is another story.
(kids, work..you know..)

brgds
Jesper Kiaer

http://www.jezzper.com

Jesper Kiaer, December 23, 2005 4:05 AM

Can we presume that part of the cruft you're throwing out is the summary buffer limit?

Richard Schwartz, December 23, 2005 7:00 AM

Tim - A lot of the architecture and design decisions I'm making are with portability to other platforms in mind. And lightweight devices are definitely included.

Dave - Initially I want a very simplified, end user centric IDE. This way I don't compete with mature, powerful IDEs target at professional software developers (that's a very crowded market). Instead I'm competing with IDEs that let end users (office manager, secretaries, etc) build simple, replicatable Web apps, and I think the browsers can meet that requirement. And yes, I hope to have it self hosted, where the IDE is just another Couch app. That would be cool!

Richard - CouchDb has no limit on per document summary data.

Damien, December 23, 2005 10:18 AM

"I'd consider it to be Domino done right."

Man, you must not have gone out of the house since you started working on this...

..otherwise, your ego would've never got back through the doorway....

Shame on you

kudla, December 23, 2005 8:24 PM

Yes, my ego is big. But maybe not as big as you think.

Firstly, I'm not talking about Notes & Domino as a whole, there is no way in hell I can rebuild that. Actually there is no way in hell I can even build all this. I'm going to reuse lots of infrastructure and I'm going to need lots of development help.

When I say "Domino done right", I am specifically talking about Domino HTTP and ease of building Domino Web applications. The whole thing, while brilliantly executed, was flawed in its quest to always translate Notes applications for the Web, rather than focusing on being a first class web application platform on its own.

And being grafted onto an already mature code base and carrying forward functionality intended for a propriatary fat client, it carried along a lot of baggage from decisions made long ago. I've developed many Notes applications for the web. This much I know: It doesn't have to be this difficult.

I think if IBM would have placed more priority on making it a first class web development platform separate from Notes, it would be a major web server player today. But for some reason that didn't happen.

So I'm building Couch because I believe the model that Notes pioneered is an unappreciated one, one that hasn't been explored enough. I'm building what I know works (i.e. I'm not very imaginative). And yes, I think I can improve on an already great design. I'm egotistical that way.

Damien, December 24, 2005 1:15 AM

Cool idea.
Some thoughts:
- Are you planning to have a import/export/sync facility to POD (plain old databases)?
- what's about a @formula converter?
- How can we help?

:-) stw

Stephan H. Wissel, December 24, 2005 2:40 AM

I love your emphasis on end-user authorable design. I've always thought Lotus's R4 move away from this (not by the introduction of their designer client for advanced stuff, but by the removal of simple design from the end user license) betrayed a fundamental lack of understanding of why Notes kicked ass.

The days of unlocked R3 environments before admins got "smart" and stopped allowing anyone to put anything on the server was a magical time. If you can't prevent bad programming (e.g. @Today in selection formulas) from degrading the server, you'll only encourage IT depts to be lawnmowers of any grassroots support you'd hope to enjoy.

John Smart, December 25, 2005 2:33 PM

Can't argue with Notes' great success, and the web's, but putting them into the thin client vs. bloated client model, and the goals of Couch, it makes me wonder whatever happened to Lotus eSuite? Maybe not exactly the same but similar high level goals, would you say? Back around 1998 I thought having a spreadsheet on the web for same intended users as yours would have been big, but fizzled out and was discontinued. But back then, "thin client" meant "network computer"- not necessarily mobile phone, or pda. No one actually wound up buying any network computers- the market chose to stick with fat clients instead. I would question your viewing Notes/Domino as too fat- something that might be irrelevant, as far as the market's concerned (not that any of us know the Couch market yet, granted- might be a ton of people out there waiting for this or some highly anticipated app that Couch does the best way). Notes is cool, as you said, by not being relational yet still valuable at the same time; but if you're going to reinvent that wheel, unappreciated as it is, there must be some killer apps that take advantage of Couch's thinness that Domino isn't agile enough to pull off, and where people actually demand thinness in that case.

David Boudreau, December 25, 2005 11:44 PM

We are all conditioned by our past, so are all compelled to compare what you are building with what we have known and believed in as being "the" solution to putting applications on the web. Ray Ozzie reinvented Notes with Groove (and took groupware back to being a small group application but interestingly broke us out of Internet=browser mindset). Andrew Ressler reinvented Notes brilliantly with Radnet WebShare (smartest person I have ever met). It was sad that Radnet didn't survive. (My suggestion that WebShare should have an NSF interface didn't go over so well, but that probably wouldn't have mattered anyway.) And now Couch. I think all three approaches honor what drew us all to Notes in the first place, the ability to quickly build applications that people can use from anywhere. The mechanics of how that is done is what distinguishes the four efforts.

Kent Kurchak, December 26, 2005 10:52 AM

Stephan:

I have no plans yet for an importer/exporter/sync thingy other than to export and import databases and documents as XML. But I like the way you think, and I definitely see such things being built, but there are too many unknowns at this point to know what such a thing would look like.

An @Formula converter is a terrific idea, but like the import/export facility its a bit premature right now.

I've been getting lots of offers for help. It's fantastic! However, right now the help I most need is input and feedback. I've already gotten some good feedback about Fabric that is prompting me to make changes in the syntax. Soon enough though, I'll be ready for all kinds of help: development, testing, documentation, setting a project website Wiki, etc. Lots to do!

John:
I think you "get it". I think many Notes professionals, particularly those around in the R3 days, understand something that IBM doesn't seem to get: Notes got popular because it was so damn easy.

I started off my career doing Notes consulting. Almost every company I did work for had numerous Notes applications that were developed by secretaries, office managers and sales people. And some of the applications became core parts of the companies' businesses. And they kept getting modified and tweaked as the business needs changed.

This is how Notes infiltrated so many companies: people solving their own problems and building just what they need, when they need it. Truly software that evolves with you. And once it's in place, the custom applications seem to mulitply, and then it's nearly impossible to get rid of Notes. :)

But the funny thing is, I heard very little about this phenomenon at Iris/IBM. These "amateurs" building little in-house applications to run their companies, they pretty much weren't on the radar screen at all. The people who were working on the Designer client certainly were aware, but on the Client team (which the Designer was part of) *everything* seemed to revolve around mail, and the Designer effort was seriously underfunded and developer and QA resources were constantly pulled off to work on other things (usually mail).

But, hey IBM managed to sell 200 trillion licenses anyway, so maybe they made the right decision. But they still dropped the ball on the end user development aspects, and there is still lots of unexploited opportunity there.

David & Kent:
Your thoughts mirror many of my own.

Couch won't replace Notes, and therefore also won't replace Domino. It also won't replace other web development tools, relational or otherwise. But that's ok, for Couch to be successful, its not necessary for others to lose.

Damien, December 26, 2005 3:04 PM

T-shirt idea:
"Couch sayeth: Schema shmeema!"

err, Sorry. Made _me_ giggle, anyway. ;-)

John Smart, December 27, 2005 11:35 AM

Damien, thanks for the clarification.
I, of course, understood what you meant and the context in which you placed the quote.
I was just...um...well.. just me being me.

The only thing that still rankles about the whole Domino/Notes thing is that somehow Domino got branded as the application development platform.

Notes was always and ever about application development. IT WAS NEVER ABOUT SOME FAT--SS CLIENT THAT YOU COULDN"T TAKE HOME TO HAVE YOUR IT DEVELOPMENT MANAGER MEET AND EVEN NEVER MORE WAS IT ABOUT E-MAIL!!!

Ok, sorry. Time to take my meds.

I miss you man. Best of luck with your Couch stuff...

P.S. Have you developed a jump shot yet?

kudla, December 29, 2005 5:04 PM

Damien,
put me on the helpers list.
Have you ever come across Thingamy? My gut feeling says Fabric, Couch and Thingamy would be a terrific strong brew.
:-) stw

Stephan H. Wissel, December 31, 2005 10:30 AM

Post a comment




Remember Me?

(you may use HTML tags for style)