Typical Open Source Software Moment

The thing open source advocates always tout about open source software is that you can open the hood and poke around and change things if necessary. The ability to customize your app and the source code level is nothing short of Real Ultimate Power. (My friend Mark said that he saw a OSS guy totally uppercut some kid just because the kid opened Windows)

So last night, I put that to the test. I wanted to find out some details about how Erlang handles some low level file operations. My new Erlang based CouchDb engine, which is now storing and retrieving documents, seems a little too fast right out of the gate, and I suspect it's because the flush operations ('sync' in Erlang) I'm performing aren't actually flushing to disk. Which is possible because the Erlang documentation says the 'sync' operation doesn't work on all platforms. But it doesn't say which platforms. Swell.

But wait! It's open source, I'm a coder! I can just look at the code and figure it out!

And yet, after many hours of searching the Erlang source code, I still haven't figured out how Erlang translates file system operations down to OS level calls. I've scanned many thousands of lines of source code, I've searched for file function calls in every file API I can think of. And still I can't figure out how it's doing it. I've looked at lots of interesting networking code, and I now have a better understanding of how the Erlang source is organized. But by just looking at the source code, I haven't been able to answer my rather simple question: Do Erlang disk flushes actually work on Windows?

This isn't to say it's impossible for me figure out, but it going to require me to understand the core underpinnings of Erlang's bootstrapping process. Essentially, to answer a very simple question about the Erlang source code, I now have to spend who knows how many hours becoming familiar with the guts of Erlang's runtime. And while that's kind of fun, it isn't helping me get Couch built. It's a big distraction.

For now I'm just going to write it down as a possible outstanding issue. Often answers to questions like this pop up at unexpected times, so its best just to move on and wait for it. If I never find the answer then I'll pester someone who's familiar with Erlang source. If the answer is that I must modify the source to get the behavior I need, then I'll cross that bridge when I get to it.

Having the source code isn't quite the benefit open source advocates would have you believe. It isn't like having detailed instructions for a model battleship, it's more like all the parts were dumped out onto the table. Good luck with all that.

Posted February 28, 2006 5:21 PM