Typical Open Source Software Moment
The thing open source advocates always tout about open source software is that you can open the hood and poke around and change things if necessary. The ability to customize your app and the source code level is nothing short of Real Ultimate Power. (My friend Mark said that he saw a OSS guy totally uppercut some kid just because the kid opened Windows)
So last night, I put that to the test. I wanted to find out some details about how Erlang handles some low level file operations. My new Erlang based CouchDb engine, which is now storing and retrieving documents, seems a little too fast right out of the gate, and I suspect it's because the flush operations ('sync' in Erlang) I'm performing aren't actually flushing to disk. Which is possible because the Erlang documentation says the 'sync' operation doesn't work on all platforms. But it doesn't say which platforms. Swell.
But wait! It's open source, I'm a coder! I can just look at the code and figure it out!
And yet, after many hours of searching the Erlang source code, I still haven't figured out how Erlang translates file system operations down to OS level calls. I've scanned many thousands of lines of source code, I've searched for file function calls in every file API I can think of. And still I can't figure out how it's doing it. I've looked at lots of interesting networking code, and I now have a better understanding of how the Erlang source is organized. But by just looking at the source code, I haven't been able to answer my rather simple question: Do Erlang disk flushes actually work on Windows?
This isn't to say it's impossible for me figure out, but it going to require me to understand the core underpinnings of Erlang's bootstrapping process. Essentially, to answer a very simple question about the Erlang source code, I now have to spend who knows how many hours becoming familiar with the guts of Erlang's runtime. And while that's kind of fun, it isn't helping me get Couch built. It's a big distraction.
For now I'm just going to write it down as a possible outstanding issue. Often answers to questions like this pop up at unexpected times, so its best just to move on and wait for it. If I never find the answer then I'll pester someone who's familiar with Erlang source. If the answer is that I must modify the source to get the behavior I need, then I'll cross that bridge when I get to it.
Having the source code isn't quite the benefit open source advocates would have you believe. It isn't like having detailed instructions for a model battleship, it's more like all the parts were dumped out onto the table. Good luck with all that.
Posted February 28, 2006 5:21 PM
Comments
Making a blanket statement like yours isn't really fair; obviously half of all OSS code is going to be below-average. :P
To match your anecdote with another, we just recently patched Python where I work to get around a locking problem. Having the source made possible both being sure of the problem in the first place, and patching it ourselves. It took us under half an hour.
Jonathan Ellis, February 28, 2006 6:00 PM
It looks you need a debugger.
Danny, February 28, 2006 6:05 PM
I'm not complaining about the Erlang source code, I've found it to be well written. It's just that things like compilers and language runtimes are complex and hard to understand by their nature. Having the source doesn't mean you'll have an easy time with it, and often trying to accomplish small things can be a big time sink but you won't know it until you've already wasted a bunch of time.
Mostly I'm just bitching.
Damien, February 28, 2006 6:08 PM
I've always thought that having the source was only half the problem -- you also need a debuggable build where you can set breakpoints and watch what the code is really doing.
Bob Balaban, February 28, 2006 8:08 PM
Open source, definately a love-hate relationship for a non-programmer.
big dogs, February 28, 2006 9:21 PM
Try running SysInternals disk monitor, FileMon.
http://www.sysinternals.com/Utilities/Filemon.html
It'll tell you if stuff is REALLY being written to disk.
RegMon is also good if you suspect an app is reading/writing stuff to the registry, but can't suss out how or where.
Jeff Atwood, March 1, 2006 1:47 AM
This is far too practical, informative and useful.
Post something stoopid already!
Ronnie, March 1, 2006 5:41 PM
FWIW erts/emulator/drivers/win32/win32_efile.c is where to look. sync() seem to be a no-op.
Life sounds hard in Windows if you don't have strace to show you what system calls a program is making.
Anonymous, March 3, 2006 4:46 AM
Thanks Anonymous. I guess I didn't have the full source pack (doh), but now I've found it online.
And the sync call does look like a no-op. Crap.
Damien, March 3, 2006 3:05 PM
If Erlang was closed source, even if you knew the answer to your question, you still wouldn't be able to fix it. So, it seems like open source is still a lot better, and you learned something in the process.
Dusty, March 6, 2006 9:01 AM
Have you tried asking your questions on the #erlang irc channel? I am not familiar with this particular channel but I've had very good experiences in the past with other channels.
Olivier Ansaldi, March 6, 2006 7:22 PM
Post a comment