April 30, 2005
Freedom to Speak
My friend Pete recently posted some negative comments about his former employer, IBM. They were simply his opinion of the working environment at IBM, but IBM legal told him to take it down. So he did, no doubt wanting to avoid a big hassle from IBM's legendary attack lawyers. But he has a right to have his opinion expressed, IBM doesn't have a right to whitewash from the world the negative opinions of its corporate culture, they don't have a right to squash dissent. When large corporations succeed in doing that, they erode everyones rights. Freedom of speech is perhaps our single most important right, without it, all other rights can be silently taken away.
IBM is a fantastically powerful corporation, who can sink their teeth into government and politicians and spends huge amounts on lawyers, bending the rules as much as possible in its own favor rather than compete on a level playing field. As you might imagine I hate stuff like that, and I like to do whatever small thing I can to fight such behaviors. So before I get too preachy and without any communication or permission from Pete, I've decided to repost 2 of Pete's removed articles here. Enjoy.
Apr 28 2005 - Closing thoughts on Workplace and IBM
That little pig drawing analysis post from the other day may be more on target than I knew. It said - quote:
I am realist.
I am direct, enjoy playing devil's advocate and neither fear nor avoid discussions.
I am emotional, naive and a risk-taker.
I am secure, stubborn, and stick to my ideals.
I am a good listener.
My recent comments about my last job have caused a bit of a stir. Before I move on and stop talking about the topic all together I have a few last words.
I still believe in the Workplace product vision. While I may have disdain for some of the technical leaders, the bulk of the staff are great people who work hard to perform in the mostly schizophrenic environment of the IBM software group. I also have nothing but good feelings toward the two levels of management with whom I dealt. Their jobs are among the worst I could imagine and they perform them professionally and diligently.
That said, as someone who took the ideals of IBM to heart: honesty, integrity and service to the customer, I found the Workplace environment unbearable. In my opinion, decision after decision has been made that favors personal agendas over either the goals of the organization or the needs of its customers. I never expected to agree with everything but when you agree with next to nothing I think there's a problem.
As the pig said: "I'm emotional, naive and a risk-taker". Deep down, I actually hoped that by being up front about the situation, maybe someone who could actually make a difference would finally hear about the mess and maybe some real change could come about. When I worked for IBM I was just a band 9 peon, but once outside, I at least had a free voice: Emotional - certainly, naive - probably, risk-taker - you be the judge.
While I've received much support from my old co-workers on the facts, many have also advised that I was perhaps a bit rash in saying such things in public. Time will tell what the consequences of my actions will be. As for the STSM's and DE's that I disparaged in my previous post, there was already a pretty clear understanding of my opinions regarding them before I left IBM. No new bridges were burnt there. But it did feel good to vent about it.
Back before I worked for IBM I used to wonder how a big company with all those resources could develop such crappy products like Topview or fail to capitalize on things like OS2. Working on Workplace has answered all those question in my mind. For all the talk about needing to execute and being responsive to customers I see a company drowning in a quagmire of process and ego. If I was Sam P, I would cut out half the management levels and flatten the whole software organization. And I would also familiarize myself with software anti patterns like: Architect Don't Code, Blow Hard Jamboree and Intellectual Dishonesty. There's no excuse for being organized for failure.
The IBM sales and support organizations may be amazing but they can only sell and support crap for so long before customers will take notice. In my opinion, they owe it to the staff, to the customer and to the stock holders to do a better job. That's my two cents and last words on the subject.
Apr 13 2005 - My Hardest Interview Question Before I started my job hunt a few months ago, it had been almost ten years since I had last interviewed outside of Iris or IBM. During that time I had forgotten the powerful collection of mixed emotions the process can stir up. On the down side there was self doubt and fear of rejection. On the positive side there's invigorating talk of new and different technology and the excitement of a new challenge. I can't say I enjoyed the process but it did provide a vehicle for some interesting self examination and introspection.
This time out, I spoke to a couple of companies before I finally landed my new job at Kubi Software. Both companies were in the storage space, building cheap scalable grids of commodity hardware based storage - much like Damien's original Couch idea. I liked one company, called Archivas, a lot but they decided I wasn't a good fit. Archivas was my first interview and I think I know why they reached their conclusion. There was one question they asked that I struggled with: "Tell us about the most difficult part of your most recent project". You would think that answer would roll off the tongue. The funny thing was I was stumped. I hemmed and hawed and just picked a tricky integration problem we had using DB2 row locks as a synchronization mechanism between mtas in a multi machine mail cell. When they probed on why this was so difficult I had nothing. The post interview feedback was simple: "We liked him but he didn't dive deep enough on the technical issues".
I've thought a lot about that question since then and I've come to a conclusion. While I may not be the deepest technician my main problem was that, by far, all the most difficult problems I had working on Workplace were organizational and procedural rather than technical. Sure there were tricky technical issues I faced but they were nothing compared to the bureaucratic mess - goat rodeo - cluster fuck, that was Workplace 1.1 and 2.0. (2.5 is better and 1.0 was a lot of fun but they're both different stories).
The worst example of a procedural problem gone out of control was the Portal install. In Workplace 1.1 and much of 2.0 when you need to install the latest build of Portal server it took anywhere from two days, if you were lucky, to a week or more of tedious hand editing of configuration files before the server would run. There was no automation for the process and no organizational investment in automating the process, so every few weeks all progress on Workplace would grind to a halt as people manually updated their servers. Hundreds and hundreds of Workplace developers and QE engineers would have to stop what they were working on and just follow pages of detailed install instructions. There were rouge attempts to address the issue but none gained team wide traction. All the while the drum beat of management would be pressing for faster action and more progress from development while apparently ignoring the team wide appeal for help on the issue. In retrospect it was the most frustrating and difficult development environment I've ever worked in.
I now better understand why that interview question was so hard. In comparison to the mind numbing procedural issues faced on Workplace, database deadlocks and thread safety errors paled as problems; days spent pouring over pages of logs looking for hints to a problem were a luxury and rewriting large chunks of someone else's code at the last minute to fix a critical issue was simple child's play.
In the end I'm glad I didn't get that Archivas job, as interesting as the storage issues are I like the collaboration space and am happy to have found a job where I can continue to work in it. I start at Kubi on Monday.
April 29, 2005
I'm going to describe some of the core of CouchDb architecture.
In the fewest word possible, the purpose of CouchDb is to be an indexable schema-less database. To explain briefly the properties of a Couch database:
1. CouchDb will store primarily "objects", but these aren't objects with behavior, only structure and data. The object is split in to two parts, summary and payload data. The summary data will be the information that is used in generating tables, and will consist of simple name/value pairs and meta data about the object. The payload data will be the bulky data, like file attachments and long html bodies.
2. CouchDbs will have tables and indexes. These tables are built using the summary data from the object. The tables will be built using custom logic specified by the user/developer, and each row in the table will correspond to a single object.
3. CouchDbs will replicatable.
If you know the Lotus Notes NSF data model, all of this will sound very familiar. It's very similar to the key underpinnings of Notes and Domino (and still its biggest differentiator from other development products).
From the outside, CouchDb will look and act very similarly to NSF, but the internal architecture will be somewhat different. CouchDb uses combination of append-only storage, and atomic updates. Notes uses atomic-updates (it also has transaction logging capability as well).
Append-only databases (also known as zero-overwrite databases) are databases where the persistent storage is never overwritten. All database actions (insert, update, delete), even index building, are the results of appends to the end of the file, existing data in the database isn't overwritten, instead it just becomes "outdated", with the newer data taking precedence. Databases designed this way have some interesting properties:
1. Updates to the database are very fast. Since all writes are at the end, there is much less seeking for the disk heads, so writes are very fast.
2. Truncated databases aren't corrupt, instead they are simply an earlier version of the database. This makes incremental back-ups a breeze, and even simple tools can backup an open database.
3. Incomplete writes aren't a problem, if a database update fails (for instance, due to power failure), it can't corrupt the existing state of the database.
This design has one obvious shortcoming, the database file is always growing with every change (even deletes), and old, useless data continues to waste disk space. All append-only databases have some way to compact or "vacuum" the file. Often it is a simple as creating a new file and writing all the current data into it, then replacing the old file with the new, but more sophisticated schemes are also used to compact the database in place.
Unfortunately, this design has a serious limitation that I think make it a poor choice for Couch. In my experience, the single biggest performance bottleneck in most Notes applications and servers is the view indexing. Typically when a Notes database is slow to open, it is because one or more needed view indexes is out of date and is being incrementally updated. Now if the data to build these views is scattered all over the file, then it will take many disk reads to load up the data before it can compute the table and index values. Since every update happens at the end of the file, the updates, over time, will increasingly be scattered over the file, meaning many seeks and data reads. Disk IO is the performance killer, and anything that can reduce disk IO helps.
To speed thing ups, Notes has an optimization where all the summary data for documents are placed into "summary buckets", such that many document's summary data can be packed into in a single bucket. Then when building the index, the summary data for the many documents can be loaded in one shot.
So if Couch utilized an append-only design, then when building it's tables and indexes, it would often have to jump around, reading small parts of the file, and greatly increasing the disk IO, and greatly slowing down the rate at which it can build the tables. So instead Couch adopts a combination of streaming zero-overwrite storage and atomic updates.
Couch Streaming Storage
Inside a CouchDb, the beginning of the file contains a traditional header that contains the meta-data about the database and information about where key structures are located inside in the database files. Following the header is the raw storage.
All raw storage is just a stream of data, and the stream is made up of file segments, which are essentially linked lists. Each segment has a small header and footer. The header specifies the length of the segment and has a pointer to the previous segment (for navigating a stream backwards), and the footer has a pointer to the next segment. All the space in between is the data for the stream.
Here is a diagram of a stream segment. It is the first segment in the stream (no previous segment), the data in the segment is 14 (0x0E) bytes long, and the location of the next segment is at file position 17772 (0x456C):
| Prev | Data | ... Data ... | Next |
| Segment| Length | | Segment|
And here is a diagram of a CouchDb file with three streams, X, Y and Z, each with multiple segments:
HEADER XXXXX YYYY ZZZ XXXXX YYYYYYY YYYYY ZZZZZ XXXX
| | +-|---|-|-----|-|-----+ |
| +-----|---|-+ +-+ |
When a new segment is allocated (usually at the end of the file), its header is written in and its footer is zero'd and data can be streamed into it. Once the segment is filled with data, a new segment is allocated and the footer of the existing filled segment is written with the location of the new segment. This means that any storage stream is unbounded in length, which makes it easy to store data where the length isn't known ahead of time.
This streaming architecture is the core of the CouchDb storage model. Since streaming data is such a big part of how Couch will work, I thought about renaming it to Streamy. Then I realized I could be really trendy and call it xStream storage. Then I punched myself in the nuts for coming up with such stupid name.
The main storage streams in Couch are:
Summary stream - This contains all the meta-information about each object. As each object is written to the database, all it's summary information is appended to this stream.
Update Log stream - For each object written or deleted, a small record of the update is appended to this stream. Each record contains the object's ID, its summary stream position and payload position, and the action taken (update or delete).
The Update Log stream is intended to be a stream that can be quickly "played back" to determine where the actual summary and payload data for an object resides.
To get around the inherent deficiencies in always playing back the full streams in order to find out the latest state of the stream (including its end), pointers to the ends or other key places of the streams can be stored in a header (it can also be stored in more streaming storage). Also, if streams get too wasteful (the percentage of invalidated entries is high), the streams can be copied out to new streams with the invalidated data removed.
Doubly Written Headers
The headers in CouchDb are doubly written headers. That is, the header is duplicated, one right after the other. This way, when "committing" writes to the database by updating a header, if the power suddenly goes out in the middle of the write, the header should still be in a recoverable state. If one header is partially written, then the other header can be used. When this happens, the data written to the stream is lost, but the stream and database are not corrupt, and no "fixup" or consistency check will be necessary after a crash. And since the headers are typically small, the double writing shouldn't add much disk overhead.
So whenever an index is opened, it will have a record of the last entry in the update log stream that it processed. It will then start reading the log from that point and build a list of all the objects that have been updated or deleted since. Then it will stream out all object summaries from the summary update stream, and add the new records to the index. Since the object stream is mostly contiguous on disk, loading the summaries from disk should be pretty efficient.
The indexes it creates will be stored likewise in an append-only storage stream. As it loads and processes the summaries, it will add the modifications to the index in RAM memory. Once it's used a certain amount of memory, the index changes will flushed to the index stream.
Again, because of the streaming nature of the database, both the reads and the writes should be pretty efficient, with relatively little disk seeking.
Ok, I've written enough for now. During the process of developing CouchDb, I am making many unproven assumptions, so if anyone sees any flaws or mistaken assumptions above, please tell me. Also, I welcome any questions, as they'll help me clarify and explain this stuff better in the future.
April 25, 2005
I'm leaving tomorrow morning for Palo Alto, where I'm going to be an expert witness in a patent lawsuit. I'll be there until May 6th. It apparently involves producing lots of documents and getting deposed, maybe even dramatic testimony on the witness stand! (that is if you find technical jargon dramatic) I can't say much about it, I may have said too much already (I'll never understand the law). And it will probably take me a little bit longer to blog about the CouchDb architecture, since according the lawyers I will be very busy.
I still be available via email, but I may be a little slow to respond during the day.
"Someone keeps stealing my letters..."
I was trying to make "Damien is cool" but someone kept taking my O's. So I settled for ruling.
April 24, 2005
First thing ever stored in a CouchDb?
"Damien is cool"
"Damien is coolDamien is cool"
I was testing streaming appends. That was a few days ago and today I just got a complete "note" to save and load. It held one field named "Foo", and it stored the string "Damien is cool". Indeed, I am cool.
So I now have the basic storage model for Couch all worked out, I'll be blogging the architecture and my design rationales very soon. However, the reasons why I am so cool will not be revealed, you'll just have to trust me.
April 22, 2005
A very cool game where it displays results for a Google image search and you guess the word being searched.
April 20, 2005
Bruce Elgort gives a good introduction to Trigger Happy and a very positive review. Thanks Bruce!
Trigger Happy is a Lotus Notes and Domino add-on I created that lets developers control low level Domino events using LotusScript. See, every time I try to describe it it sounds like mumbo-jumbo. Which is why you should read Bruce's article, unless you not a Notes/Domino developer, in which case you'd just be puzzled.
I'm rude as shit
Speaking of international dining etiquette reminds me of a story from my mother-in-law. In Slovak culture, a guest is asked three times whether they would like more to eat. Proper etiquette dictates that you kindly turn down the first two requests and then acquiesce on the third.
This explains why my mother-in-law (of Slovak descent) is always trying to get me to eat more food.
Her: "Damien... there's some more porks chops for you."
Me: "No thanks. Had my fill, but they were really good."
"How about some green bean cassarole?"
"No I'm full. Stuffed really."
...20 seconds later...
"Would you like some cranberries?"
"No thanks, already had some."
"There's ice cream in the fridge. It's pecan!"
[a little annoyed] "I ate too much already..."
"How about some of these cookies I bought on sale? 50% off!"
"Look dammit... I'm not HUNGRY ANYMORE!"
And then I feel like a heel.
She's the sweetest women ever (really!), but please... stop with the food already! BTW, I scored 3 of 11 on the Don't Gross Out the World quiz. Sounds about right.
SongMeanings.net goes down, people freak out
Yesterday Songmeaning.net went down (just checked, still down). I started getting some comments and emails about it, which is odd because I have no affiliation with the site, I only posted a link to it. So, I googled "Songmeanings" and for some reason my blog post is the 3rd result. I guess that explains why people are contacting me. One email bordered on hysterical, the person clearly has a very strong attachment to the site.
Based on what little information I have, it sounds like the site has a very active set of users whose needs aren't quite being met. I've spent a little time on the site, and it could use a lot of usability tweaks, and apparently a much more reliable backend. I'd guess there is some real opportunity here for an enterprising web developer, the demand appears to be solid and unmet.
Cheer up Fatty
Once again a quick review of American Idol by moi.
- Constantine - zzzzzzz...
- Carrie - Something was missing from her voice.
- Scott - Now I remember why I like him so much, he's a damn good singer.
- Anthony - Good performance again. He still annoys me.
- Vonzell - Great performance. She's almost too cute.
- Anwar - Started a little hesitant. Goofy dancing.
- Bo Bice - Sounded awesome. AWESOME.
The competition is just so strong now, it's hard to make a prediction. I think Anwar will go.
April 19, 2005
Way cool. If I had the space I'd make my own.
Aligning structures on disk
For some reason, I feel likes it necessary when allocating disk storage space inside CouchDb that it has to be aligned to a power-of-two boundary. It's seem likes that's necessary somehow, but I think it's just a carry-over from writing Compute and having to deal with memory alignment problems. I can't think of any reason disk storage should have alignment issues, I think even 1 Byte alignment is fine. Yet somehow it feels wrong.
April 18, 2005
Politicizing of Education
It used to be, if you wanted to change education you'd have change one school system at a time. BORRRRRRING. Now thankfully that drudgery is gone, and you can just focus on the DC politicians and change it for everybody all at once. I predict a flurry of useful and innovative advances in education now that the federal government is involved.
Yes, I'm in a pissy mood.
It's Netflix for sex toys. Ewwwwwwwww.
What happened to FDR's plans?
Browsing the NY Times most popular articles I found the number one most emailed article is an Op-Ed piece where the columnist wonders what happened to all of FDR's great ideals. Further down the list is an article discussing the belief that socialist Norway is one of the richest countries in the world.
I'm not much of a gamer, but I found this article fascinating. It makes me wish I was working with really smart people again.
April 13, 2005
They are accepting applications for a real position as the CMT Dukes of Hazzard Institute vice president.
The job responsibilities are:
- watch The Dukes of Hazzard every weeknight on CMT;
- know the words to The Dukes of Hazzard theme song, "Good Ol' Boys," written and performed on the series by the legendary Waylon Jennings;
- serve as media expert on The Dukes of Hazzard for the CMT Dukes of Hazzard Institute: must be available for TV, radio and newspaper interviews to share passion for The Dukes of Hazzard on CMT;
- write the CMT Dukes of Hazzard Institute online blog for cmt.com;
- be passionate about The Dukes of Hazzard on CMT;
- make appearances at special events such as Dukesfest 2005 in Bristol, Tenn., (June 4-5, 2005).
Me! Pick me! PICK MEEEEE!!!!
I am really going to try to get this job. It is my duty to mankind that I get paid to watch the Dukes of Hazzard.
<rebel yell> YEEE HAAWWWW! </rebel yell>
My quick review of American Idol last night. You don't watch? I pity you.
In order of appearance:
- Nadia Turner - zzzzz....
- Bo Bice - Should have been better. I think the band's arrangement was partially at fault.
- Anwar Robinson - Sounded great. A little boring.
- Anthony Fedorov - I actually don't want him to die after that performance.
- Vonzell Solomon - So complete, so damn pretty too. It's hard to stop watching her.
- Scott Savol - A little hesitant, but he worked it out. I really like him for some reason.
- Carrie Underwood - That voice is amazing, I think she sounded better than Pat Benatar. A slightly goofy performance though.
- Constantine - Wow. Really impressed. While he can't possibly sound as good as Freddie Mercury (it's not possible, just isn't), he did nail it. He's hot XOXOX
Prediction: Nadia will go. And I think based on her performance she should.
April 12, 2005
Wikipedia: Software Development Stories
I created this page last night, and I included two interesting software development stories, plus my own. I don't know if this is an abuse of the Wikipedia or what, it's a bit of an experiment: I liked reading those other two stories, and of course I think my own is da bomb, and I'd like to read more. So if you know of any other interesting tales of software development, add them to the page.
Of course, if some Wikipedian thinks that's page doesn't belong there, then I guess they're free to delete it. I hope I haven't broken any rules of etiquette, but like I said it's an experiment.
Ok, so I've been very remiss in that I haven't blogged anything Couch related. There is a good reason for that, I haven't been working on it very much. Why? I just wasn't in the right mindset for it, which is a wuss way of saying I didn't feel like it.
I've been doing some contract work, but mostly I've worked on lots of little personal projects. Some, like Clicky and Idiot-B-Gone, were only a day or two. Others, like my random code generator, I just I lost interest. And of course I spent a lot of time on Trigger Happy.
In case you didn't know, Couch is going to be the storage system for a large scale object database. My long term goal is a database that can serve as the back end to a very large scale RSS aggregator.
The lowest layer of the project is CouchDB. CouchDB will be the code layer that actually shuffles the bits around on disk. It will be responsible for object storage, retrieval and indexing. Think Notes NSF and NIF.
So for the past couple of weeks I've been doing more and more work on CouchDB. When I was trying to figure out how I was going to architect all the on disk structures I reached a big breakthrough....I know nothing about building a database. And I need to accept that, and just move forward anyway. Even though I've worked a lot on NSF, I think that is just confusing me. Because NSF has all this complexity that sort of grew organically. I don't mean that as a bad thing at all. It's a big part of its success. But it's also not how you'd design a database system if you're building from scratch. To put in that complexity ahead of time is asking for disaster.
So I'm reminding myself that writing code is like molding clay, not carving a sculpture. Code is plastic, it can change oh so very easily. So I need to just write the damn code! When I rewrote Compute, that's exactly what I did. I was so precocious.
So I need to push. And I'm pretty sure I know what will happen. My current design will start to form, but I will be unhappy with it. Suddenly it will dawn on what is wrong, then I will throw most of it away and embark on the new architecture. Yeah, that sounds about right. So the faster I get going down the wrong path, the sooner I can get to the right one.
I largely now have the main on disk structures all worked out, and I'm implementing the bucket bit map code, which is basically the outermost memory manager for disk space.
Oh, BTW, I'm writing the CouchDB layer in C++. I know, I'm a fucking hypocrite. But I have my reasons: It's mature and portable and the language is stable. I've thought of using Java, but I need something that's more lightweight and something that can be implemented as Crash Only. I've also considered Python and Ruby. One thing I don't like about Python is how often I'm surprised by undiscovered behaviors. In my mostly uninformed opinion, it's starting to turn into C++, with little special cases and obscure language features creeping in. And the APIs for it are very poorly specified. Ruby seems too immature as well. This is code that needs to be highly reliable, I need to know exactly what the code is going to do at all times. I want no surprises.
The more I code this stuff, the more excited I get. I'm having fun just coding all the details. I'm sure making design mistakes left and right and I can't even see them. And I don't care. Because later as more of the big picture is in place I'll be more able to see which parts I screwed up on and then I can fix them or throw them away.
Once I get the bucket management code done I'll talk about how it's implemented and I'll also talk about more of the storage design. Stay tuned.
Botero takes on Abu Ghraib
John Paul II's Last Words:
Stolen from the Onion.
April 11, 2005
Highest Paying Keywords
This site lists the highest paying Adsense keywords, which are quite surprising.
For instance, laser hair removal for politicians and bureaucrats must be quite popular and profitable, as "dc hair laser removal washington", "hair removal washington dc" and "hair laser removal virginia" are all in the top ten. Damn dirty apes.
The number one paying adsense keywords are "domains yahoo", which pays a whopping $97.44 per click. Umm, don't domains only cost like $8 to register? How can that possibly be profitable?
April 10, 2005
I realized today when I "IM", my lips move as I type.
They're Fighting About What?
Short version of the story: Some Jewish leaders are mad because Mormons are doing secret "proxy baptismals" where a Mormon is baptised while pretending to be a particular Holocaust victim or other deceased Jew. So mad, in fact, they're trying to get Senator Hillary Clinton to intervene.
In unrelated news, Jimmy won't stop staring at his sister during dinner.
April 8, 2005
Worst Phisher Ever
I get the feeling this scammer doesn't quite understand how phishing is supposed to work. I almost feel sorry for him.
Holy Crap! I guess this phisher is way more clever than I gave him credit for. When viewed in Firefox, the text below is completely scrambled. In IE, it appears normal. I'm not sure why that is, but looking at the HTML it looks like its exploiting some flaw in IE. It even makes it all the way through Bloglines (my RSS aggregator) like that: scrambled in Firefox, normal in IE.
This appears to be a new exploit, and quite clever too. Ned has written a detailed analysis. That's the last time I almost feel sorry for a scammer.
Dera Bcralays Merebm,
Tsih eamil was setn by the Balcrays svreer to virefy yruo
eamil adsserd. You mtsu comelpte tsih procses by cilcking
the lkni bwole and entegnir in the samll wiwodn yruo
Balcrays Mbmeership nebmur, paedocss and mbaromele wrod.
is deno for yuor protcetion - baceuse semo of our membres
no lregno hvae aseccs to tehir eliam asserddes and
mtsu veryfi it. To vefiry yruo eliam arddess and accsse yruo bakn acocunt , clkci on the lkni belwo:
April 7, 2005
Reading code is WAY harder than writing it
"Reading code is WAY harder than writing it"
I've heard that before, and I read that again today on a discussion board. My experience is the opposite, but then I've read a lot of code over the years, my brain is trained to do it. And writing good code isn't easy, it takes planning, discipline and meticulous attention to details.
But perhaps my world is different from most developers. Do people generally think that's true?
From Hell's Heart, I Stab at Thee
I got the money back from Vonage. Easy as pie! The thing is, it didn't take the mandatory 2-3 business days they said it would take. It was the same night. Weird. I guess when properly motivated, they can get something done quickly. Anyone with Vonage problems should call Melissa of the Executive Response Team at 732-226-3414, she works miracles. Because when you need something done, you need an Executive Response! But I'm not sure if you have to be an executive or not to get the response, they never asked.
So, sadly, it looks like my battle is over. I have no more angle to exploit. So I'll just have to let my Fuck Vonage page rest, and wait and watch as more people link to it. Too bad too, I was hoping I could be a new Ralph Nader, only less ghoulish and more proactive. Well, it was fun for a little while.
Oh and by the way, you might want to look at this VOIP round-up article from Philip Greenspun. He looks at a lot of VOIP providers and options, but the stuff he says about Vonage is of course what I find interesting. A small excerpt:
After nearly a week of not getting any phone calls I was able to determine that Vonage owned the number. I called Vonage tech support and waited 45 minutes in queue before someone answered. He transferred me to a woman in the porting group. She tried to put me on hold for a moment to test my number but inadvertently hung up on me. She never called back.
Philip is being charitable, that hang-up was no accident. Or perhaps it is true that people at Vonage need more practice operating their telephone devices. Who can really say?
Anyway, he's got more good stuff there about Vonage. Hey Phil, did you ever receive your refund when you cancelled? I'm betting not, but being the busy guy you are, you probably didn't notice. But don't worry, it went to good use, someone is counting on that money. Helicopter rides to work don't pay for themselves!
So I have one last thing for Vonage. 1600 unique visitor have seen the page in the last 3 days. Now I'm betting most of those visitors aren't going to use Vonage in the future, they'll shop around a bit and choose a competitor. Excellent. But I'd like for more people to see it. So I decided to take that $98.97 that took me 5 months to get back, and I'm buying Google Ads. I'm buying the Adword "Vonage", and it will cost me probably around 3-5 cents a click, which would bring in roughly 2500 visitors. That's a hell of a bargain, and unlike traditional advertisers, I don't have to worry about making a sale. So all traffic is good traffic.
| VOIP Horror Stories |
It took 5 months to get back my
$100. Many never will. Learn more.
And of course it links to the Fuck Vonage page.
So, other than some last cleanup and tweaks of the Vonage posts, that's all I'm going to do for now. I'm ready to move on and get back to what I do best, writing software. It's been fun Vonage.
Glad I switched to Movable Type
I would have never switched if Blogger.com weren't so damned flakey.
April 6, 2005
Damien the Angry Consumer
So some people might be wondering why it is I feel so strongly about the Vonage stuff. You might think they're a great company with cool commercials or whatever, that's fine. You might think I'm a self-important ass who for some reason thinks he can take on a big company, that's fine. But I thought I'd spend a little time explaining why I'm fighting, and why I think it's important and why I'm so angry.
Partially I'm fighting because I want to believe there are people out there who are willing to stand up, who won't roll over and just take it. I'm fighting because I believe that other people out there get angry too and and want to make a stand, even if its just something small, like writing about what's happened. I'm fighting because someone has to get angry. And I want to tell you why I'm angry.
Obviously part of why I'm angry is from the terrible, frustrating service. The most unbelievably bad service. Oh so many wasted hours, always feeling like we're on the verge of getting the situation fixed, and getting disconnected, or finding out again they didn't give us back our money. But if that were everything, you'd would have just seen my one post with all my anger and vitriol showing. And that would be that. But that's not that.
I'm very angry about is how the system has failed in the case of Vonage CEO Jeffery A. Citron. The SEC charges against him are pretty clear: blatant stealing, blatant fraud. This wasn't "questionable" practices issue, it was clearly criminal. As clearly criminal as the guy who stole my car stereo last week. It's also clear that he paid what to him is a petty fine $22.5 million - only a fraction of the money stolen. This is the guy who loved to brag about his money, his mansions and jet. He took a helicopter to work because he's so important. He loves living it up and flaunting his wealth. And when he was caught stealing the money he used to fuel his excessive lifestyle, he didn't go to jail. He didn't go broke. He didn't do community service. He simply walked away, and with the vast majority of his stolen hundreds of millions intact.
That could be the retirement money for many people. My grandmother is in danger of going broke in her assisted care living (she has dementia), and might soon have to go a nursing home. I know excellent engineers with long successful careers who wanted to retire but now can't because the market wiped out a huge part of their savings.
And it's because jerks like this are completely gaming the system, they take their millions and never care about their unseen victims. And why? So someone waaay more self-important than I can have his own private jet and expensive mansions and take his helicopter to work.
I don't think anyone is so naive to think this stuff never happens. We've all seen Enron and Tyco. I expect that even in the best circumstances corporate crime is going to happen and you just can't catch most of it. But we'd like to believe the people who get caught are made to pay. So here they caught the bad guys ripping off the system in a big big way, caught red-handed, a well documented, clear cut case. But they were let go. Why? Because on the very same day the charges were filed, the bad guys "settled" in a deal where they pay a fraction of the money they stole and $70,000,000 goes to the SEC. The SEC gets to keep that money, and furnish their offices in a style befitting the SEC, I suppose.
And so not only did Citron get away with it, HE'S DOING IT AGAIN. He's stealing from decent people, by deliberately making it impossible and frustrating for people to get their own money back. Google "Vonage problems", it's the same story over and a over. Not everyone can afford to be ripped off. Some people choose this service because it's cheap. They NEED cheap, they don't make a fat salary. Some people really need that money. And they're getting ripped off and don't have the time to deal with it.
So I'm doing what I can, which is to point out what's going on. To what end? I don't know, but I'm pretty sure I'm getting my money back. And I'm really happy I got to yell at the Chief Marketing Officer (the second best person to yell at IMO). And every person who chooses another carrier and avoids getting ripped off again is a victory.
I was bluffing at first when I said I'd sue them. I knew I could turn it into a PR stunt, but it was an idle threat. I had better things to do, but I hoped it would get some results. But then I got really angry. I began to see this wasn't pure incompetence I was facing, this was a company ripping people off intentionally. I realized I could make it a very interesting story if I had to actually sue them. If they give me my money back by tommorrow (I have to call AMEX to verify) then I have no interesting story to tell, I've lost the angle. I almost hope I don't get my money back this time.
If I get my money back, I have one last low-effort measure I'll employ to draw attention. After that I'll just keep the page up, update it occasionally with people's latest stories. As more and more Vonage customers go through what I did (wait until you cancel), they too will get angry. And they will rememeber that lunatic Damien who had some trouble. And they will find and link to the page. It's not going to make the first page of "Vonage" search results overnight, it will take some time and it might never happen, but as long as they keep ripping people off, more people will find and link to my page.
Or maybe they clean-up their act and that's that.
Of course, if they don't pay the money back I really will sue them in small claims court and write about the whole thing and make it as interesting as possible. Ever wonder what really happens when you sue a big company in small claims court? I do. Hell, my family and I are pretty reality TV friendly and my wife is completely on board (it was originally her anger and frustration that drove me blog this), maybe some network would like to film my Don Quixote antics. Shoot it in a Roger and Me documentary style. Use the buzzword "blog" on every promo for it. And show me doing all the incredibly moronic stuff I do all day (ask my coworkers). Whittle it down, it could make for some television. (not a typo)
And I'm also sure the PR departments for their competitors would love to give me some press contacts. I quite certain I can find some powerful allies that can benefit from my rantings.
My point is, I'm a very resourceful guy, and right now writing software isn't as interesting as trying to mess with a criminal corporation. I've not even begun to show how resourceful I am, or how big of a pain in the ass I can be. I'm a hard guy to stop when motivated.
You might think I'm crazy, but at some point you too will get angry, you'll see what is allowed to happen, you will be disgusted that the people who steal the most from us, even when caught, aren't made to pay for what they've done. And when you catch them stealing from you once again, you too will want to strike back with what you can. And when that happens to you, I hope you have the courage and energy to join the fun.
April 5, 2005
Who wouldn't want an invisble alien friend that grants wishes?
April 4, 2005
This page is about our dealings with Vonage. In Novemeber of 2004, we ordered Vonage service for our new house, and almost immediately it was clear we didn't want to be their customer. So we tried to cancel. It has taken 5 months and countless hours on the phone trying to get our money back. Our story is hardly unique. Please read more, or just research another VOIP provider. This page, of which I am not affiliated, provides a good real world roundup of some popular providers and technologies. But stay away from Vonage, life is too short to be put through that hassle.
You may remember my previous dealings with Vonage 5 months ago. Well, a few days after I posted my article, we received a call from Vonage saying the charges were going to be taken off the credit card. We thought the issue was done. Not even close.
My wife somehow noticed that the charges were never taken off. She initially called the card company to dispute, but it had been too long, we only get 2 months. Our only option according to Amex is to call Vonage. So we did. After much time on the phone on hold, I finally got through to an Indian man named Viren, who claimed he was in New Jersey. He found the charges, said they were going to be credited back to the card within 24 hours. Miracle of miracle, I called, got through and the situation is handled. That was last Wednesday.
Well, it’s Monday, and the charges haven't been credited back. When I call their customer line today, 1-vonage-help (they should be sued by the FTC for calling it that), after going through the phone menu maze, I get a busy signal. Not "on-hold" music, a beep beep beep busy signal. A you-must-hang-up-and-try-again busy signal. And so I tried again. And again. How ironic, this is a phone company, yet they can't do a simple thing like put me on hold. It takes a little while to navigate the menu before getting that busy signal, so I gave up.
We found the phone number to the CEO of Vonage, which is Jeffrey A. Citron at 732-528-2600. I called and asked to speak with Mr. Citron, then got the voice mail of someone in Mr. Citron's office. So I called back and asked the operator to be connected to an actual person, not voice mail. Viola, transferred again to someone else's voice mail. Then I called and said that I was transferred to voice mail, and could I just talk to an actual person even if I have to hold, and the operator hung up on me. I called again, I asked for the operators name, and was hung up on again. I called again, finally got the operators name, Renee (maybe a different operator from before), who transferred me to the "VP of Customer Care's" voice mail. I left a long voice mail, explaining the situation and that I'm seething mad and need this resolved today.
And yet, they still answer their sales line after one ring.
[long string of expletives deleted]
If I can't get this resolved today, I'm going to file a lawsuit in small claims court. I'll blog the whole thing and it will be a PR nightmare for them. I'll probably increase my readership along the way, and I'll let as many people as possible know what they've done. I'll have some great fun with this. Vonage, I'm going to make you pay.
Update: Execute Response Team
My wife started calling all the names and numbers on this page and finally was able to get Dean Harris, the Vonage Chief Marketing Officer, on the phone. I told him to look at damienkatz.net too see the whole story. He was very polite, even though I was very angry. Dean Harris then assured me get was walking down to customer service to to see what is happening and I will be contacted by someone. I told him I wanted this resolved today or else I sue and turn it into a PR stunt amd give them tons of bad publicity.
An hour later, I receive a call from Melissa on the Vonage Executive Response Team. She explains that I was indeed refunded the money last week, it was credited to my account, but that the representitive didn't actually credit it back to my card. In otherwords, I had a $98.97 credit on my Vonage account. I asked why he hadn't done that, she said she wasn't sure, that he may have not known there was another step necessary or that maybe he wasn't approved to credit money back to credit cards. I asked, "So you guys screwed up again?". She said no, it wasn't a screw up, it was credited back to my account, just not back to my credit card. This is is ridiculous, as the guy last week assured me he was putting it back on my card and even gave me a special "transaction number" for it. So after much badgering, I finally got her to admit that Vonage had screwed up again, but it took three tries asking her point blank, "Did you guys screw up again?".
She said the charge will now be back on the card in 2 to 3 business days. I asked her how can I trust this will happen, when it was supposed to happen 3 times already? She said since I have her direct number, I can call her directly if I need to, but she can give no other assurances other than that. I guess that will have to be enough for now.
Update 2: Criminal CEO?
As Ronnie points out in the comments, Jeffrey A. Citron the CEO of Vonage, was indicted for Securities Fraud in 2003 (http://www.sec.gov/litigation/complaints/comp17929.htm) and he and his Datek cohorts agreed to pay the SEC $70 million in fines, of which Citron's portion was $22.5 million(http://www.sec.gov/news/press/2003-5.htm). What were the charges? From the SEC page:
The Securities and Exchange Commission today charged Sheldon Maschler, Jeffrey A. Citron, Michael McCarty, Erik Maschler, and Heartland Securities Corp. with participating in an extensive fraudulent scheme involving the Nasdaq Stock Market's Small Order Execution System (SOES) from 1993 to June 2001.
According to the complaint, Sheldon Maschler, Citron, McCarty, and Erik Maschler accessed the SOES system to execute millions of unlawful proprietary trades, generating tens of millions of dollars in illegal profits. These defendants hid their fraudulent use of the SOES system from regulators by allocating proprietary trades to dozens of nominee customer accounts, creating fictitious books and records, and filing false reports with the SEC.
Sounds like the guy should be sitting in prison cell, instead he's sitting in a board room. Maybe their business model really is ripping off customers $100 at a time.
The amount of money they stole, STOLE, in total, is believed to be in excess of $200 million dollars. But on the same day the SEC filed the charges, Citron and his accomplices agreed to settle for $70,000,000. That's right, they just wrote a big check to the SEC and all was forgiven. And the SEC kept that money.
Is it any wonder I've had such a hard time getting my money back. It looks like Mr. Citron has learned that as long as you have a big ad budget, you can screw people as much as you like. Maybe you'll have to give some of your stolen money back, but that's about it.
Update 3: Not Alone
There are many other tales of poor service, but I'm going to focus people not getting their money back. Actually, I'm just going to focus on the first link when I Google Vonage Problems.
njjoe Wed Mar 09, 2005: 2 months now and they still have not transfered my number.
So I am still paying for my old service and new!
I am sorry I picked VONAGE as my Voice over IP service!
I have been trying all week to get trough to technical support and all I get is busy signals!
davisual, Fri Mar 11, 2005: I've waited for over two months to get my number transfered from BellSouth to Vonage. Meanwhile being charged by both ... I cannot get Vonage to respond or correct the problem . They will not acknowlede faxes, phone calls or any means of communication.
GotRidofVonage, Mon Mar 14, 2005: I was in the same position: paying for 3 month for two services because Vonage was unable to transfer my phone number, could not get to a live person at their customer support. Eventually, found a cancellation department number: 1-800-860-5491. It worked fairly well for me. Almost no waiting time, they cancelled my service in two minutes. Cancellation fee was $40 (weird!), but they promised to credit this amount after I return the equipment. No way: in a month after they received my router I stilll did not get the money back. I called them back (this morning) and got my credit (not sure if it really happened, will check my credit card statement in a week).
Anyway, I am so happy my Vonage period is over.
stoib, Mon Mar 21, 2005: We have not been able to make or recieve calls (about 1 of 20 will get through, but drop out.) for over a month now with Vonage. we have done all the troubleshooting and the router and cable modem work fine. I called numerous times from work since I can't call from home and have not been able to resolve anything with Vonage. We decided to switch back to SBC on Feb. 23, 2005 and still have not been able to because Vonage will not release the line. We called Vonage with a cell phone and were on hold so long (> 45 mins) that the battery went dead. Not tomention the minutes used. Vonage has been totally indifferent and rude concerning this issue. On top of it all, they continue to bill us!And my personal favorite:
Bren, Tue Mar 22, 2005: If I could offer advice to the world, it would be run away screaming from Vonage! 'Cause if you don't, you will be screaming in frustration! They don't say on their website that you can't get TiVo series one to work with Vonage. I could not, so I decided to cancel. I have sent cancellation emails every day for almost 2 weeks. Yesterday I called, FINALLY got thru to someone named Fred, who just kept repeating "hello...hello...oh man, hello?". He couldn't hear a single word I was saying! In frustration I was screaming into the phone, "Can't hear me? You must be using your own sucky phone service!". I had to hang up, dial again, waited another 45 minutes, got a guy named Vincent who told me that "The termination lines are experiencing heavy call volume (there's a surprise, huh? lol) and their calls are being rolled over to reuglar customer service lines." He told me they would call me back...yeah, right! I am still not cancelled. The customer service is the worst I have EVER dealt with! I am going to have to go thru my bank to cancel the debits to my account because I can't get cancelled. What is worse, I cancelled on the 14th day from when I ordered, so I should be refunded my whole amount, but I do NOT believe I will ever see this money again. Please people, don't be sucked in by the thought of cheaper phone service like I was!
Update 4: Alternative VOIP Carriers
I've not dealt with any other VOIP carrier, so I can't vouch anyone else. But so far I've gotten a good recommendation for Voicepulse (thanks Thomas). I you have VOIP carrier you'd recommend, please tell me about it in the comments.
Also, Philip Greenspun wrote about his real world experiences with different VOIP providers.
Why Are You Doing This?
I'm a software engineer. I've had a successful career and I'm respected for my work. I have never done anything like this. I have never sued, never picketed, never protested, never even threatened a lawsuit before. Until I dealt with Vonage.
It's not about the $100. It's just not. It's about a company that rips people off, that gives them the worst, most incompetent service, and then makes it impossible to get your money back. The first time they said they'd give me my money and didn't, 5 months ago, I believed it was incompetence. The second time they said they'd give me my money and didn't, I believed it was incompetence. The third time they said they'd give me my money and didn't, I believed it was incompetence.
When I found out the CEO, who's net worth is over $200 million, got that fortune by making fraudulent and illegal trades, and then filed phoney documents to the SEC to hide it (according to SEC court documents), I began to realize that my troubles are more than simple incompetence. These people are deliberately trying to make it as hard as possible to get my money. The more I looked, the more I saw it happening to other people. When they said, "We've refunded you money back to your card", they did nothing of the sort. They hoped we would just forget about it, just like most customers will. We almost did, except my wife is quite on top of things.
I don't like be taken advantage of, especially not by lowlife scum like Jeffery Citron. He owns jets, mansions, takes a helicopter to work. He has some of your money already, and he's living it up. So I'll do whatever small thing I can, and help other people avoid getting ripped off, feeling cheated, feeling frustrated and abused. It sucks. But Mr. Citron cares not, he only wants to take your money. Once he's got it, he's going to hang on to it.
That's why I'm angry. Because he rips off decent people. I'm not alone, very few people actually get their money back, most don't even know they got ripped off. I want to help stop that.
So I'm showing how much effort I must go though to get back my money from this company. I'm showing how angry they made me and my wife. I never even had actual service from them. I never had equipment, never placed a call. I'm here to make whatever dent I can in their giant corporate armor, and show people what happens if you become their
You might think I'm crazy. But the next time a big company really puts it to you, and your time is getting wasted, and you're being billed for things you haven't gotten, and the person who's supposed to help you just "accidently" hangs up for the third time, then at that point you will understand. And if you found out the slime running the company already has stolen hundreds of millions and is living high on your labor, then you too might finally say enough is enough. You too might want to hit back with what you can.
If shady, dishonest corporations make you angry, please link to this article with Vonage as a word in the link. I'd love to see "Fuck Vonage" right on the first Google page when people search on Vonage.
April 2, 2005
Interview with LCD Soundsystem's James Murphy
I think what people really liked about "Losing My Edge" was that you were willing make yourself part of the joke, that it wasn't ironic?
"Or it's ironic in the actual meaning of 'ironic', not the pose, not the cartoon televised version of irony. 'Irony' is a dirty word now, but it's actually very good... I'm not saying I don't care. I do, I care a lot. This is my life. It's not a 'whatever'. This is all I've done. I've never done anything else. I worked at a computer company for a minute. I tried to go to college. Neither one of them worked. This is what I've done since I was a kid, so I care a lot about it. I've put my life into it. I'm fully aware that it's my life. I don't have parents - they're gone. I don't get another life. I'm 34 years old and this is it. My entire youth is gone and dedicated to this, so I care enormously. I meet lots of people who don't realise that this is their only life. I just don't think a lot of people are aware that they've committed. You've got to know what you've committed. I'm just painfully aware of that. It doesn't mean that it's not hilarious and embarrassing [laughs] and humiliating."
I identify with this sentiment so much. My work to me is huge. So so huge. I care, I'm committed. So much it's embarrassing. It really is.
April 1, 2005
Here is a list of some free software I have written. (Free as in speech and beer)
Trigger Happy - A Lotus Notes and Domino add-on that let you use LotusScript to intercept low level events.
DbDesign - A Lotus Notes and Domino script library that allows easy access to design elements.
RightDoubleClickBack - A Firefox GreaseMonkey script that allows you to navigate back in your browser by right double clicking web pages. Simple and handy.
I really like feedback. If you find anything above useful, send me an email.
Very sad. This isn't an April Fools joke (I wish it were). This guy made a lot of people laugh very hard. He will be missed.
My new venture
I found something to supplement my income, apparently this is something that's recession proof. See my homepage for details. Wish me luck!
The April Fools page is now here http://damienkatz.net/paysite.html