Fabric Test Suite
Ok, so I've been working feverishly the past month or so trying to get the core features of the Fabric Formula Language to all fit together. You might say I've been busy as a beaver, but I don't like to think of myself as a beaver, I'd rather be a cougar. A cougar that's real busy working on something.
Anyway, now that I have it working and I'm happy with the design of things, I want to refactor the hell out the code (it looks like a cougar wrote it). So I need to go through each line of code, restructing and cleaning things up, looking for bugs and unmet edge cases and what not. But before I do all that, I really should have a comprehensive test suite of Fabric formulas to verify they continue compile and compute correctly as I make changes. I figure once that's in place it should only be a few days to get the code into good shape, it's only a few thousand lines so far.
So now the problem is how do I want to write the test suite? I'm thinking what I want is a seperate .exe that reads text files and the text files contain the formula tests and expected results. The .exe runs each formula and validates it computed correctly. This is pretty much how we did it at Iris. I'm not sure how I want to structure the text file, but I think something like this:
#InitialFields
foo - "a" "b" "c" "d"
bar - "g" "h" "i" "j"
#Formula
FIELD baz := foo + bar;
#Results
baz - "ag" "bh" "ci" "dj"
--
#InitialFields
foo - 1 2 3 4
#Formula
Sum(foo);
#Results
%return - 10
--
...etc...
So my question, is this a good way to go about it? The most obvious alternative is to use XML, but I don't want to have to escape the formula source for every "<" and ">" character, blech. I started to look around to see if the Python source has a core language test suite but I gave up after half an hour and I decided to "lazy web" the issue. I just now made the above format up, so I'm sure there's a better way, something more cougarish?
Posted January 4, 2006 6:37 PM
Comments
Use XML, just put the stuff into CDATA sections, so you don't have to escape anything.
And you don't even have to do that manually: Create a simple Notes DB, where each note defines a test, and have a agent/action to export all that (out of a view) into an XML file for you, essentially how automated RSS/Atom feeds are generated, too. Plus, you have your test cases nicely organized and stored in a single place, and can re-generate the XML file to run whenever you want.
Thomas Gumz, January 4, 2006 7:49 PM
Good suggestion about XML CDATA, I wasn't aware it let you avoid escaping text.
I'm not going to introduce a dependency on Notes for the test suite. I prefer to keep as much as possible the project in text files, it makes version control and casual project exploration far easier.
Damien, January 4, 2006 8:01 PM
Using Notes doesn't neccessarily mean having to introduce it as a dependency. Instead of having to edit all the files manually, you just edit and organize them in a DB and export them, and only keep those exported files under source control, and the test suite requires these files only. So think of it as a better productivity IDE in this case, instead of using Notepad or VS.NET to write all that XML manually.
Also, check out http://feedvalidator.org/testcases/rss/must/
as an example for using XML-based files for a test suite (these test RSS feeds for validity). They are more complex than your requirements, so don't start barfing right away ;-)
Also, these files were most likely automatically generated, too, so the idea of Notes for your purposes isn't too far off, either. I'd think you're faster whipping up a db with a form and view and export agent to create your XML test suite files, than grappling with Python and using Ned's COG tool (which is fine if you're cool with Python, don't get me wrong).
Thomas Gumz, January 4, 2006 8:13 PM
I wouldn't get too hung up on the format of the file. After all, no matter what the format, you'll have code that reads it, so if you later decide you want a different format, it'll be a piece of cake to write them out the new way.
I like Thomas's idea of using Notes as a "test case IDE". I did a similar thing when working on the DXL DTD. The DTD was actually generated by Java agents in a Notes database which was designed for writing DTDs. It let me organize the entries and work with them in a structured way, then push a button to write them all out as a text DTD.
Ned Batchelder, January 5, 2006 1:01 PM
Post a comment