« Basic exceptions added | Main | JewelScript 1.0 feature freeze »
Testing a programming language
By jewe | February 17, 2010
Expressions in an object-oriented programming language can become utterly complex, so how do you make sure that everything that should work does work? And how do you make sure that things that must not work, don’t? Here is how I do it.
Well, the first question is answered rather easily. Just make a file, or a set of files, that use all possible kinds of language constructs – then compile them. If you don’t get an error and it runs, it’s an indicator that the tested features work.
I have collected a set of 60+ test scripts for exactly that purpose. They are automatically compiled and run by a command-line version of the JewelScript runtime, and I use it frequently after having made changes to the compiler, just to verify that everything still works like it should.
But there’s a slight problem: You just cannot think off all of the possible combinations of expressions and other language elements, so you have to generalize and simplify. Therefore, if your set of files compiles, it is only an indication that the features work. Sometimes just exchanging a discrete type declaration by a typeless one somewhere can make a big difference. Or a type-conversion invoked that you haven’t tried.
Therefore I have a few applications, real life ones, not ones written for the sole purpose of testing, that actually use JewelScript for some purpose. These are very important, especially to test the runtime environment, but also to proof the language.
These applications are for example PCalc, a programmable calculator, NetRadio, an internet-radio where JewelScript handles the GUI, and AppletForge, a “2D game construction kit” if you will. Its an application that offers graphics and sound / music services and allows to program applets in JewelScript – for example games or any other graphical application. All of these applications use the runtime in a different way and therefore test a different set of features.
Like I said, testing if something works is rather easy, you can just try and see what happens. Testing if something does not work, because it shouldn’t, is a whole different story.
It’s a bit trickier to verify, that doing something that shouldn’t be possible, really results in an error – and the correct one. This is mainly because JewelScript will stop compiling after the first error. So you would have to make an own test script for every single expression or statement you want to fail-proof.
For cases like that, JewelScript has an internal self-test statement. Basically it allows me to have the language compile a block of code and then compare the actual result with the one I expect. Here’s a code example:
__selftest 0 { s = "hello"+"bye".toUpper(); } // must compile to pass
__selftest 0 { s = ("hello"+"bye").toUpper(); } // must compile
__selftest 1216 { i = ++1; } // must result in error 1216
__selftest 0 { i = {10,20,30}[1]++; } // must compile
__selftest 0 { return {10,20,30}.format("%d,%d,%d"); } // must compile
__selftest 1216 { return 1++; } // must result in error 1216
Dependent on the result, the compiler will notify me if the test passed or failed, but it will not stop compilation on such blocks. So all I have to do is compile the files and look at the log-output to see if everything has a PASS.
It doesn’t tell me all the combinations that someone somewhere probably might come up with, though. I still don’t have an automated solution to solve that problem, but i’m working on it. ;-)
Topics: blog | No Comments »
Comments
You must be logged in to post a comment.