Links

Categories

Tags


« | Main | »

Benchmarking

By Jewe | March 30, 2015

I have done some benchmarking, using different compilers to build the same runtime environment. It showed some interesting results.

There is a small demo-application in JewelScript’s SVN repository. It’s called “JILRunOnly”. I made this to show developers how to embed JewelScript in any application. It’s a simple command-line program written in ANSI-C, so it can be compiled on any platform using any compiler.

Having just added a new project for Microsoft’s VS Express 2013 to the repository, I came up with the idea to compare the outputs of all compilers I have access to. The compilers used are TDM-GCC 4.8.1 (via DevC++ 5.9.2), Microsoft VS Express 2008 and VS Express 2013.

I made fully optimized release builds of “JILRunOnly”, targeting 32-bit in VS 2008 and 32/64-bit in the other two compilers. Then I ran a synthetic benchmarking script and compared their execution times and virtual machine instructions per second.

Here are the results:

alloc loop fib ack IPS
TDM-GCC 32 3.287 s 1.594 s 0.922 s 0.42 s 91.18 M
TDM-GCC 64 3.105 s 1.513 s 0.905 s 0.421 s 95.52 M
VS2008 32 2.98 s 1.466 s 0.952 s 0.483 s 96.55 M
VS2013 32 3.619 s 2.356 s 1.138 s 0.609 s 73.38 M
VS2013 64 2.98 s 1.373 s 0.982 s 0.422 s 98.63 M

Benchmark descriptions:

alloc: Allocates, initializes and destroys 10 million script objects. Basically it measures memory allocation, deallocation and copying performance.

loop: Simply runs through six nested loops, each of them counting to 18, resulting in a total number of 34,012,224 iterations. This measures raw byte-code execution performance.

fib: Recursively calculates the Fibonacci sequence for N=32, resulting in 7,049,155 recursions. This mainly measures function call and variable stack performance.

ack: Recursively calculates the Ackerman sequence for M=3 N=8, resulting in 2,785,999 recursions. Measures function call and variable stack performance as well.

IPS: The average number of executed virtual machine instructions per second calculated by the script.

Conclusions:

The arithmetic / recursive benchmarks perform better with the GCC output, while the memory / iteration based benchmarks perform better with the Microsoft compiler. The 32-bit output of the VS 2013 compiler is a different story though. I have no idea why the 32-bit build would be so much slower than the 64-bit build of the same compiler, or the 32-bit build from VS 2008 for that matter.

It seems building 32-bit applications with VS 2013 is not recommended. If you have both VS 2008 and 2013 installed, I would recommend using VS 2008 to build 32-bit versions and VS 2013 for 64-bit versions.

Topics: blog, news | Comments Off on Benchmarking

Comments are closed.