Here are the current Fungicide rankings for a selection of interpreters. You may skip ahead to the results if you wish.
Page history
2010–04–07
- Fixed the memory vs. time plots to use dashed lines (no more solid lines of the same colour for different data sets) and to display the lines for timed out interpreters as well.
2010–04–05
- Initial release.
Interpreters benchmarked
Details about the different interpreters can be read over at the Mycology results comparison, this is just a listing:
- CCBI version 1.0.20
- CCBI version 2.0, in both 32-bit and 64-bit configurations
- cfunge r828, in both 32-bit and 64-bit configurations
- Language::Befunge version 4.12
- PyFunge version 0.5-rc2
- Rc/Funge–98 version 1.27
- Rc/Funge–98 version 2.02.00
- Stinkhorn r4
CCBI 1.0.20 was included just to compare against 2.0 and will be dropped from any future runs.
Interpreters were compiled with the maximum optimization settings they offered in their standard build system. For those that don’t optimize at all by default (Rc/Funge–98), or don’t have a build system for my platform (Stinkhorn), I simply used gcc -O3 -march=native -DNDEBUG.
All optional features were also enabled, with the one exception being cfunge’s HARDENED, which enables generic security-related things like stack smash protection.
It’s worth mentioning that this implies that more speed can be wrung out of at least CCBI 2.0, since features like support for the TRDS fingerprint add noticeable amounts of extra work even when they aren’t being used.
Language versions
- C: gcc 4.4.3
- D: LDC r1646, with LLVM r98957 for optimizing (the separate LLVM was used only for CCBI 2.0: CCBI 1.0.20 used the LLVM 2.6 built into LDC)
- Python 2.6.5
- Perl 5.10.1
Environment
The machine used for benchmarking was an Intel Q9550 clocked to 3.16 GHz, with 8 Gio of RAM and running Linux 2.6.33. All executables involved (the interpreters themselves or the interpreters’ interpreters, such as perl and python) were x86–64 ELF binaries.
Results
Plots are in the SVG format: if your browser can’t handle it, upgrade to a newer version or switch to something more capable, as appropriate; or simply download them and view them separately.
In tables, the best result is strongly emphasized.
Downloads
If you want to store the results for your own local perusal, downloads are available:
- The plots and tables as separate SVG and HTML files:
- The plain data preprocessed for easier analysis. This doesn’t include the necessities for memory-versus-time plots:
- The full, original, raw data as it came out from
runallruns.sh.
Note on misbehaviour
To start with, I note a few problems that skew the results a bit:
- PyFunge has a bug causing it to mishandle a
;marker at the start of the file. This, in turn, has prevented it from running all of the*-p.b98benchmarks. - CCBI 1.0.20 can’t handle
fork.b98past 4096 threads at all, crashing in some way.
Summaries
First a few cactus plots: the vertical axis displays the maximum amount of resources (time or memory) used to solve the corresponding number of benchmarks. For example, if at x=50 CCBI is at y=600 seconds, then given 600 seconds of time, CCBI could solve any (not all) of those fifty benchmarks.
All time measurements are in seconds (s) and all memory measurements are in mebioctets (Mio).
Unfortunately some interpreters are so fast that one can’t easily see what’s going on in the bottom right corner of the time plot. Here’s a zoomed in version:
Lastly, a table. “Ran” refers to the number of benchmark sets which were either successfully completed or terminated due to timeout: this varies only due to the issues noted earlier.
| Interpreter | Ran | Total time | Total memory | Time ratio | Memory ratio | Maximum time | Maximum memory | Time ratio | Memory ratio |
|---|---|---|---|---|---|---|---|---|---|
| ccbi2–32 | 74 | 234.2 | 4810.3 | 1.00 | 1.00 | 51.1 | 904.5 | 1.01 | 1.00 |
| ccbi2–64 | 74 | 247.7 | 8249.2 | 1.06 | 1.71 | 50.7 | 1807.5 | 1.00 | 2.00 |
| cfunge–32 | 74 | 3628.5 | 42323.5 | 15.49 | 8.80 | 2318.5 | 4904.8 | 45.71 | 5.42 |
| cfunge–64 | 74 | 3834.1 | 48426.0 | 16.37 | 10.07 | 2403.4 | 5787.0 | 47.39 | 6.40 |
| stinkhorn | 74 | 12888.2 | 40035.0 | 55.03 | 8.32 | timeout | 4920.2 | timeout | 5.44 |
| rcfunge2 | 74 | 153968.0 | 6554.2 | 657.46 | 1.36 | timeout | 1035.8 | timeout | 1.15 |
| rcfunge1 | 74 | 154831.0 | 6540.7 | 661.15 | 1.36 | timeout | 1040.6 | timeout | 1.15 |
| language-befunge | 74 | 173077.0 | 42468.2 | 739.06 | 8.83 | timeout | 5476.8 | timeout | 6.05 |
| ccbi1 | 69 | 40190.8 | 54413.8 | 171.62 | 11.31 | timeout | 5174.1 | timeout | 5.72 |
| pyfunge | 42 | 10281.2 | 27918.1 | 43.90 | 5.80 | 4786.7 | 5197.1 | 94.38 | 5.75 |
Details
For each benchmark set, there are two line plots: the other plots time against the problem size (whose meaning is benchmark-specific: see the Fungicide page for that information), and the other plots memory usage against the problem size. In addition to the line plots, there is a summarizing table with some numeric data.
Results for each individual problem size are also displayed: in the form of a line plot of memory usage versus time elapsed, and a table showing the measurements garnered. Note that the “ran” value here depends on the time elapsed for the first run (as described at the Fungicide page) and variations in it are to be expected.
Note that logarithmic axes are used almost throughout!