BenchKit

Do you want to evaluate time and memory consumption of any software? BenchKit is made for you. This is a tool that allows you to evaluate these data. It measures:

  • the time to run your software (the “user experience” point of view),
  • the CPU average consumption of your software,
  • the maximum memory used by your software,
  • the evolution of time and CPU over the execution of your software (as a percentage of what is allocated).

BenchKit is of particular interest when you want to evaluate these measures on numerous inputs for a given program. You also confine the software execution up to a maximal execution time or memory usage.

Compared to other solutions such as memtime, an efficient solution developed by the UPPAAL community, BenchKit is able to evaluate memory and CPU consumption of multi-processes (or multi-threaded) applications.

Example of Outputs

BenchKit provides a summary of the execution in a CSV file as shown below:

tool,test 1,ltl,2.68151048738,17.5,7.23868918419,normal,135055571400033_n_1
tool,test 2,ltl,2.55138092947,16.1,6.51366186142,timeout,135055571400035_n_1

Columns indicate:

  • the name of the tool (in order to merge these files later),
  • the name of the test,
  • the name of the examination performed on the test,
  • the maximum amount of memory used during the execution (percentage),
  • the average percentage of CPU used during the execution,
  • the time required to execute your program,
  • the status of the termination (normal or timeout when time confinement is reached),
  • a unique run identifier that allows you to access additional data.

BenchKit also samples memory usage and CPU over the execution of a tool in a CSV file as shown below:

1350560133.4,2.18068239019,13.6
1350560133.98,2.18020601597,13.5
1350560134.57,2.23562421696,13.6
1350560135.17,2.17822112339,13.9
1350560135.76,2.18362003122,14.5
1350560136.35,2.66309068418,15.7
1350560136.91,2.72247867034,16.8
1350560137.5,2.81846807578,17.9

Columns indicate:

  • time (seconds since 1970) of sampling,
  • CPU sampled at this time (percentage),
  • memory sampled at this time (percentage).

From such a sampled file, one can generate the chart shown below (this one is selected from the MCC’2015).

How to use BenchKit

Current version

BenchKit is distributed under the GPL and available on GitHub. We are sorry but documentation is ongoing.

Personal BenchKit (old version)

BenchKit is distributed under the GPL. You can use it freely to operate and evaluate your tool. We provide the following elements:

Version History

  • 1.1 (May 2, 2015): third version to be published in the context of the Model Checking Contest @ Petri Nets 2015; it now allows to affect several cores to a given virtual machine,
  • 1.0 (May 1, 2014): second version to be published in the context of the Model Checking Contest @ Petri Nets 2014 ; this version supports several virtual machines per host and is suitable for machines with a large number of cores,
  • β1 (Feb. 5, 2013): First version to be published (only one virtual machine per remote host) in the context of the Model Checking Contest @ Petri Nets 2013,
  • β2 (Apr. 14, 2013): Minor corrections and improvement for more stability, in particular with the use of top in the VM to extract memory and CPU information (still only one virtual machine per remote host). Once again in the context of the Model Checking Contest @ Petri Nets 2013.

Citation

If you use BenchKit please cite:

@inproceedings{benchkit:2013,
Author = {F. Kordon and F. Hulin-Hubard},
booktitle = {14th International Conference on Application of Concurrency to System Design, {ACSD} 2014,
Tunis La Marsa, Tunisia, June 23-27, 2014},
Title = {{BenchKit, a Tool for Massive Concurrent Benchmarking}},
pages = {159--165},
url = {http://dx.doi.org/10.1109/ACSD.2014.12},
year = {2014}}