CUI LSV LIPN LIP6
MeFoSyLoMa CNRS Iniria
QR Code
Last modified
June 28, 2016

Displayed
Aug 18, 2017
BenchKit Home Page

Introduction

Do you want to evaluate time and memory consumption of any software? BenchKit is made for you. This is a tool that allows you to evaluate these data. It measures:

BenchKit is of particular interest when you want to evaluate these on numerous inputs for a given program. You also confine the software execution up to a maximal execution time or memory usage.

Compared to other solutions such as memtime, an efficient solution developed by the UPPAAL community, BenchKit is able to evaluate memory and CPU consumption of multi-processes (or multi-threaded) applications.

Example of Outputs

BenchKit provides a summary of the execution in a CSV file as shown below:

tool,test 1,ltl,2.68151048738,17.5,7.23868918419,normal,135055571400033_n_1
tool,test 2,ltl,2.55138092947,16.1,6.51366186142,timeout,135055571400035_n_1

Columns indicate : the name of the tool (in order to merge these files later), the name of the test, the name of the examination performed on the test, the maximum amount of memory used during the execution (percentage), the average percentage of CPU used during the execution, the time required to execute your program, the status of the termination (normal or timeout when time confinement is reached), a unique run identifier that allows you to access additional data.

BenchKit also samples memory usage and CPU over the execution of a tool in a CSV file as shown below:

1350560133.4,2.18068239019,13.6
1350560133.98,2.18020601597,13.5
1350560134.57,2.23562421696,13.6
1350560135.17,2.17822112339,13.9
1350560135.76,2.18362003122,14.5
1350560136.35,2.66309068418,15.7
1350560136.91,2.72247867034,16.8
1350560137.5,2.81846807578,17.9

Columns indicate : time (seconds since 1970) of sampling, CPU sampled at this time (percentage), memory sampled at this time (percentage). From such a sampled file, one can generate the chart shown below (this one is selected from the MCC'2015).

Example of chart

How to use BenchKit

Current version

BenchKit is distributed under the GPL and available on GitHub. We are sorry but documentation is ongoing.

Personal BenchKit (old version)

BenchKit is distributed under the GPL. You can use it freely to operate and evaluate your tool. We provide the following elements:

Version History

Citation

If you use BenchKit please cite:

@inproceedings{benchkit:2013,
  Author = {F. Kordon and F. Hulin-Hubard},
  booktitle = {14th International Conference on Application of Concurrency
    to System Design, {ACSD} 2014, Tunis La Marsa, Tunisia, June 23-27, 2014},
  Title = {{BenchKit, a Tool for Massive Concurrent Benchmarking}},
  pages = {159--165},   url = {http://dx.doi.org/10.1109/ACSD.2014.12},   year = {2014}}

background BenchKit