class to do benchmarking More...

Inheritance diagram for hpp.corbaserver.benchmark.Benchmark:

Collaboration diagram for hpp.corbaserver.benchmark.Benchmark:

Public Member Functions
def	__init__ (self, client, robot, problemSolver)

def	do
	Solve the same problem for the specified cases, for various random seed. More...

def	getCase (self, iter)

def	writeResume (self, filename=None)
	Write data to file. More...

def	tryResumeAndDelete (self, filename=None)
	In case of crash of HPP, the benchmark class writes temporary datas to a file. More...

def	resumeFrom (self, fname)

def	plotTime (self, axes)

def	plotPathLength (self, axes)

def	writeDatabase (self, nameDatabase, experimentName='default', nameLogFile='temp.log', append=False)
	This method create a database which store the benchmark results. More...

def	__str__ (self)

Public Attributes
	seedRange
	A list of seed to initialize the random generator. More...

	cases
	A list of cases for which benchmarking will be done. More...

	iterPerCase
	Number of times one case is repeated (for one seed). More...

	client

	robot

	ps

	current

	results

Static Public Attributes
	toSeconds = np.array([60*60,60,1,1e-3])
	Used to transform HPP output into seconds. More...

string	crashFile = "/tmp/resume.hpp.corbaserver.benchmark.pickle"
	The filename of the crash file. More...

Detailed Description

class to do benchmarking

Basic usage

 # Here goes your script to load the problem.
 robot = hpp.corbaserver.robot.Robot (...)
 ps = hpp.corbaserver.problem_solver.ProblemSolver (...)
 ...
 
 from hpp.corbaserver import Benchmark
 benchmark = Benchmark (robot.client, robot, ps)
 benchmark.iterPerCase = 10
 results = benchmark.do()

See also: hpp.corbaserver.benchmark.Benchmark.do hpp.corbaserver.benchmark.Benchmark.seedRange

If you wish to replot datas stored in a file:

 from hpp.corbaserver import Benchmark
 import matplotlib.pyplot as plt
 b = Benchmark (None, None, None)
 b.resumeFrom ("datafile")
 
 fig, axes = plt.subplots(nrows=1, ncols=2)
 b.plotTime (axes[0])
 b.plotPathLength (axes[1])
 plt.show ()

See also: hpp.corbaserver.benchmark.Benchmark.plotTime hpp.corbaserver.benchmark.Benchmark.plotPathLength

Advanced usage

 from hpp.corbaserver import Benchmark
 b = Benchmark (robot.client, robot, ps)
 b.seedRange = range (10)
 b.iterPerCase = 10
 
 b.cases = list()
 for type in ["Progressive", "Global"]:
     for param in [0.1, 0.2]:
         b.cases.append((type,param))
 b.cases.append(("None",0.1))
 
 b.tryResumeAndDelete ()
 
 def initialize (bench, case, iter):
     bench.ps.selectPathProjector (case[0], case[1])
 
 results = b.do(initCase = initialize)

See also: hpp.corbaserver.benchmark.Benchmark.cases hpp.corbaserver.benchmark.Benchmark.iterPerCase hpp.corbaserver.benchmark.Benchmark.tryResumeAndDelete

What if HPP crashes

Note: This section assumes you have installed https://github.com/humanoid-path-planner/hpp-tools

You can do the following

 try:
   b.do ()
 except:
   import sys
   sys.exit(1)
 
 b.writeResume (filename = "yourresults")

Then, launch your server with this:

1 hppautorestart hppcorbaserver

Finally, launch your script with this:

1 hpp_run_benchmark path_to_python_script_file.py

This will restart the server whenever it crashes and will resume the benchmarks where it stopped.

Constructor & Destructor Documentation

def hpp.corbaserver.benchmark.Benchmark.__init__	(	self,
		client,
		robot,
		problemSolver
	)

Member Function Documentation

def hpp.corbaserver.benchmark.Benchmark.__str__ ( self )

References hpp.corbaserver.benchmark.Benchmark._printStats(), hpp.corbaserver.benchmark.Benchmark.results, and hpp.corbaserver.benchmark.Benchmark.toSeconds.

def hpp.corbaserver.benchmark.Benchmark.do	(	self,
		initCase = `lambda this`,
		case,
		iter
	)

Solve the same problem for the specified cases, for various random seed.

Parameters

initCase

a function of 3 arguments:

the calling Benchmark instance
the current element of the list of Benchmark.cases
the current _BenchmarkIterator (Normally not useful).

References hpp.corbaserver.benchmark.Benchmark.cases, hpp.corbaserver.benchmark.Benchmark.current, hpp.corbaserver.benchmark.Benchmark.getCase(), hpp.corbaserver.benchmark.Benchmark.iterPerCase, hpp.corbaserver.benchmark.Benchmark.results, hpp.corbaserver.benchmark.Benchmark.seedRange, hpp.corbaserver.benchmark.Benchmark.toSeconds, and hpp.corbaserver.benchmark.Benchmark.writeResume().

def hpp.corbaserver.benchmark.Benchmark.getCase	(	self,
		iter
	)

References hpp.corbaserver.benchmark.Benchmark.cases.

Referenced by hpp.corbaserver.benchmark.Benchmark.do().

def hpp.corbaserver.benchmark.Benchmark.plotPathLength	(	self,
		axes
	)

References hpp.corbaserver.benchmark.Benchmark._boxplot(), hpp.corbaserver.benchmark.Benchmark.cases, hpp.corbaserver.benchmark.Benchmark.iterPerCase, hpp.corbaserver.benchmark.Benchmark.results, and hpp.corbaserver.benchmark.Benchmark.seedRange.

def hpp.corbaserver.benchmark.Benchmark.plotTime	(	self,
		axes
	)

References hpp.corbaserver.benchmark.Benchmark._boxplot(), hpp.corbaserver.benchmark.Benchmark.cases, hpp.corbaserver.benchmark.Benchmark.iterPerCase, hpp.corbaserver.benchmark.Benchmark.results, hpp.corbaserver.benchmark.Benchmark.seedRange, and hpp.corbaserver.benchmark.Benchmark.toSeconds.

def hpp.corbaserver.benchmark.Benchmark.resumeFrom	(	self,
		fname
	)

References hpp.corbaserver.benchmark.Benchmark.cases, hpp.corbaserver.benchmark.Benchmark.current, hpp.corbaserver.benchmark.Benchmark.iterPerCase, hpp.corbaserver.benchmark.Benchmark.results, and hpp.corbaserver.benchmark.Benchmark.seedRange.

Referenced by hpp.corbaserver.benchmark.Benchmark.tryResumeAndDelete().

def hpp.corbaserver.benchmark.Benchmark.tryResumeAndDelete	(	self,
		filename = `None`
	)

In case of crash of HPP, the benchmark class writes temporary datas to a file.

The method will check if the crash file exists and:

if it exists, the benchmarking will be initialized at the state before the crash. No datas are lost. The crash file is deleted after having been loaded.
if it does not exist, the method does nothing.
Parameters

filename if None, it uses member Benchmark.crashFile

References hpp.corbaserver.benchmark.Benchmark.crashFile, and hpp.corbaserver.benchmark.Benchmark.resumeFrom().

def hpp.corbaserver.benchmark.Benchmark.writeDatabase	(	self,
		nameDatabase,
		experimentName = `'default'`,
		nameLogFile = `'temp.log'`,
		append = `False`
	)

This method create a database which store the benchmark results.

You can then use it in the platform http://plannerarena.org/ to plot your results.

Parameters

nameDatabase	the name of the created file (extension must be in .db)
experimentName	the name of the current scenario/problem (used when you append several scenario in the same database)
nameLogFile	the name of the text file writed
append	if True, the current result will be added in the given database.

References hpp.corbaserver.benchmark.Benchmark.cases, hpp.corbaserver.benchmark.Benchmark.iterPerCase, hpp.corbaserver.benchmark.Benchmark.results, hpp.corbaserver.benchmark.Benchmark.seedRange, and hpp.corbaserver.benchmark.Benchmark.toSeconds.