rlberry.manager.compare_agents

rlberry.manager.compare_agents(agent_source, method='tukey_hsd', eval_function=None, n_simulations=50, alpha=0.05, B=10000, seed=None)

Compare several trained agents using the mean over n_simulations evaluations for each agent.

Parameters:

agent_sourcelist of ExperimentManager, list of str

• If list of ExperimentManager, load data from it (the agents must be fitted).

• If str, each string must be the path of a agent_manager.obj.

• If pandas DataFrame with column agent (containing agent’s names) and mean_eval containing the scores, it is used as input data.

method: str in {“tukey_hsd”, “permutation”}, default=”tukey_hsd”

Method used in the test. “tukey_hsd” use scipy implementation [1] and “permutation” use permutation test with Step-Down method for multiple testing [2]. Tukey HSD method suppose Gaussian model on the aggregated evaluations and permutation test is non-parametric and does not make assumption on the distribution. permutation is the safe choice when the reward is likely to be heavy-tailed or multimodal.

eval_function: callable or None, default = None

Function used the evaluate the agents. lambda manager : ExperimentManager, eval_budget : int or None, agent_id: int -> eval:float If None, the mean of the eval function of the agent is used over n_simulations evaluations.

n_simulations: int, default = 50

Number of evaluations to use if eval_function is None.

alpha: float, default = 0.05

Level of the test, control the Family-wise error.

B: int, default = 10_000

Number of random permutations used to approximate the permutation test if method = “permutation”

seed: int or None,

The seed of the random number generator from which we sample permutations. If None, create one.

Returns:

results: a DataFrame summarising the results.

References

[1]: https://scipy.github.io/devdocs/reference/generated/scipy.stats.tukey_hsd.html

[2]: Testing Statistical Hypotheses by E. L. Lehmann, Joseph P. Romano (Section 15.4.4), https://doi.org/10.1007/0-387-27605-X, Springer