rlberry.manager.compare_agents¶
- rlberry.manager.compare_agents(agent_source, method='tukey_hsd', eval_function=None, n_simulations=50, alpha=0.05, B=10000, seed=None)[source]¶
Compare several trained agents using the mean over n_simulations evaluations for each agent.
- Parameters:
- agent_sourcelist of
ExperimentManager, list of str If list of ExperimentManager, load data from it (the agents must be fitted).
If str, each string must be the path of a agent_manager.obj.
If pandas DataFrame with column agent (containing agent’s names) and mean_eval containing the scores, it is used as input data.
- method: str in {“tukey_hsd”, “permutation”}, default=”tukey_hsd”
Method used in the test. “tukey_hsd” use scipy implementation [1] and “permutation” use permutation test with Step-Down method for multiple testing [2]. Tukey HSD method suppose Gaussian model on the aggregated evaluations and permutation test is non-parametric and does not make assumption on the distribution. permutation is the safe choice when the reward is likely to be heavy-tailed or multimodal.
- eval_function: callable or None, default = None
Function used the evaluate the agents. lambda manager : ExperimentManager, eval_budget : int or None, agent_id: int -> eval:float If None, the mean of the eval function of the agent is used over n_simulations evaluations.
- n_simulations: int, default = 50
Number of evaluations to use if eval_function is None.
- alpha: float, default = 0.05
Level of the test, control the Family-wise error.
- B: int, default = 10_000
Number of random permutations used to approximate the permutation test if method = “permutation”
- seed: int or None,
The seed of the random number generator from which we sample permutations. If None, create one.
- agent_sourcelist of
- Returns:
- results: a DataFrame summarising the results.
References
[1]: https://scipy.github.io/devdocs/reference/generated/scipy.stats.tukey_hsd.html
[2]: Testing Statistical Hypotheses by E. L. Lehmann, Joseph P. Romano (Section 15.4.4), https://doi.org/10.1007/0-387-27605-X, Springer
