rlberry.manager.evaluate_agents

rlberry.manager.evaluate_agents(experiment_manager_list, n_simulations=5, choose_random_agents=True, fignum=None, show=True, plot=True)

Evaluate and compare each of the agents in experiment_manager_list.

Parameters:

experiment_manager_listlist of ExperimentManager objects.

n_simulations: int

Number of calls to the eval() method of each ExperimentManager instance.

choose_random_agents: bool

If true and n_fit>1, use a random fitted agent from each ExperimentManager at each evaluation. Otherwise, each fitted agent of each ExperimentManager is evaluated n_simulations times.

fignum: string or int

Identifier of plot figure.

show: bool

If true, calls plt.show().

plot: bool

If false, do not plot.

Returns:

dataframe with the evaluation results.

Examples

>>> from rlberry.agents.torch import A2CAgent, DQNAgent
>>> from rlberry.manager import ExperimentManager, evaluate_agents
>>> from rlberry.envs import gym_make
>>>
>>> if __name__=="__main__":
>>>     managers = [ ExperimentManager(
>>>         agent_class,
>>>         (gym_make, dict(id="CartPole-v1")),
>>>         fit_budget=1e4,
>>>         eval_kwargs=dict(eval_horizon=500),
>>>         n_fit=1,
>>>         parallelization="process",
>>>         mp_context="spawn",
>>>         seed=42,
>>>          ) for agent_class in [A2CAgent, DQNAgent]]
>>>     for manager in managers:
>>>         manager.fit()
>>>     data = evaluate_agents(managers, n_simulations=50, plot=False)