baselines算法库run.py模块分析

baselines算法库地址：

https://gitee.com/devilmaycry812839668/baselines

========================================

对baselines算法库中 run.py模块代码分析：

记录gym所有的游戏环境：

_game_envs = defaultdict(set)
for env in gym.envs.registry.all():
    # TODO: solve this with regexes
    env_type = env.entry_point.split(':')[0].split('.')[-1]
    _game_envs[env_type].add(env.id)

测试：

打印结果：

algorithmic {'ReversedAddition-v0', 'RepeatCopy-v0', 'DuplicatedInput-v0', 'ReversedAddition3-v0', 'Copy-v0', 'Reverse-v0'}
classic_control {'CartPole-v0', 'CartPole-v1', 'Acrobot-v1', 'Pendulum-v0', 'MountainCarContinuous-v0', 'MountainCar-v0'}
box2d {'BipedalWalker-v3', 'LunarLander-v2', 'LunarLanderContinuous-v2', 'CarRacing-v0', 'BipedalWalkerHardcore-v3'}
toy_text {'KellyCoinflipGeneralized-v0', 'KellyCoinflip-v0', 'GuessingGame-v0', 'Taxi-v3', 'CliffWalking-v0', 'NChain-v0', 'FrozenLake-v0', 'FrozenLake8x8-v0', 'HotterColder-v0', 'Roulette-v0', 'Blackjack-v0'}
mujoco {'Ant-v2', 'Hopper-v2', 'Reacher-v2', 'InvertedDoublePendulum-v2', 'Humanoid-v2', 'HumanoidStandup-v2', 'Walker2d-v2', 'Thrower-v2', 'InvertedPendulum-v2', 'HalfCheetah-v2', 'Swimmer-v2', 'Striker-v2', 'Pusher-v2'}
half_cheetah_v3 {'HalfCheetah-v3'}
hopper_v3 {'Hopper-v3'}
swimmer_v3 {'Swimmer-v3'}
walker2d_v3 {'Walker2d-v3'}
ant_v3 {'Ant-v3'}
humanoid_v3 {'Humanoid-v3'}
robotics {'HandManipulateBlockRotateZTouchSensorsDense-v1', 'FetchReach-v1', 'FetchPush-v1', 'HandManipulateBlockTouchSensors-v0', 'HandManipulatePenFull-v0', 'FetchPickAndPlace-v1', 'HandManipulateBlockTouchSensors-v1', 'HandManipulateBlockRotateParallelTouchSensorsDense-v1', 'HandManipulateBlockRotateParallel-v0', 'HandManipulateBlockRotateXYZ-v0', 'FetchSlideDense-v1', 'HandManipulatePenRotateTouchSensors-v1', 'HandManipulateBlockFullDense-v0', 'HandManipulateEggFullDense-v0', 'HandManipulateBlockRotateZTouchSensors-v0', 'HandManipulatePenDense-v0', 'HandManipulateEggDense-v0', 'HandManipulatePenRotateTouchSensors-v0', 'HandReachDense-v0', 'HandManipulateBlockRotateZTouchSensorsDense-v0', 'HandManipulateEggTouchSensors-v1', 'FetchReachDense-v1', 'HandManipulateEggRotateTouchSensorsDense-v1', 'HandManipulatePenRotateTouchSensorsDense-v1', 'HandManipulateEggRotate-v0', 'HandManipulateEggRotateDense-v0', 'HandManipulateBlockRotateXYZTouchSensors-v0', 'HandManipulateBlockTouchSensorsDense-v1', 'HandManipulateEggTouchSensorsDense-v0', 'HandManipulateBlockRotateXYZDense-v0', 'HandManipulateBlock-v0', 'HandReach-v0', 'HandManipulateBlockRotateParallelDense-v0', 'HandManipulateBlockRotateParallelTouchSensors-v0', 'HandManipulateBlockRotateZTouchSensors-v1', 'HandManipulateEggFull-v0', 'HandManipulatePen-v0', 'HandManipulateEggRotateTouchSensors-v1', 'HandManipulateBlockRotateZDense-v0', 'HandManipulateBlockRotateXYZTouchSensorsDense-v1', 'HandManipulatePenRotateDense-v0', 'HandManipulatePenRotate-v0', 'HandManipulateBlockRotateParallelTouchSensorsDense-v0', 'HandManipulatePenTouchSensorsDense-v0', 'HandManipulatePenTouchSensors-v1', 'HandManipulateEggTouchSensorsDense-v1', 'HandManipulatePenTouchSensorsDense-v1', 'HandManipulateBlockRotateXYZTouchSensors-v1', 'HandManipulateBlockRotateParallelTouchSensors-v1', 'HandManipulatePenFullDense-v0', 'HandManipulateEggTouchSensors-v0', 'HandManipulateEgg-v0', 'HandManipulateBlockRotateZ-v0', 'HandManipulateBlockTouchSensorsDense-v0', 'FetchSlide-v1', 'HandManipulatePenRotateTouchSensorsDense-v0', 'FetchPushDense-v1', 'HandManipulateBlockFull-v0', 'HandManipulateBlockDense-v0', 'HandManipulateEggRotateTouchSensors-v0', 'HandManipulateEggRotateTouchSensorsDense-v0', 'FetchPickAndPlaceDense-v1', 'HandManipulatePenTouchSensors-v0', 'HandManipulateBlockRotateXYZTouchSensorsDense-v0'}
atari {'ElevatorAction-ramDeterministic-v0', 'RoadRunner-ramDeterministic-v0', 'HeroDeterministic-v4', 'Bowling-ramNoFrameskip-v4', 'Bowling-ramDeterministic-v0', 'Assault-ramDeterministic-v0', 'MsPacman-v4', 'MsPacmanDeterministic-v0', 'BowlingNoFrameskip-v4', 'Atlantis-ram-v4', 'Boxing-v4', 'ChopperCommand-ramDeterministic-v4', 'NameThisGame-ramDeterministic-v4', 'JourneyEscapeNoFrameskip-v4', 'Solaris-ramDeterministic-v4', 'CrazyClimber-ramNoFrameskip-v4', 'NameThisGame-ramNoFrameskip-v4', 'QbertNoFrameskip-v4', 'Gravitar-v4', 'ZaxxonNoFrameskip-v4', 'DoubleDunkDeterministic-v0', 'KangarooDeterministic-v4', 'Pitfall-ram-v0', 'BattleZoneDeterministic-v4', 'Amidar-v4', 'Enduro-ramDeterministic-v0', 'StarGunner-ram-v0', 'BankHeistDeterministic-v4', 'BattleZone-ramDeterministic-v0', 'ChopperCommand-ram-v0', 'Jamesbond-ramNoFrameskip-v0', 'DoubleDunk-v4', 'Pong-v0', 'Qbert-v0', 'Tutankham-ramNoFrameskip-v0', 'BattleZone-ramNoFrameskip-v0', 'Defender-ramDeterministic-v4', 'Kangaroo-ram-v0', 'MontezumaRevengeNoFrameskip-v4', 'Frostbite-ramNoFrameskip-v4', 'Hero-ramDeterministic-v0', 'QbertDeterministic-v4', 'EnduroDeterministic-v0', 'RobotankDeterministic-v4', 'SolarisDeterministic-v4', 'Enduro-ramDeterministic-v4', 'Asterix-ram-v4', 'Asterix-ramDeterministic-v4', 'CrazyClimberDeterministic-v4', 'KrullNoFrameskip-v4', 'Freeway-ram-v4', 'JourneyEscape-v4', 'Asterix-ramNoFrameskip-v0', 'DemonAttack-v0', 'Boxing-ram-v4', 'SpaceInvaders-ram-v4', 'Kangaroo-v0', 'Solaris-v4', 'Breakout-ram-v4', 'KungFuMasterDeterministic-v4', 'CrazyClimber-ramNoFrameskip-v0', 'IceHockey-ramDeterministic-v0', 'Adventure-ramNoFrameskip-v4', 'HeroNoFrameskip-v0', 'PrivateEyeNoFrameskip-v0', 'SpaceInvaders-ramDeterministic-v4', 'Freeway-ramNoFrameskip-v0', 'AirRaid-ram-v4', 'Zaxxon-v0', 'Riverraid-v0', 'Robotank-ramNoFrameskip-v4', 'AmidarDeterministic-v0', 'WizardOfWor-ram-v4', 'VideoPinball-v4', 'BeamRider-ramNoFrameskip-v0', 'DefenderDeterministic-v0', 'Centipede-ram-v0', 'AirRaidNoFrameskip-v4', 'BankHeist-ramDeterministic-v4', 'JourneyEscape-ramDeterministic-v4', 'TimePilot-ramNoFrameskip-v4', 'MsPacman-ramDeterministic-v4', 'Skiing-v0', 'UpNDownDeterministic-v4', 'Pong-ramNoFrameskip-v0', 'Centipede-ramNoFrameskip-v0', 'AirRaidDeterministic-v0', 'KungFuMaster-ramNoFrameskip-v0', 'PrivateEye-ramNoFrameskip-v4', 'AssaultNoFrameskip-v4', 'WizardOfWor-ramNoFrameskip-v0', 'BeamRiderDeterministic-v4', 'PhoenixDeterministic-v0', 'PongDeterministic-v4', 'Jamesbond-v0', 'VideoPinball-ram-v0', 'RoadRunner-ram-v0', 'GopherNoFrameskip-v0', 'Assault-v4', 'Defender-ram-v4', 'UpNDown-ram-v0', 'MontezumaRevenge-ram-v0', 'Defender-ram-v0', 'BankHeist-ramNoFrameskip-v4', 'DoubleDunk-ramNoFrameskip-v0', 'VentureDeterministic-v0', 'MontezumaRevenge-ramDeterministic-v4', 'Adventure-ramDeterministic-v0', 'MsPacman-ramNoFrameskip-v0', 'Krull-ram-v4', 'EnduroNoFrameskip-v4', 'JamesbondNoFrameskip-v0', 'Atlantis-ramNoFrameskip-v4', 'ElevatorAction-ramNoFrameskip-v4', 'DemonAttackNoFrameskip-v4', 'AdventureNoFrameskip-v4', 'Carnival-ramNoFrameskip-v4', 'PitfallDeterministic-v4', 'TimePilot-v4', 'Solaris-ramNoFrameskip-v0', 'PitfallDeterministic-v0', 'DoubleDunkNoFrameskip-v0', 'UpNDown-ramDeterministic-v0', 'JourneyEscapeNoFrameskip-v0', 'Asteroids-v0', 'NameThisGame-v0', 'BattleZone-v0', 'Tennis-v0', 'ChopperCommandNoFrameskip-v4', 'YarsRevenge-ram-v4', 'Hero-v4', 'PongNoFrameskip-v4', 'NameThisGame-ram-v0', 'Assault-ramNoFrameskip-v4', 'RiverraidDeterministic-v4', 'ZaxxonDeterministic-v4', 'VentureNoFrameskip-v0', 'SpaceInvadersNoFrameskip-v4', 'Tutankham-ram-v0', 'BreakoutNoFrameskip-v0', 'AirRaid-ramDeterministic-v4', 'AsteroidsDeterministic-v0', 'BankHeistNoFrameskip-v4', 'Bowling-ramNoFrameskip-v0', 'Alien-ramDeterministic-v4', 'Alien-ramNoFrameskip-v0', 'IceHockey-v0', 'AirRaid-v0', 'SeaquestNoFrameskip-v4', 'Atlantis-ramDeterministic-v0', 'SpaceInvaders-ram-v0', 'Jamesbond-v4', 'Kangaroo-ramNoFrameskip-v0', 'AlienNoFrameskip-v4', 'Phoenix-ram-v0', 'GravitarNoFrameskip-v0', 'PitfallNoFrameskip-v4', 'DoubleDunk-ram-v4', 'StarGunner-v0', 'CrazyClimber-ramDeterministic-v0', 'Robotank-ram-v0', 'Breakout-ramNoFrameskip-v0', 'AsterixNoFrameskip-v0', 'AtlantisNoFrameskip-v4', 'Boxing-ramNoFrameskip-v0', 'GopherDeterministic-v4', 'Gravitar-ram-v0', 'YarsRevenge-ramDeterministic-v0', 'KungFuMaster-ramDeterministic-v0', 'VideoPinballDeterministic-v4', 'Asteroids-ramNoFrameskip-v0', 'PhoenixNoFrameskip-v0', 'Jamesbond-ramNoFrameskip-v4', 'RiverraidNoFrameskip-v0', 'Pong-ramDeterministic-v4', 'TimePilot-ramDeterministic-v0', 'IceHockeyDeterministic-v4', 'Freeway-ram-v0', 'Adventure-ram-v4', 'FishingDerby-v4', 'Hero-ramNoFrameskip-v0', 'Seaquest-ramNoFrameskip-v4', 'SpaceInvadersDeterministic-v0', 'WizardOfWor-ram-v0', 'SpaceInvadersNoFrameskip-v0', 'Zaxxon-ram-v4', 'SpaceInvaders-v4', 'FreewayNoFrameskip-v0', 'BoxingNoFrameskip-v4', 'TimePilot-ram-v0', 'MontezumaRevengeNoFrameskip-v0', 'BreakoutDeterministic-v4', 'Defender-ramNoFrameskip-v4', 'TutankhamDeterministic-v0', 'Venture-ram-v0', 'Pong-v4', 'Robotank-v4', 'Atlantis-v4', 'Skiing-ram-v4', 'MsPacman-ramNoFrameskip-v4', 'AssaultNoFrameskip-v0', 'NameThisGame-ramDeterministic-v0', 'BattleZone-ramNoFrameskip-v4', 'Krull-ramDeterministic-v4', 'Breakout-ram-v0', 'MontezumaRevenge-ramNoFrameskip-v0', 'AssaultDeterministic-v0', 'Qbert-ram-v0', 'Seaquest-ram-v4', 'Seaquest-v4', 'Solaris-ram-v4', 'Solaris-ram-v0', 'RoadRunnerDeterministic-v4', 'CarnivalDeterministic-v4', 'Hero-ramDeterministic-v4', 'Amidar-ramNoFrameskip-v0', 'YarsRevenge-ramNoFrameskip-v4', 'GopherNoFrameskip-v4', 'FishingDerbyNoFrameskip-v0', 'AirRaidDeterministic-v4', 'Asterix-v4', 'MsPacmanNoFrameskip-v4', 'Bowling-v0', 'KungFuMaster-ram-v4', 'GopherDeterministic-v0', 'AirRaidNoFrameskip-v0', 'BankHeist-ram-v0', 'SkiingDeterministic-v0', 'StarGunnerDeterministic-v0', 'FishingDerby-v0', 'StarGunner-ramDeterministic-v0', 'RobotankNoFrameskip-v4', 'KangarooNoFrameskip-v0', 'TutankhamNoFrameskip-v0', 'Robotank-ramNoFrameskip-v0', 'AirRaid-ram-v0', 'Assault-ramDeterministic-v4', 'FishingDerby-ramNoFrameskip-v4', 'RoadRunner-v0', 'Krull-ram-v0', 'Pooyan-ramDeterministic-v0', 'Pitfall-ramNoFrameskip-v0', 'UpNDownDeterministic-v0', 'AsterixDeterministic-v4', 'Kangaroo-v4', 'SkiingNoFrameskip-v0', 'Gopher-v4', 'CarnivalDeterministic-v0', 'Solaris-ramNoFrameskip-v4', 'VideoPinball-ramNoFrameskip-v0', 'FrostbiteNoFrameskip-v4', 'RoadRunnerNoFrameskip-v4', 'ElevatorAction-ram-v0', 'Defender-ramNoFrameskip-v0', 'Frostbite-ramDeterministic-v4', 'EnduroNoFrameskip-v0', 'Bowling-ram-v0', 'IceHockey-ramNoFrameskip-v0', 'Defender-v0', 'CarnivalNoFrameskip-v4', 'Zaxxon-ramNoFrameskip-v0', 'Frostbite-ramDeterministic-v0', 'ZaxxonDeterministic-v0', 'IceHockey-ramDeterministic-v4', 'AsteroidsNoFrameskip-v0', 'Phoenix-v0', 'DoubleDunk-v0', 'Gravitar-ramNoFrameskip-v4', 'NameThisGameDeterministic-v0', 'BeamRiderNoFrameskip-v0', 'JamesbondDeterministic-v0', 'BattleZoneNoFrameskip-v4', 'YarsRevenge-ramDeterministic-v4', 'Breakout-v4', 'Adventure-ramDeterministic-v4', 'FrostbiteNoFrameskip-v0', 'Freeway-v4', 'ElevatorActionDeterministic-v0', 'BankHeist-v4', 'Frostbite-ram-v0', 'PongDeterministic-v0', 'AirRaid-v4', 'GravitarDeterministic-v4', 'UpNDown-ramNoFrameskip-v4', 'DefenderNoFrameskip-v4', 'Pitfall-ramDeterministic-v4', 'IceHockeyNoFrameskip-v0', 'JourneyEscape-ramNoFrameskip-v0', 'Pitfall-v4', 'Qbert-ram-v4', 'YarsRevenge-ram-v0', 'BeamRider-ramNoFrameskip-v4', 'BeamRider-ram-v0', 'PhoenixNoFrameskip-v4', 'KrullNoFrameskip-v0', 'Seaquest-v0', 'AmidarDeterministic-v4', 'SpaceInvaders-v0', 'SolarisNoFrameskip-v0', 'Venture-v4', 'Pong-ramNoFrameskip-v4', 'Enduro-v4', 'Pooyan-ram-v4', 'AlienDeterministic-v4', 'MsPacman-ram-v0', 'Phoenix-ram-v4', 'Seaquest-ramDeterministic-v0', 'Amidar-ramDeterministic-v0', 'Bowling-v4', 'SpaceInvadersDeterministic-v4', 'Gopher-ram-v0', 'JourneyEscape-ramNoFrameskip-v4', 'Phoenix-ramNoFrameskip-v4', 'RoadRunner-ramNoFrameskip-v0', 'Boxing-ram-v0', 'DoubleDunk-ramDeterministic-v4', 'JamesbondNoFrameskip-v4', 'Breakout-ramDeterministic-v4', 'Phoenix-v4', 'SeaquestDeterministic-v0', 'QbertDeterministic-v0', 'Krull-v0', 'Jamesbond-ramDeterministic-v0', 'Riverraid-ramNoFrameskip-v0', 'IceHockeyDeterministic-v0', 'Phoenix-ramNoFrameskip-v0', 'PrivateEye-v4', 'Centipede-v0', 'CrazyClimberNoFrameskip-v0', 'Jamesbond-ram-v4', 'SkiingDeterministic-v4', 'Freeway-ramNoFrameskip-v4', 'BattleZone-v4', 'Carnival-ramDeterministic-v4', 'IceHockeyNoFrameskip-v4', 'KrullDeterministic-v0', 'Venture-ramDeterministic-v4', 'Kangaroo-ramDeterministic-v4', 'YarsRevengeNoFrameskip-v4', 'IceHockey-ram-v4', 'DemonAttack-ramNoFrameskip-v0', 'PrivateEye-ram-v4', 'Alien-v0', 'AirRaid-ramNoFrameskip-v4', 'UpNDown-v0', 'SpaceInvaders-ramNoFrameskip-v4', 'VideoPinball-ramDeterministic-v0', 'Frostbite-v4', 'Tutankham-v0', 'SeaquestDeterministic-v4', 'Atlantis-ramDeterministic-v4', 'Gopher-ramDeterministic-v0', 'Hero-ram-v4', 'UpNDown-ramDeterministic-v4', 'NameThisGameNoFrameskip-v4', 'TimePilot-v0', 'FrostbiteDeterministic-v0', 'Amidar-ram-v4', 'ChopperCommandNoFrameskip-v0', 'PitfallNoFrameskip-v0', 'CrazyClimber-v4', 'RoadRunner-ramNoFrameskip-v4', 'BankHeistNoFrameskip-v0', 'Tutankham-v4', 'KungFuMasterNoFrameskip-v4', 'YarsRevengeDeterministic-v0', 'HeroDeterministic-v0', 'AdventureDeterministic-v0', 'Berzerk-ram-v4', 'Centipede-ram-v4', 'Skiing-ramDeterministic-v4', 'VideoPinball-ramNoFrameskip-v4', 'AdventureNoFrameskip-v0', 'DefenderDeterministic-v4', 'Alien-ramNoFrameskip-v4', 'ElevatorAction-v0', 'PrivateEyeDeterministic-v4', 'MontezumaRevenge-ram-v4', 'Asterix-ramNoFrameskip-v4', 'Alien-v4', 'TennisNoFrameskip-v0', 'Atlantis-v0', 'Krull-ramNoFrameskip-v4', 'BerzerkDeterministic-v4', 'Tennis-v4', 'VideoPinballNoFrameskip-v0', 'Zaxxon-ramDeterministic-v0', 'NameThisGameNoFrameskip-v0', 'JourneyEscape-ramDeterministic-v0', 'Pooyan-ramNoFrameskip-v4', 'Pitfall-v0', 'Riverraid-ramDeterministic-v0', 'VideoPinballDeterministic-v0', 'Tutankham-ramDeterministic-v4', 'Phoenix-ramDeterministic-v0', 'Tennis-ramNoFrameskip-v4', 'Hero-ram-v0', 'Frostbite-ram-v4', 'TennisNoFrameskip-v4', 'Carnival-ramDeterministic-v0', 'DemonAttack-ram-v4', 'JourneyEscape-ram-v4', 'UpNDown-v4', 'PooyanNoFrameskip-v4', 'Centipede-ramDeterministic-v4', 'Seaquest-ramDeterministic-v4', 'Pooyan-v4', 'AlienNoFrameskip-v0', 'Pong-ram-v4', 'UpNDownNoFrameskip-v0', 'AsterixNoFrameskip-v4', 'Breakout-ramDeterministic-v0', 'Venture-ramNoFrameskip-v4', 'Pong-ram-v0', 'Adventure-ramNoFrameskip-v0', 'Asteroids-ram-v4', 'Carnival-v0', 'PooyanNoFrameskip-v0', 'PrivateEye-ramNoFrameskip-v0', 'Qbert-v4', 'TutankhamNoFrameskip-v4', 'Skiing-ramNoFrameskip-v0', 'Riverraid-v4', 'VentureNoFrameskip-v4', 'CentipedeNoFrameskip-v4', 'Adventure-v4', 'MsPacmanNoFrameskip-v0', 'TimePilotDeterministic-v0', 'BattleZoneDeterministic-v0', 'BankHeist-ramNoFrameskip-v0', 'FishingDerbyDeterministic-v4', 'Boxing-ramDeterministic-v4', 'Asteroids-ramDeterministic-v0', 'PooyanDeterministic-v4', 'CrazyClimber-ram-v4', 'RoadRunner-v4', 'Amidar-v0', 'DemonAttackDeterministic-v0', 'Riverraid-ram-v0', 'PrivateEye-ramDeterministic-v0', 'RiverraidDeterministic-v0', 'RobotankNoFrameskip-v0', 'Seaquest-ram-v0', 'CarnivalNoFrameskip-v0', 'FishingDerby-ramDeterministic-v0', 'KangarooNoFrameskip-v4', 'FishingDerbyNoFrameskip-v4', 'ElevatorActionDeterministic-v4', 'Enduro-v0', 'AtlantisDeterministic-v0', 'KungFuMasterDeterministic-v0', 'Breakout-ramNoFrameskip-v4', 'PrivateEyeDeterministic-v0', 'BowlingNoFrameskip-v0', 'DoubleDunkNoFrameskip-v4', 'AsteroidsNoFrameskip-v4', 'CrazyClimberDeterministic-v0', 'Assault-ram-v0', 'Venture-ramNoFrameskip-v0', 'Gravitar-v0', 'GravitarNoFrameskip-v4', 'StarGunner-v4', 'KungFuMaster-ramDeterministic-v4', 'Tennis-ram-v4', 'Asterix-ramDeterministic-v0', 'Venture-ramDeterministic-v0', 'Tennis-ramDeterministic-v0', 'SolarisDeterministic-v0', 'VentureDeterministic-v4', 'VideoPinball-ramDeterministic-v4', 'Gravitar-ramDeterministic-v0', 'IceHockey-ramNoFrameskip-v4', 'ChopperCommand-ramDeterministic-v0', 'Asteroids-v4', 'ElevatorActionNoFrameskip-v0', 'BeamRider-ram-v4', 'Qbert-ramDeterministic-v0', 'Robotank-ramDeterministic-v4', 'Hero-ramNoFrameskip-v4', 'Skiing-ramNoFrameskip-v4', 'SkiingNoFrameskip-v4', 'Berzerk-v4', 'ChopperCommand-ramNoFrameskip-v0', 'BoxingDeterministic-v4', 'Venture-ram-v4', 'KungFuMaster-v0', 'PrivateEyeNoFrameskip-v4', 'HeroNoFrameskip-v4', 'Asteroids-ramDeterministic-v4', 'Krull-ramDeterministic-v0', 'GravitarDeterministic-v0', 'Kangaroo-ramDeterministic-v0', 'Gopher-ramNoFrameskip-v0', 'Breakout-v0', 'TimePilotDeterministic-v4', 'Jamesbond-ram-v0', 'Freeway-ramDeterministic-v4', 'JamesbondDeterministic-v4', 'JourneyEscapeDeterministic-v4', 'Gopher-v0', 'Tutankham-ramNoFrameskip-v4', 'KungFuMasterNoFrameskip-v0', 'SpaceInvaders-ramNoFrameskip-v0', 'Enduro-ram-v0', 'YarsRevengeNoFrameskip-v0', 'AirRaid-ramNoFrameskip-v0', 'Tutankham-ram-v4', 'Kangaroo-ram-v4', 'FreewayDeterministic-v4', 'RoadRunnerNoFrameskip-v0', 'RoadRunnerDeterministic-v0', 'CentipedeNoFrameskip-v0', 'AssaultDeterministic-v4', 'WizardOfWorDeterministic-v0', 'AsterixDeterministic-v0', 'SolarisNoFrameskip-v4', 'FishingDerbyDeterministic-v0', 'ChopperCommand-ramNoFrameskip-v4', 'Enduro-ramNoFrameskip-v4', 'Bowling-ramDeterministic-v4', 'YarsRevenge-ramNoFrameskip-v0', 'UpNDownNoFrameskip-v4', 'BankHeistDeterministic-v0', 'Tennis-ramNoFrameskip-v0', 'CrazyClimber-ram-v0', 'EnduroDeterministic-v4', 'TennisDeterministic-v0', 'Riverraid-ram-v4', 'Amidar-ram-v0', 'MsPacman-ram-v4', 'AirRaid-ramDeterministic-v0', 'Adventure-v0', 'Enduro-ramNoFrameskip-v0', 'KangarooDeterministic-v0', 'BreakoutNoFrameskip-v4', 'AtlantisNoFrameskip-v0', 'TimePilot-ramDeterministic-v4', 'StarGunner-ramDeterministic-v4', 'UpNDown-ram-v4', 'Carnival-v4', 'Frostbite-ramNoFrameskip-v0', 'ZaxxonNoFrameskip-v0', 'MontezumaRevengeDeterministic-v4', 'Gopher-ramDeterministic-v4', 'Asterix-v0', 'Riverraid-ramNoFrameskip-v4', 'Hero-v0', 'QbertNoFrameskip-v0', 'Jamesbond-ramDeterministic-v4', 'KungFuMaster-v4', 'Seaquest-ramNoFrameskip-v0', 'ChopperCommandDeterministic-v4', 'StarGunnerNoFrameskip-v4', 'BeamRider-v0', 'BattleZone-ram-v0', 'AsteroidsDeterministic-v4', 'FishingDerby-ram-v4', 'WizardOfWorNoFrameskip-v0', 'Alien-ram-v0', 'AtlantisDeterministic-v4', 'Qbert-ramNoFrameskip-v0', 'AlienDeterministic-v0', 'Kangaroo-ramNoFrameskip-v4', 'DoubleDunkDeterministic-v4', 'WizardOfWor-v4', 'MontezumaRevenge-ramNoFrameskip-v4', 'BerzerkNoFrameskip-v4', 'YarsRevenge-v4', 'DemonAttack-v4', 'NameThisGameDeterministic-v4', 'DoubleDunk-ram-v0', 'Zaxxon-ramDeterministic-v4', 'Robotank-v0', 'Pooyan-ramDeterministic-v4', 'DoubleDunk-ramDeterministic-v0', 'Krull-ramNoFrameskip-v0', 'TimePilot-ram-v4', 'DemonAttackDeterministic-v4', 'Pitfall-ramDeterministic-v0', 'PooyanDeterministic-v0', 'Carnival-ram-v0', 'Pooyan-v0', 'ChopperCommand-ram-v4', 'AmidarNoFrameskip-v0', 'BerzerkNoFrameskip-v0', 'ElevatorAction-ramDeterministic-v4', 'KrullDeterministic-v4', 'TimePilotNoFrameskip-v4', 'Enduro-ram-v4', 'Tennis-ram-v0', 'Carnival-ram-v4', 'ChopperCommand-v4', 'MontezumaRevenge-v4', 'TimePilotNoFrameskip-v0', 'Skiing-ram-v0', 'BankHeist-ram-v4', 'BankHeist-ramDeterministic-v0', 'JourneyEscape-v0', 'Krull-v4', 'Tutankham-ramDeterministic-v0', 'MsPacmanDeterministic-v4', 'Alien-ramDeterministic-v0', 'ChopperCommandDeterministic-v0', 'Pooyan-ramNoFrameskip-v0', 'Bowling-ram-v4', 'Riverraid-ramDeterministic-v4', 'MontezumaRevenge-ramDeterministic-v0', 'PrivateEye-ram-v0', 'ElevatorAction-ram-v4', 'SpaceInvaders-ramDeterministic-v0', 'Pooyan-ram-v0', 'Berzerk-ramNoFrameskip-v4', 'CentipedeDeterministic-v0', 'AdventureDeterministic-v4', 'PhoenixDeterministic-v4', 'YarsRevengeDeterministic-v4', 'ElevatorActionNoFrameskip-v4', 'PrivateEye-v0', 'FreewayNoFrameskip-v4', 'Asterix-ram-v0', 'NameThisGame-ram-v4', 'ElevatorAction-ramNoFrameskip-v0', 'BeamRiderDeterministic-v0', 'ChopperCommand-v0', 'DemonAttack-ram-v0', 'BowlingDeterministic-v4', 'StarGunnerDeterministic-v4', 'JourneyEscapeDeterministic-v0', 'StarGunnerNoFrameskip-v0', 'Zaxxon-ram-v0', 'Freeway-v0', 'Defender-v4', 'Centipede-ramNoFrameskip-v4', 'Robotank-ramDeterministic-v0', 'IceHockey-ram-v0', 'NameThisGame-ramNoFrameskip-v0', 'StarGunner-ramNoFrameskip-v0', 'Pitfall-ramNoFrameskip-v4', 'WizardOfWor-ramDeterministic-v0', 'BeamRider-ramDeterministic-v0', 'MontezumaRevengeDeterministic-v0', 'RoadRunner-ramDeterministic-v4', 'Solaris-v0', 'Venture-v0', 'Amidar-ramNoFrameskip-v4', 'Assault-ram-v4', 'Alien-ram-v4', 'Atlantis-ramNoFrameskip-v0', 'RoadRunner-ram-v4', 'Assault-ramNoFrameskip-v0', 'Qbert-ramNoFrameskip-v4', 'BattleZone-ramDeterministic-v4', 'Pong-ramDeterministic-v0', 'MsPacman-v0', 'FishingDerby-ramNoFrameskip-v0', 'BattleZone-ram-v4', 'YarsRevenge-v0', 'Gopher-ramNoFrameskip-v4', 'FishingDerby-ram-v0', 'DoubleDunk-ramNoFrameskip-v4', 'BeamRider-v4', 'DefenderNoFrameskip-v0', 'Gopher-ram-v4', 'Qbert-ramDeterministic-v4', 'Gravitar-ram-v4', 'VideoPinballNoFrameskip-v4', 'TennisDeterministic-v4', 'BeamRiderNoFrameskip-v4', 'WizardOfWorDeterministic-v4', 'Frostbite-v0', 'TutankhamDeterministic-v4', 'ElevatorAction-v4', 'MontezumaRevenge-v0', 'Amidar-ramDeterministic-v4', 'Skiing-ramDeterministic-v0', 'Zaxxon-v4', 'Skiing-v4', 'Assault-v0', 'Berzerk-v0', 'Asteroids-ramNoFrameskip-v4', 'CrazyClimber-v0', 'DemonAttack-ramDeterministic-v4', 'WizardOfWor-ramNoFrameskip-v4', 'FishingDerby-ramDeterministic-v4', 'Boxing-v0', 'KungFuMaster-ram-v0', 'VideoPinball-ram-v4', 'BattleZoneNoFrameskip-v0', 'BankHeist-v0', 'Berzerk-ram-v0', 'Carnival-ramNoFrameskip-v0', 'MsPacman-ramDeterministic-v0', 'Asteroids-ram-v0', 'PongNoFrameskip-v0', 'DemonAttack-ramNoFrameskip-v4', 'WizardOfWorNoFrameskip-v4', 'Atlantis-ram-v0', 'Freeway-ramDeterministic-v0', 'BoxingNoFrameskip-v0', 'WizardOfWor-ramDeterministic-v4', 'KungFuMaster-ramNoFrameskip-v4', 'Zaxxon-ramNoFrameskip-v4', 'WizardOfWor-v0', 'Berzerk-ramNoFrameskip-v0', 'Phoenix-ramDeterministic-v4', 'Gravitar-ramNoFrameskip-v0', 'CentipedeDeterministic-v4', 'DemonAttackNoFrameskip-v0', 'BreakoutDeterministic-v0', 'NameThisGame-v4', 'AmidarNoFrameskip-v4', 'RiverraidNoFrameskip-v4', 'Tennis-ramDeterministic-v4', 'VideoPinball-v0', 'BerzerkDeterministic-v0', 'Boxing-ramDeterministic-v0', 'Berzerk-ramDeterministic-v0', 'UpNDown-ramNoFrameskip-v0', 'DemonAttack-ramDeterministic-v0', 'Gravitar-ramDeterministic-v4', 'TimePilot-ramNoFrameskip-v0', 'FrostbiteDeterministic-v4', 'BeamRider-ramDeterministic-v4', 'Robotank-ram-v4', 'Defender-ramDeterministic-v0', 'Centipede-ramDeterministic-v0', 'FreewayDeterministic-v0', 'IceHockey-v4', 'StarGunner-ram-v4', 'Berzerk-ramDeterministic-v4', 'RobotankDeterministic-v0', 'Centipede-v4', 'PrivateEye-ramDeterministic-v4', 'Adventure-ram-v0', 'CrazyClimber-ramDeterministic-v4', 'Solaris-ramDeterministic-v0', 'BowlingDeterministic-v0', 'SeaquestNoFrameskip-v0', 'Pitfall-ram-v4', 'JourneyEscape-ram-v0', 'Boxing-ramNoFrameskip-v4', 'BoxingDeterministic-v0', 'StarGunner-ramNoFrameskip-v4', 'CrazyClimberNoFrameskip-v4'}
unittest {'CubeCrashSparse-v0', 'CubeCrash-v0', 'CubeCrashScreenBecomesBlack-v0', 'MemorizeDigits-v0'}

================================

函数：

def get_default_network(env_type):
    if env_type in {'atari', 'retro'}:
        return 'cnn'
    else:
        return 'mlp'

根据输出参数返回指定哪种神经网络结构，CNN还是MLP。

===========================================

函数：

def get_alg_module(alg, submodule=None):
    submodule = submodule or alg
    try:
        # first try to import the alg module from baselines
        alg_module = import_module('.'.join(['baselines', alg, submodule]))
    except ImportError:
        # then from rl_algs
        alg_module = import_module('.'.join(['rl_' + 'algs', alg, submodule]))

    return alg_module

get_alg_module函数的作用就是 输入的alg变量是个字符串，比如 alg为 'os' ， 这个函数内部的作用就是import os; return os;

该函数就是在程序启动时识别要启动的具体算法，然后在算法库中根据具体的算法名称调用该算法。

函数：

def get_learn_function(alg):
    return get_alg_module(alg).learn

调用具体强化学习算法模块的learn函数

函数：

def get_learn_function_defaults(alg, env_type):
    try:
        alg_defaults = get_alg_module(alg, 'defaults')
        kwargs = getattr(alg_defaults, env_type)()
    except (ImportError, AttributeError):
        kwargs = {}
    return kwargs

alg为要调用的算法，如：deepq、a2c、acer等，这里alg为字符串类型，如下：

baselines中的所有算法都以名字单独有一个文件夹。

env_type为游戏环境类型，也是最前面的字典_game_envs中的key值：

我们比较常用的游戏环境为atari 和 mujoco 。

函数get_learn_function_defaults的作用就是返回调用的强化学习算法模块下的default子模块中定义的名称为对应的env_type的函数来返回算法运行参数字典，这里我们以deepq算法模块下的default子模块为例：

函数：

def get_env_type(args):
    env_id = args.env

    if args.env_type is not None:
        return args.env_type, env_id

    # Re-parse the gym registry, since we could have new envs since last time.
    for env in gym.envs.registry.all():
        env_type = env.entry_point.split(':')[0].split('.')[-1]
        _game_envs[env_type].add(env.id)  # This is a set so add is idempotent

    if env_id in _game_envs.keys():
        env_type = env_id
        env_id = [g for g in _game_envs[env_type]][0]
    else:
        env_type = None
        for g, e in _game_envs.items():
            if env_id in e:
                env_type = g
                break
        if ':' in env_id:
            env_type = re.sub(r':.*', '', env_id)
        assert env_type is not None, 'env_id {} is not recognized in env types'.format(env_id, _game_envs.keys())

    return env_type, env_id

根据输入的环境名env_id识别出所属的环境类型env_type 。

对输入参数的解析：

def arg_parser():
    """
    Create an empty argparse.ArgumentParser.
    """
    import argparse
    return argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)

def common_arg_parser():
    """
    Create an argparse.ArgumentParser for run_mujoco.py.
    """
    parser = arg_parser()
    parser.add_argument('--env', help='environment ID', type=str, default='Reacher-v2')
    parser.add_argument('--env_type', help='type of environment, used when the environment type cannot be automatically determined', type=str)
    parser.add_argument('--seed', help='RNG seed', type=int, default=None)
    parser.add_argument('--alg', help='Algorithm', type=str, default='ppo2')
    parser.add_argument('--num_timesteps', type=float, default=1e6),
    parser.add_argument('--network', help='network type (mlp, cnn, lstm, cnn_lstm, conv_only)', default=None)
    parser.add_argument('--gamestate', help='game state to load (so far only used in retro games)', default=None)
    parser.add_argument('--num_env', help='Number of environment copies being run in parallel. When not specified, set to number of cpus for Atari, and to 1 for Mujoco', default=None, type=int)
    parser.add_argument('--reward_scale', help='Reward scale factor. Default: 1.0', default=1.0, type=float)
    parser.add_argument('--save_path', help='Path to save trained model to', default=None, type=str)
    parser.add_argument('--save_video_interval', help='Save video every x steps (0 = disabled)', default=0, type=int)
    parser.add_argument('--save_video_length', help='Length of recorded video. Default: 200', default=200, type=int)
    parser.add_argument('--log_path', help='Directory to save learning curve data.', default=None, type=str)
    parser.add_argument('--play', default=False, action='store_true')
    return parser

def parse_unknown_args(args):
    """
    Parse arguments not consumed by arg parser into a dictionary
    """
    retval = {}
    preceded_by_key = False
    for arg in args:
        if arg.startswith('--'):
            if '=' in arg:
                key = arg.split('=')[0][2:]
                value = arg.split('=')[1]
                retval[key] = value
            else:
                key = arg[2:]
                preceded_by_key = True
        elif preceded_by_key:
            retval[key] = arg
            preceded_by_key = False

    return retval

def parse_cmdline_kwargs(args):
    '''
    convert a list of '='-spaced command-line arguments to a dictionary, evaluating python objects when possible
    '''
    def parse(v):

        assert isinstance(v, str)
        try:
            return eval(v)
        except (NameError, SyntaxError):
            return v

    return {k: parse(v) for k,v in parse_unknown_args(args).items()}

arg_parser = common_arg_parser()
args, unknown_args = arg_parser.parse_known_args()
extra_args = parse_cmdline_kwargs(unknown_args)

print(args)
print(unknown_args)
print(extra_args)

运行：

python test.py --aaa=me --xxx=11.11  --abc=True   --cde=1+99

结果：

Namespace(env='Reacher-v2', env_type=None, seed=None, alg='ppo2', num_timesteps=1000000.0, network=None, gamestate=None, num_env=None, reward_scale=1.0, save_path=None, save_video_interval=0, save_video_length=200, log_path=None, play=False)
['--aaa=me', '--xxx=11.11', '--abc=True', '--cde=1+99']
{'aaa': 'me', 'xxx': 11.11, 'abc': True, 'cde': 100}

其中，函数parse_unknown_args 是对没有成功解析的字符串构成的list转为dict类型，这里就是将list数据：

['--aaa=me', '--xxx=11.11', '--abc=True', '--cde=1+99']

转成字典类型：

{'aaa:'me', 'xxx':'11.11', 'abc':'True', 'cde':'1+99'}

而操作：{k: parse(v) for k,v in parse_unknown_args(args).items()}

再将字符类型的value转为对应的python类型，也就有了最后解析出来的字典：

{'aaa': 'me', 'xxx': 11.11, 'abc': True, 'cde': 100} 。

不过这个字符解析的操作搞得这么复杂在这里其实意义不太大，毕竟对于未定义的变量进行解析本就是一个极为小众的操作，这里不得不是baselines库的编写有些太不精简和优化了。

baselines所支持的游戏环境，由于可以兼容retro模块，也就是说可以安装一些任天堂的游戏，因此需要为支持的环境字典中添加retro下的游戏名：

_game_envs['retro'] = {
    'BubbleBobble-Nes',
    'SuperMarioBros-Nes',
    'TwinBee3PokoPokoDaimaou-Nes',
    'SpaceHarrier-Nes',
    'SonicTheHedgehog-Genesis',
    'Vectorman-Genesis',
    'FinalFight-Snes',
    'SpaceInvaders-Snes',
}

打印完整的环境类型和环境名：

import gym
from collections import defaultdict

_game_envs = defaultdict(set)
for env in gym.envs.registry.all():
    # TODO: solve this with regexes
    env_type = env.entry_point.split(':')[0].split('.')[-1]
    _game_envs[env_type].add(env.id)

_game_envs['retro'] = {
    'BubbleBobble-Nes',
    'SuperMarioBros-Nes',
    'TwinBee3PokoPokoDaimaou-Nes',
    'SpaceHarrier-Nes',
    'SonicTheHedgehog-Genesis',
    'Vectorman-Genesis',
    'FinalFight-Snes',
    'SpaceInvaders-Snes',
}

for env_type in _game_envs:
    print(env_type, '\n', list(_game_envs[env_type]))
    print('......')

打印结果：

algorithmic
 ['DuplicatedInput-v0', 'ReversedAddition-v0', 'Copy-v0', 'RepeatCopy-v0', 'Reverse-v0', 'ReversedAddition3-v0']
......
classic_control
 ['CartPole-v1', 'CartPole-v0', 'MountainCar-v0', 'Acrobot-v1', 'MountainCarContinuous-v0', 'Pendulum-v0']
......
box2d
 ['LunarLander-v2', 'LunarLanderContinuous-v2', 'CarRacing-v0', 'BipedalWalkerHardcore-v3', 'BipedalWalker-v3']
......
toy_text
 ['FrozenLake-v0', 'KellyCoinflip-v0', 'Blackjack-v0', 'NChain-v0', 'GuessingGame-v0', 'FrozenLake8x8-v0', 'Taxi-v3', 'HotterColder-v0', 'CliffWalking-v0', 'Roulette-v0', 'KellyCoinflipGeneralized-v0']
......
mujoco
 ['Walker2d-v2', 'Swimmer-v2', 'Humanoid-v2', 'InvertedDoublePendulum-v2', 'Reacher-v2', 'Hopper-v2', 'HalfCheetah-v2', 'Striker-v2', 'InvertedPendulum-v2', 'Thrower-v2', 'HumanoidStandup-v2', 'Ant-v2', 'Pusher-v2']
......
half_cheetah_v3
 ['HalfCheetah-v3']
......
hopper_v3
 ['Hopper-v3']
......
swimmer_v3
 ['Swimmer-v3']
......
walker2d_v3
 ['Walker2d-v3']
......
ant_v3
 ['Ant-v3']
......
humanoid_v3
 ['Humanoid-v3']
......
robotics
 ['FetchPush-v1', 'HandManipulateBlockRotateXYZTouchSensorsDense-v1', 'HandManipulateBlockRotateParallelTouchSensorsDense-v0', 'HandManipulateEggRotateTouchSensors-v1', 'HandManipulateBlockTouchSensors-v0', 'HandManipulateEggTouchSensorsDense-v0', 'HandManipulateEggFullDense-v0', 'HandManipulatePenDense-v0', 'HandManipulateBlockTouchSensorsDense-v1', 'HandManipulateBlockRotateParallelTouchSensors-v1', 'HandManipulateEggFull-v0', 'FetchPickAndPlaceDense-v1', 'HandManipulateEggRotateTouchSensors-v0', 'HandManipulateBlockRotateXYZTouchSensors-v0', 'HandManipulateBlockRotateParallelTouchSensors-v0', 'HandManipulateEggTouchSensors-v1', 'FetchPushDense-v1', 'HandManipulateBlockRotateZTouchSensorsDense-v1', 'HandReach-v0', 'HandManipulatePenTouchSensors-v1', 'HandManipulateBlockRotateXYZDense-v0', 'HandManipulateEggRotateTouchSensorsDense-v1', 'HandManipulateBlockRotateXYZTouchSensorsDense-v0', 'HandManipulatePenFullDense-v0', 'FetchPickAndPlace-v1', 'HandManipulatePenRotateTouchSensors-v0', 'HandManipulateBlockRotateZDense-v0', 'HandManipulateEggRotateTouchSensorsDense-v0', 'HandManipulateBlockRotateParallelDense-v0', 'HandManipulateBlock-v0', 'HandManipulateEggTouchSensorsDense-v1', 'HandManipulateBlockRotateXYZTouchSensors-v1', 'FetchReachDense-v1', 'HandManipulatePenTouchSensorsDense-v0', 'HandManipulatePenTouchSensorsDense-v1', 'HandManipulateEgg-v0', 'HandManipulateEggDense-v0', 'HandManipulateBlockRotateZTouchSensors-v1', 'FetchReach-v1', 'HandManipulatePen-v0', 'HandManipulateBlockFullDense-v0', 'HandManipulateBlockFull-v0', 'HandManipulateBlockRotateZ-v0', 'HandManipulateBlockRotateZTouchSensors-v0', 'HandManipulatePenTouchSensors-v0', 'HandManipulateEggRotate-v0', 'HandReachDense-v0', 'HandManipulateBlockRotateXYZ-v0', 'HandManipulateBlockDense-v0', 'FetchSlide-v1', 'HandManipulateBlockRotateParallel-v0', 'FetchSlideDense-v1', 'HandManipulatePenRotate-v0', 'HandManipulatePenFull-v0', 'HandManipulateBlockRotateParallelTouchSensorsDense-v1', 'HandManipulatePenRotateDense-v0', 'HandManipulateEggRotateDense-v0', 'HandManipulateBlockTouchSensors-v1', 'HandManipulatePenRotateTouchSensorsDense-v0', 'HandManipulateBlockTouchSensorsDense-v0', 'HandManipulatePenRotateTouchSensors-v1', 'HandManipulateEggTouchSensors-v0', 'HandManipulatePenRotateTouchSensorsDense-v1', 'HandManipulateBlockRotateZTouchSensorsDense-v0']
......
atari
 ['RoadRunner-ramDeterministic-v4', 'YarsRevenge-v4', 'TimePilot-ram-v0', 'Zaxxon-ramDeterministic-v0', 'AsteroidsNoFrameskip-v0', 'MsPacmanNoFrameskip-v0', 'SpaceInvaders-ramDeterministic-v4', 'Gopher-ramNoFrameskip-v0', 'Venture-v4', 'TutankhamNoFrameskip-v0', 'Skiing-ramDeterministic-v0', 'Phoenix-ram-v4', 'Asterix-v4', 'Pong-ram-v4', 'TimePilotNoFrameskip-v0', 'Assault-ramDeterministic-v4', 'Riverraid-ramDeterministic-v0', 'BerzerkDeterministic-v0', 'DefenderDeterministic-v4', 'Gopher-ramDeterministic-v0', 'Gravitar-ramNoFrameskip-v0', 'Skiing-ramDeterministic-v4', 'ChopperCommand-ramDeterministic-v0', 'Atlantis-ramDeterministic-v0', 'DefenderDeterministic-v0', 'BankHeistNoFrameskip-v0', 'ZaxxonDeterministic-v4', 'Solaris-ramNoFrameskip-v4', 'AmidarNoFrameskip-v0', 'HeroDeterministic-v0', 'AsterixDeterministic-v4', 'TimePilot-ramDeterministic-v4', 'IceHockeyDeterministic-v0', 'Centipede-ram-v0', 'BeamRiderNoFrameskip-v0', 'BeamRider-ramDeterministic-v4', 'AtlantisDeterministic-v0', 'MsPacmanDeterministic-v0', 'DefenderNoFrameskip-v4', 'Jamesbond-ramDeterministic-v4', 'Venture-ram-v4', 'QbertDeterministic-v0', 'Solaris-ramDeterministic-v0', 'Solaris-ramNoFrameskip-v0', 'Jamesbond-ramNoFrameskip-v0', 'Krull-v4', 'Tutankham-v4', 'WizardOfWor-ramDeterministic-v0', 'BattleZone-ramNoFrameskip-v4', 'FishingDerbyDeterministic-v4', 'RoadRunner-v4', 'TimePilot-ramDeterministic-v0', 'YarsRevengeDeterministic-v0', 'AsterixNoFrameskip-v4', 'BeamRiderDeterministic-v0', 'Bowling-ramNoFrameskip-v0', 'JourneyEscape-v0', 'TimePilot-v0', 'GopherDeterministic-v4', 'PhoenixDeterministic-v0', 'StarGunner-v4', 'Alien-v0', 'Pong-ramNoFrameskip-v4', 'Frostbite-ram-v4', 'Pooyan-ram-v4', 'Bowling-ramDeterministic-v0', 'RoadRunner-ram-v4', 'TimePilotNoFrameskip-v4', 'Phoenix-ram-v0', 'Assault-ramDeterministic-v0', 'Seaquest-ramNoFrameskip-v4', 'AlienNoFrameskip-v0', 'Defender-ram-v0', 'SolarisDeterministic-v4', 'Tutankham-v0', 'Kangaroo-ramDeterministic-v0', 'PhoenixDeterministic-v4', 'VideoPinballDeterministic-v0', 'Freeway-ramDeterministic-v4', 'Asteroids-v0', 'JourneyEscape-ram-v0', 'StarGunner-ramNoFrameskip-v4', 'VideoPinball-v4', 'Carnival-ramNoFrameskip-v0', 'Assault-ram-v4', 'PrivateEye-ramDeterministic-v0', 'Venture-v0', 'PrivateEyeDeterministic-v0', 'Alien-ram-v4', 'ElevatorActionNoFrameskip-v0', 'MsPacman-ram-v0', 'Asterix-ram-v0', 'Bowling-ram-v4', 'MontezumaRevenge-ramNoFrameskip-v0', 'VentureDeterministic-v0', 'AmidarDeterministic-v0', 'ElevatorActionNoFrameskip-v4', 'SpaceInvaders-ram-v0', 'ChopperCommandNoFrameskip-v0', 'SeaquestDeterministic-v4', 'StarGunner-ramDeterministic-v0', 'SpaceInvadersDeterministic-v4', 'VideoPinballNoFrameskip-v0', 'MontezumaRevenge-ram-v4', 'SpaceInvaders-ram-v4', 'Frostbite-v4', 'Pong-ramDeterministic-v4', 'Adventure-ramNoFrameskip-v4', 'AdventureDeterministic-v0', 'KrullNoFrameskip-v4', 'NameThisGame-ramNoFrameskip-v0', 'Centipede-v0', 'WizardOfWor-ram-v0', 'AtlantisNoFrameskip-v4', 'AsteroidsDeterministic-v0', 'BeamRiderDeterministic-v4', 'Centipede-ramDeterministic-v4', 'YarsRevenge-ramDeterministic-v0', 'PooyanDeterministic-v0', 'DemonAttack-ramDeterministic-v4', 'RoadRunnerDeterministic-v4', 'PongNoFrameskip-v4', 'CrazyClimber-ram-v4', 'IceHockey-v4', 'Adventure-v0', 'Krull-v0', 'BerzerkNoFrameskip-v0', 'Freeway-v0', 'Amidar-ramNoFrameskip-v0', 'MsPacmanDeterministic-v4', 'GopherDeterministic-v0', 'YarsRevenge-v0', 'Pong-ramDeterministic-v0', 'Robotank-ramDeterministic-v0', 'AtlantisNoFrameskip-v0', 'TennisNoFrameskip-v4', 'VentureNoFrameskip-v4', 'BankHeist-ramDeterministic-v4', 'Gravitar-ramDeterministic-v4', 'PrivateEye-v4', 'StarGunner-ramNoFrameskip-v0', 'VideoPinball-v0', 'BreakoutDeterministic-v4', 'StarGunnerDeterministic-v4', 'KungFuMaster-ramNoFrameskip-v0', 'RoadRunner-v0', 'Krull-ramNoFrameskip-v0', 'AirRaidNoFrameskip-v4', 'Gravitar-ram-v0', 'Tutankham-ramNoFrameskip-v0', 'Adventure-ram-v4', 'FishingDerby-ramDeterministic-v0', 'PrivateEye-ram-v0', 'Boxing-ramNoFrameskip-v4', 'KungFuMaster-ram-v4', 'ZaxxonNoFrameskip-v4', 'YarsRevenge-ram-v4', 'JourneyEscapeDeterministic-v4', 'BoxingNoFrameskip-v0', 'PrivateEye-ramNoFrameskip-v0', 'Zaxxon-ram-v4', 'RobotankNoFrameskip-v4', 'Amidar-v0', 'SolarisDeterministic-v0', 'Gopher-ram-v0', 'WizardOfWorDeterministic-v0', 'DoubleDunkNoFrameskip-v0', 'CentipedeNoFrameskip-v4', 'Berzerk-ramDeterministic-v0', 'Skiing-ram-v4', 'Gopher-ramDeterministic-v4', 'Amidar-ram-v4', 'Breakout-ramDeterministic-v4', 'Freeway-ramNoFrameskip-v0', 'NameThisGame-ramDeterministic-v4', 'Hero-v4', 'PrivateEye-ram-v4', 'Riverraid-v4', 'MontezumaRevenge-ramNoFrameskip-v4', 'Seaquest-ram-v0', 'CentipedeDeterministic-v0', 'Phoenix-ramNoFrameskip-v4', 'JamesbondDeterministic-v0', 'IceHockey-ramNoFrameskip-v0', 'Atlantis-ram-v4', 'Seaquest-ramDeterministic-v4', 'Adventure-ramDeterministic-v0', 'AssaultNoFrameskip-v0', 'BeamRider-ramNoFrameskip-v4', 'BerzerkNoFrameskip-v4', 'NameThisGameDeterministic-v4', 'SkiingNoFrameskip-v4', 'CrazyClimberDeterministic-v0', 'ZaxxonDeterministic-v0', 'Phoenix-ramDeterministic-v0', 'Boxing-ramDeterministic-v4', 'Pong-v4', 'SpaceInvadersNoFrameskip-v4', 'DoubleDunk-v4', 'PooyanNoFrameskip-v4', 'MsPacman-ramDeterministic-v4', 'RoadRunner-ramNoFrameskip-v4', 'Enduro-ramNoFrameskip-v0', 'Freeway-ram-v4', 'Berzerk-v0', 'Zaxxon-ramNoFrameskip-v4', 'Enduro-v0', 'Freeway-ram-v0', 'FishingDerby-ramDeterministic-v4', 'MsPacman-ram-v4', 'TutankhamDeterministic-v0', 'UpNDown-ram-v4', 'Venture-ram-v0', 'Assault-v0', 'DemonAttack-v0', 'TimePilotDeterministic-v0', 'RoadRunner-ramNoFrameskip-v0', 'BattleZoneDeterministic-v0', 'DemonAttack-ramNoFrameskip-v4', 'Boxing-ramDeterministic-v0', 'MontezumaRevengeNoFrameskip-v4', 'DemonAttack-v4', 'PrivateEyeDeterministic-v4', 'MontezumaRevenge-v0', 'AssaultDeterministic-v4', 'SpaceInvadersNoFrameskip-v0', 'Robotank-ram-v0', 'WizardOfWorNoFrameskip-v0', 'Asteroids-ramDeterministic-v0', 'CrazyClimberNoFrameskip-v0', 'DoubleDunk-ramNoFrameskip-v4', 'Zaxxon-ramNoFrameskip-v0', 'PooyanDeterministic-v4', 'RobotankDeterministic-v4', 'Krull-ram-v0', 'Carnival-ramDeterministic-v0', 'BankHeistDeterministic-v4', 'IceHockey-ramDeterministic-v0', 'Jamesbond-ram-v0', 'StarGunnerNoFrameskip-v0', 'BoxingNoFrameskip-v4', 'Gopher-v4', 'AlienDeterministic-v0', 'Zaxxon-ramDeterministic-v4', 'VideoPinball-ramNoFrameskip-v4', 'TimePilot-v4', 'NameThisGame-ram-v0', 'PitfallDeterministic-v0', 'GravitarDeterministic-v4', 'DoubleDunk-ramNoFrameskip-v0', 'YarsRevenge-ram-v0', 'PhoenixNoFrameskip-v4', 'Gravitar-v4', 'Pooyan-ramDeterministic-v4', 'Berzerk-ramDeterministic-v4', 'RiverraidDeterministic-v0', 'CrazyClimber-ramNoFrameskip-v0', 'KungFuMasterDeterministic-v0', 'Phoenix-ramNoFrameskip-v0', 'StarGunner-ram-v0', 'BeamRider-ram-v4', 'Defender-ramDeterministic-v0', 'MsPacman-ramNoFrameskip-v0', 'Pitfall-ramDeterministic-v0', 'BoxingDeterministic-v0', 'Enduro-ram-v4', 'Adventure-ramDeterministic-v4', 'Riverraid-ramNoFrameskip-v0', 'Enduro-ramDeterministic-v0', 'Boxing-v4', 'Skiing-v0', 'BattleZoneNoFrameskip-v0', 'Pitfall-ramNoFrameskip-v0', 'BowlingNoFrameskip-v0', 'DoubleDunkDeterministic-v0', 'ElevatorAction-ramDeterministic-v0', 'Boxing-ram-v0', 'WizardOfWor-ramNoFrameskip-v4', 'MsPacman-ramNoFrameskip-v4', 'Defender-v4', 'Bowling-v4', 'Berzerk-ramNoFrameskip-v4', 'MontezumaRevenge-ram-v0', 'AtlantisDeterministic-v4', 'GravitarDeterministic-v0', 'FreewayDeterministic-v0', 'Breakout-ram-v4', 'CrazyClimberDeterministic-v4', 'Zaxxon-v4', 'EnduroDeterministic-v0', 'KungFuMaster-ramNoFrameskip-v4', 'NameThisGame-ram-v4', 'Breakout-ramNoFrameskip-v0', 'ElevatorAction-ramNoFrameskip-v4', 'Carnival-v0', 'Pooyan-ram-v0', 'BankHeist-v0', 'NameThisGameDeterministic-v0', 'MontezumaRevenge-ramDeterministic-v4', 'DoubleDunk-ram-v4', 'Asteroids-v4', 'Alien-ramNoFrameskip-v0', 'Breakout-ramNoFrameskip-v4', 'AmidarDeterministic-v4', 'SpaceInvaders-v4', 'Boxing-v0', 'Riverraid-ram-v0', 'UpNDownDeterministic-v0', 'Defender-ramNoFrameskip-v0', 'StarGunner-ram-v4', 'MsPacmanNoFrameskip-v4', 'Skiing-ramNoFrameskip-v4', 'Tennis-v4', 'SolarisNoFrameskip-v4', 'TutankhamNoFrameskip-v4', 'ChopperCommand-ramNoFrameskip-v0', 'Gravitar-ramNoFrameskip-v4', 'VideoPinball-ram-v4', 'Qbert-ramDeterministic-v4', 'Pitfall-ramNoFrameskip-v4', 'Pooyan-ramDeterministic-v0', 'MsPacman-ramDeterministic-v0', 'Enduro-v4', 'Freeway-ramNoFrameskip-v4', 'JourneyEscape-ramDeterministic-v0', 'Alien-ram-v0', 'FreewayDeterministic-v4', 'Qbert-v0', 'Kangaroo-ramNoFrameskip-v4', 'Atlantis-ramNoFrameskip-v0', 'Frostbite-ramNoFrameskip-v0', 'Hero-ramNoFrameskip-v4', 'YarsRevengeNoFrameskip-v4', 'SkiingDeterministic-v0', 'Gopher-v0', 'PitfallDeterministic-v4', 'Carnival-v4', 'Riverraid-ram-v4', 'FishingDerby-ramNoFrameskip-v0', 'AirRaidDeterministic-v4', 'Pitfall-ram-v4', 'BattleZone-v4', 'Tennis-ram-v0', 'YarsRevengeNoFrameskip-v0', 'BattleZone-ram-v0', 'AirRaid-ramNoFrameskip-v0', 'Qbert-ramNoFrameskip-v0', 'TennisDeterministic-v4', 'Asteroids-ram-v4', 'AsteroidsDeterministic-v4', 'BowlingDeterministic-v4', 'Asterix-ramNoFrameskip-v0', 'Qbert-ramNoFrameskip-v4', 'ElevatorAction-v4', 'MontezumaRevenge-ramDeterministic-v0', 'BeamRider-ramNoFrameskip-v0', 'BankHeist-ramNoFrameskip-v4', 'ElevatorActionDeterministic-v0', 'FishingDerby-v0', 'FishingDerbyNoFrameskip-v0', 'Frostbite-ram-v0', 'JamesbondDeterministic-v4', 'Phoenix-v4', 'Pooyan-v0', 'NameThisGame-ramDeterministic-v0', 'Riverraid-ramDeterministic-v4', 'JamesbondNoFrameskip-v4', 'Kangaroo-ram-v0', 'Gopher-ramNoFrameskip-v4', 'Centipede-ramDeterministic-v0', 'Carnival-ram-v0', 'AlienNoFrameskip-v4', 'BowlingNoFrameskip-v4', 'Breakout-v4', 'KungFuMasterNoFrameskip-v4', 'Amidar-ramDeterministic-v4', 'Asterix-v0', 'PongDeterministic-v0', 'UpNDown-ramDeterministic-v0', 'FreewayNoFrameskip-v4', 'Gravitar-v0', 'BattleZone-ramDeterministic-v4', 'Bowling-ramDeterministic-v4', 'Frostbite-ramNoFrameskip-v4', 'Gopher-ram-v4', 'Asteroids-ramNoFrameskip-v4', 'BankHeist-ram-v4', 'Centipede-ram-v4', 'Kangaroo-ramDeterministic-v4', 'Pong-v0', 'SeaquestNoFrameskip-v0', 'Jamesbond-v4', 'FishingDerby-v4', 'Krull-ram-v4', 'BeamRider-v0', 'TennisDeterministic-v0', 'ChopperCommandDeterministic-v4', 'QbertNoFrameskip-v4', 'BattleZone-ramDeterministic-v0', 'RoadRunnerNoFrameskip-v4', 'Atlantis-v0', 'BankHeist-v4', 'TennisNoFrameskip-v0', 'SpaceInvaders-v0', 'DemonAttackNoFrameskip-v4', 'BattleZoneDeterministic-v4', 'Tutankham-ramDeterministic-v4', 'KungFuMaster-v4', 'Boxing-ram-v4', 'SkiingNoFrameskip-v0', 'Pong-ram-v0', 'Asterix-ramDeterministic-v0', 'JamesbondNoFrameskip-v0', 'StarGunner-v0', 'WizardOfWor-ramDeterministic-v4', 'FrostbiteNoFrameskip-v0', 'FishingDerbyDeterministic-v0', 'QbertNoFrameskip-v0', 'KungFuMaster-ram-v0', 'Atlantis-v4', 'CrazyClimberNoFrameskip-v4', 'Defender-ram-v4', 'JourneyEscape-v4', 'Zaxxon-v0', 'DoubleDunk-ramDeterministic-v0', 'YarsRevenge-ramNoFrameskip-v0', 'BattleZone-ram-v4', 'Hero-ram-v4', 'KungFuMaster-ramDeterministic-v4', 'CentipedeDeterministic-v4', 'Seaquest-ram-v4', 'ChopperCommand-ram-v0', 'ElevatorAction-v0', 'Atlantis-ram-v0', 'Skiing-v4', 'Robotank-ramDeterministic-v4', 'Pitfall-v0', 'Solaris-ram-v4', 'Tennis-ramNoFrameskip-v4', 'Venture-ramDeterministic-v0', 'KrullNoFrameskip-v0', 'DemonAttackNoFrameskip-v0', 'MontezumaRevengeDeterministic-v0', 'EnduroNoFrameskip-v0', 'Freeway-v4', 'CrazyClimber-ram-v0', 'FishingDerby-ramNoFrameskip-v4', 'RiverraidNoFrameskip-v4', 'Berzerk-ramNoFrameskip-v0', 'Venture-ramNoFrameskip-v0', 'Adventure-v4', 'PrivateEye-ramNoFrameskip-v4', 'UpNDown-ramNoFrameskip-v0', 'BreakoutNoFrameskip-v4', 'ElevatorActionDeterministic-v4', 'BeamRiderNoFrameskip-v4', 'DemonAttackDeterministic-v4', 'Kangaroo-ram-v4', 'Phoenix-ramDeterministic-v4', 'AirRaidDeterministic-v0', 'Hero-v0', 'UpNDown-ramDeterministic-v4', 'Defender-ramNoFrameskip-v4', 'DoubleDunk-ramDeterministic-v4', 'Centipede-v4', 'ChopperCommand-v4', 'Frostbite-ramDeterministic-v0', 'SeaquestNoFrameskip-v4', 'SolarisNoFrameskip-v0', 'AirRaid-v4', 'HeroNoFrameskip-v4', 'FishingDerby-ram-v0', 'Gravitar-ramDeterministic-v0', 'KungFuMasterDeterministic-v4', 'Pong-ramNoFrameskip-v0', 'IceHockey-ramDeterministic-v4', 'Kangaroo-ramNoFrameskip-v0', 'UpNDown-ram-v0', 'BeamRider-ram-v0', 'Hero-ramDeterministic-v0', 'Jamesbond-v0', 'Tennis-ramNoFrameskip-v0', 'Freeway-ramDeterministic-v0', 'WizardOfWor-v0', 'Tennis-ram-v4', 'FrostbiteNoFrameskip-v4', 'Skiing-ram-v0', 'NameThisGameNoFrameskip-v4', 'Alien-ramDeterministic-v0', 'Pitfall-ramDeterministic-v4', 'IceHockey-ramNoFrameskip-v4', 'Solaris-v0', 'Asterix-ram-v4', 'Adventure-ram-v0', 'HeroNoFrameskip-v0', 'Frostbite-ramDeterministic-v4', 'Bowling-ramNoFrameskip-v4', 'VideoPinballDeterministic-v4', 'VideoPinball-ramDeterministic-v0', 'DefenderNoFrameskip-v0', 'TimePilotDeterministic-v4', 'UpNDown-v0', 'Seaquest-v0', 'QbertDeterministic-v4', 'FreewayNoFrameskip-v0', 'VideoPinball-ramDeterministic-v4', 'Kangaroo-v4', 'Alien-ramNoFrameskip-v4', 'KrullDeterministic-v4', 'PongDeterministic-v4', 'RoadRunnerDeterministic-v0', 'GravitarNoFrameskip-v4', 'AlienDeterministic-v4', 'KungFuMasterNoFrameskip-v0', 'Hero-ramDeterministic-v4', 'Asterix-ramNoFrameskip-v4', 'KrullDeterministic-v0', 'AirRaid-ram-v4', 'MontezumaRevengeDeterministic-v4', 'Qbert-ramDeterministic-v0', 'Asteroids-ramNoFrameskip-v0', 'StarGunner-ramDeterministic-v4', 'GravitarNoFrameskip-v0', 'WizardOfWor-ram-v4', 'YarsRevenge-ramDeterministic-v4', 'DemonAttack-ramDeterministic-v0', 'NameThisGame-ramNoFrameskip-v4', 'PooyanNoFrameskip-v0', 'Enduro-ram-v0', 'Robotank-ramNoFrameskip-v0', 'Asteroids-ramDeterministic-v4', 'Carnival-ramDeterministic-v4', 'Berzerk-v4', 'JourneyEscape-ramDeterministic-v4', 'Kangaroo-v0', 'BattleZone-ramNoFrameskip-v0', 'NameThisGame-v0', 'Assault-ramNoFrameskip-v0', 'StarGunnerNoFrameskip-v4', 'CentipedeNoFrameskip-v0', 'Venture-ramDeterministic-v4', 'Frostbite-v0', 'JourneyEscape-ramNoFrameskip-v0', 'Krull-ramDeterministic-v0', 'VideoPinball-ramNoFrameskip-v0', 'IceHockeyDeterministic-v4', 'AdventureDeterministic-v4', 'Carnival-ram-v4', 'YarsRevenge-ramNoFrameskip-v4', 'CarnivalDeterministic-v4', 'UpNDownNoFrameskip-v4', 'FishingDerbyNoFrameskip-v4', 'Riverraid-v0', 'Robotank-ramNoFrameskip-v4', 'CrazyClimber-v0', 'RiverraidNoFrameskip-v0', 'AsterixDeterministic-v0', 'WizardOfWor-v4', 'AdventureNoFrameskip-v0', 'Solaris-ram-v0', 'Seaquest-ramNoFrameskip-v0', 'Tennis-ramDeterministic-v0', 'DoubleDunkNoFrameskip-v4', 'VentureDeterministic-v4', 'AsterixNoFrameskip-v0', 'Qbert-ram-v4', 'MontezumaRevengeNoFrameskip-v0', 'RoadRunner-ramDeterministic-v0', 'Phoenix-v0', 'BreakoutDeterministic-v0', 'WizardOfWorDeterministic-v4', 'ZaxxonNoFrameskip-v0', 'KangarooDeterministic-v0', 'Alien-v4', 'MsPacman-v4', 'Assault-ramNoFrameskip-v4', 'RobotankNoFrameskip-v0', 'MontezumaRevenge-v4', 'Solaris-v4', 'Tennis-v0', 'Amidar-ram-v0', 'SpaceInvadersDeterministic-v0', 'TutankhamDeterministic-v4', 'DoubleDunk-ram-v0', 'NameThisGameNoFrameskip-v0', 'JourneyEscapeNoFrameskip-v0', 'PitfallNoFrameskip-v4', 'TimePilot-ram-v4', 'Tutankham-ram-v4', 'CrazyClimber-v4', 'CrazyClimber-ramNoFrameskip-v4', 'Enduro-ramNoFrameskip-v4', 'Breakout-ramDeterministic-v0', 'BankHeistDeterministic-v0', 'ChopperCommandNoFrameskip-v4', 'Defender-v0', 'Pitfall-v4', 'BeamRider-v4', 'AmidarNoFrameskip-v4', 'Solaris-ramDeterministic-v4', 'ElevatorAction-ram-v0', 'SpaceInvaders-ramDeterministic-v0', 'BankHeistNoFrameskip-v4', 'JourneyEscape-ramNoFrameskip-v4', 'EnduroNoFrameskip-v4', 'Tutankham-ramNoFrameskip-v4', 'KungFuMaster-ramDeterministic-v0', 'Berzerk-ram-v0', 'UpNDownNoFrameskip-v0', 'IceHockey-v0', 'SpaceInvaders-ramNoFrameskip-v4', 'Jamesbond-ramNoFrameskip-v4', 'Bowling-ram-v0', 'PrivateEyeNoFrameskip-v4', 'ElevatorAction-ram-v4', 'WizardOfWor-ramNoFrameskip-v0', 'Seaquest-ramDeterministic-v0', 'Pooyan-ramNoFrameskip-v4', 'Zaxxon-ram-v0', 'Atlantis-ramNoFrameskip-v4', 'UpNDownDeterministic-v4', 'CarnivalNoFrameskip-v4', 'ChopperCommand-v0', 'JourneyEscapeDeterministic-v0', 'FrostbiteDeterministic-v4', 'TimePilot-ramNoFrameskip-v4', 'Assault-v4', 'Enduro-ramDeterministic-v4', 'ChopperCommand-ramNoFrameskip-v4', 'KungFuMaster-v0', 'DemonAttack-ram-v0', 'BattleZone-v0', 'Centipede-ramNoFrameskip-v4', 'Venture-ramNoFrameskip-v4', 'UpNDown-v4', 'VideoPinballNoFrameskip-v4', 'BreakoutNoFrameskip-v0', 'AirRaid-ramNoFrameskip-v4', 'Centipede-ramNoFrameskip-v0', 'ChopperCommand-ram-v4', 'Alien-ramDeterministic-v4', 'IceHockeyNoFrameskip-v4', 'AirRaidNoFrameskip-v0', 'Qbert-v4', 'SpaceInvaders-ramNoFrameskip-v0', 'Asteroids-ram-v0', 'Pooyan-ramNoFrameskip-v0', 'Qbert-ram-v0', 'CrazyClimber-ramDeterministic-v4', 'DemonAttackDeterministic-v0', 'AirRaid-v0', 'BerzerkDeterministic-v4', 'Tennis-ramDeterministic-v4', 'KangarooDeterministic-v4', 'FishingDerby-ram-v4', 'CarnivalDeterministic-v0', 'VideoPinball-ram-v0', 'HeroDeterministic-v4', 'ElevatorAction-ramNoFrameskip-v0', 'Krull-ramNoFrameskip-v4', 'UpNDown-ramNoFrameskip-v4', 'Pooyan-v4', 'ChopperCommandDeterministic-v0', 'IceHockey-ram-v0', 'StarGunnerDeterministic-v0', 'TimePilot-ramNoFrameskip-v0', 'ElevatorAction-ramDeterministic-v4', 'Jamesbond-ram-v4', 'Hero-ramNoFrameskip-v0', 'Bowling-v0', 'BowlingDeterministic-v0', 'EnduroDeterministic-v4', 'Riverraid-ramNoFrameskip-v4', 'RiverraidDeterministic-v4', 'Breakout-ram-v0', 'Robotank-v0', 'Atlantis-ramDeterministic-v4', 'Assault-ram-v0', 'Seaquest-v4', 'PhoenixNoFrameskip-v0', 'Defender-ramDeterministic-v4', 'Krull-ramDeterministic-v4', 'BankHeist-ramNoFrameskip-v0', 'NameThisGame-v4', 'AirRaid-ram-v0', 'Berzerk-ram-v4', 'Boxing-ramNoFrameskip-v0', 'AssaultDeterministic-v0', 'Asterix-ramDeterministic-v4', 'Jamesbond-ramDeterministic-v0', 'Breakout-v0', 'DemonAttack-ramNoFrameskip-v0', 'DoubleDunkDeterministic-v4', 'CrazyClimber-ramDeterministic-v0', 'CarnivalNoFrameskip-v0', 'PrivateEyeNoFrameskip-v0', 'BeamRider-ramDeterministic-v0', 'Robotank-v4', 'SkiingDeterministic-v4', 'AirRaid-ramDeterministic-v4', 'PitfallNoFrameskip-v0', 'Skiing-ramNoFrameskip-v0', 'AsteroidsNoFrameskip-v4', 'Carnival-ramNoFrameskip-v4', 'RoadRunnerNoFrameskip-v0', 'PrivateEye-ramDeterministic-v4', 'KangarooNoFrameskip-v0', 'AirRaid-ramDeterministic-v0', 'JourneyEscapeNoFrameskip-v4', 'BattleZoneNoFrameskip-v4', 'DemonAttack-ram-v4', 'JourneyEscape-ram-v4', 'Tutankham-ram-v0', 'ChopperCommand-ramDeterministic-v4', 'GopherNoFrameskip-v4', 'IceHockey-ram-v4', 'RoadRunner-ram-v0', 'Robotank-ram-v4', 'WizardOfWorNoFrameskip-v4', 'Hero-ram-v0', 'SeaquestDeterministic-v0', 'PrivateEye-v0', 'Gravitar-ram-v4', 'Amidar-v4', 'AssaultNoFrameskip-v4', 'GopherNoFrameskip-v0', 'MsPacman-v0', 'VentureNoFrameskip-v0', 'KangarooNoFrameskip-v4', 'YarsRevengeDeterministic-v4', 'Tutankham-ramDeterministic-v0', 'Adventure-ramNoFrameskip-v0', 'BankHeist-ram-v0', 'DoubleDunk-v0', 'Pitfall-ram-v0', 'PongNoFrameskip-v0', 'AdventureNoFrameskip-v4', 'IceHockeyNoFrameskip-v0', 'Amidar-ramNoFrameskip-v4', 'FrostbiteDeterministic-v0', 'BoxingDeterministic-v4', 'RobotankDeterministic-v0', 'Amidar-ramDeterministic-v0', 'BankHeist-ramDeterministic-v0']
......
unittest
 ['MemorizeDigits-v0', 'CubeCrash-v0', 'CubeCrashScreenBecomesBlack-v0', 'CubeCrashSparse-v0']
......
retro
 ['SuperMarioBros-Nes', 'Vectorman-Genesis', 'FinalFight-Snes', 'SonicTheHedgehog-Genesis', 'TwinBee3PokoPokoDaimaou-Nes', 'SpaceHarrier-Nes', 'SpaceInvaders-Snes', 'BubbleBobble-Nes']
......

在 main 函数中主要调用train函数。

在train函数中真正执行算法训练的函数操作为：

    model = learn(
        env=env,
        seed=seed,
        total_timesteps=total_timesteps,
        **alg_kwargs
    )

如果是执行deepq的算法，那么这里的这个learn函数就是deepq算法模块中的learn函数，也就是说根据指定的不同算法模块最后传入的learn函数都是其对应算法模块下的learn函数。

learn函数可以根据指定的timesteps来训练算法直到算法最后训练完成，最终的算法的网络模型给传输回来，也就是这里的model变量，model变量本身也是一个类对象，model的step函数输入变量为observation，输出变量为action的np.array类型对象。

在train函数中对游戏画面录像的操作：

    if args.save_video_interval != 0:
        env = VecVideoRecorder(env, osp.join(logger.get_dir(), "videos"), record_video_trigger=lambda x: x % args.save_video_interval == 0, video_length=args.save_video_length)

由于gym的视频录像功能并不是很好用，因此推荐使用opencv的图片录像的代码。

========================================

在这个模块还有一个最为重要的操作就是对游戏环境进行创建，这里的函数为build_env 。

def build_env(args):
    ncpu = multiprocessing.cpu_count()
    if sys.platform == 'darwin': ncpu //= 2
    nenv = args.num_env or ncpu
    alg = args.alg
    seed = args.seed

    env_type, env_id = get_env_type(args)

    if env_type in {'atari', 'retro'}:
        if alg == 'deepq':
            env = make_env(env_id, env_type, seed=seed, wrapper_kwargs={'frame_stack': True})
        elif alg == 'trpo_mpi':
            env = make_env(env_id, env_type, seed=seed)
        else:
            frame_stack_size = 4
            env = make_vec_env(env_id, env_type, nenv, seed, gamestate=args.gamestate, reward_scale=args.reward_scale)
            env = VecFrameStack(env, frame_stack_size)

    else:
        config = tf.ConfigProto(allow_soft_placement=True,
                               intra_op_parallelism_threads=1,
                               inter_op_parallelism_threads=1)
        config.gpu_options.allow_growth = True
        get_session(config=config)

        flatten_dict_observations = alg not in {'her'}
        env = make_vec_env(env_id, env_type, args.num_env or 1, seed, reward_scale=args.reward_scale, flatten_dict_observations=flatten_dict_observations)

        if env_type == 'mujoco':
            env = VecNormalize(env, use_tf=True)

    return env

如果游戏环境不属于{'atari', 'retro'}，并且调用的强化学习算法不为'her'，（游戏环境不包括'mujoco'游戏），创建环境的操作为：

        flatten_dict_observations = True
        env = make_vec_env(env_id, env_type, args.num_env or 1, seed, reward_scale=args.reward_scale, flatten_dict_observations=flatten_dict_observations)

如果游戏环境不属于{'atari', 'retro'}，并且调用的强化学习算法为'her'，（游戏环境不包括'mujoco'游戏），创建环境的操作为：

        flatten_dict_observations = False
        env = make_vec_env(env_id, env_type, args.num_env or 1, seed, reward_scale=args.reward_scale, flatten_dict_observations=flatten_dict_observations)

在这里如果gym游戏不属于'atari、retro、mujoco'那么也就是纯gym的原生游戏环境了，对于这样的游戏的创建统一调用方法：

        env = make_vec_env(env_id, env_type, args.num_env or 1, seed, reward_scale=args.reward_scale, flatten_dict_observations=flatten_dict_observations)

唯一不同的是调用的强化学习算法是否为‘her’，如果是则设置flatten_dict_observations为False，否则设置为True。

至于这个flatten_dict_observations变量的设置不是很理解，不过这个设置的意思就是说如果要用的环境为gym的原生环境，并且不调用‘her’算法那么需要对observation进行flatten操作。这里的flatten操作就是调用gym下的FlattenObservation函数。

而这个FlattenObservation操作：

import numpy as np
import gym.spaces as spaces
from gym import ObservationWrapper

class FlattenObservation(ObservationWrapper):
    r"""Observation wrapper that flattens the observation."""
    def __init__(self, env):
        super(FlattenObservation, self).__init__(env)

        flatdim = spaces.flatdim(env.observation_space)
        self.observation_space = spaces.Box(low=-float('inf'), high=float('inf'), shape=(flatdim,), dtype=np.float32)

    def observation(self, observation):
        return spaces.flatten(self.env.observation_space, observation)

def flatdim(space):
    if isinstance(space, Box):
        return int(np.prod(space.shape))
    elif isinstance(space, Discrete):
        return int(space.n)
    elif isinstance(space, Tuple):
        return int(sum([flatdim(s) for s in space.spaces]))
    elif isinstance(space, Dict):
        return int(sum([flatdim(s) for s in space.spaces.values()]))
    elif isinstance(space, MultiBinary):
        return int(space.n)
    elif isinstance(space, MultiDiscrete):
        return int(np.prod(space.shape))
    else:
        raise NotImplementedError

def flatten(space, x):
    if isinstance(space, Box):
        return np.asarray(x, dtype=np.float32).flatten()
    elif isinstance(space, Discrete):
        onehot = np.zeros(space.n, dtype=np.float32)
        onehot[x] = 1.0
        return onehot
    elif isinstance(space, Tuple):
        return np.concatenate([flatten(s, x_part) for x_part, s in zip(x, space.spaces)])
    elif isinstance(space, Dict):
        return np.concatenate([flatten(s, x[key]) for key, s in space.spaces.items()])
    elif isinstance(space, MultiBinary):
        return np.asarray(x).flatten()
    elif isinstance(space, MultiDiscrete):
        return np.asarray(x).flatten()
    else:
        raise NotImplementedError

该步操作主要是将observation进行flatten操作，尤其是将observation的spaces空间为Tuple和Dict的进行flatten。

该步操作可以将observation的spaces空间属于Box, Discrete, Tuple, Dict, MultiBinary, MultiDiscrete类型的observation全部进行flatten操作。

但是由于flatten的操作具体代码为：

    if flatten_dict_observations and isinstance(env.observation_space, gym.spaces.Dict):
        env = FlattenObservation(env)

也就是说实际上只能对observation的空间为Dict的observation进行flatten操作。

但是需要注意的是不论是在make_env函数中还是在make_vec_env函数中默认的参数设置都是flatten_dict_observations=True，也就是说在gym的原生环境中使用‘her‘’算法是不进行flatten操作的，由于对这个‘her’算法并不是很了解这里就不进行分析了。

=================================================

如果游戏环境不属于{'atari', 'retro'}，并且为'mujoco'游戏，环境创建操作为：

        flatten_dict_observations = alg not in {'her'}
        env = make_vec_env(env_id, env_type, args.num_env or 1, seed, reward_scale=args.reward_scale, flatten_dict_observations=flatten_dict_observations)

        if env_type == 'mujoco':
            env = VecNormalize(env, use_tf=True)

===========================================