Theano-based implementation of Deep Q-learning is available on github. During the DeepHack hackathon our team «’drop table table users;» (which consists of me, Alexey Ozerin, Alexander Notchenko, Mikhail and Artur Kuzin) managed to get this code up and running on Ubuntu in a couple of minutes because of the provided dep_script.sh. But running this code on Mac resulted in some problems, which we were able to solve succeffully.
First of all to run test script with ./run_nips –rom breakout you need to make sure you have corresponding roms:
$ cd ./roms
$ wget https://atariage.com/2600/roms/Breakout.zip
$ unzip -e Breakout.zip && rm Breakout.zip
You might need to alter ~/.theanorc:
$ cat ~/.theanorc
[global]
floatX = float32
device = gpu0
[nvcc]
fastmath = True
Error with unavailable numpy/multiarray.h can be solved with exporting proper PATH variables:
export PATH=/Developer/NVIDIA/CUDA-7.0/bin:$PATH
export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-7.0/lib:$DYLD_LIBRARY_PATH
and / or altering a ale_data_set.py file replacing coressponding lines:
import pyximport
import numpy as np
pyximport.install(setup_args={'include_dirs': np.get_include()})
Annoying dnn error
File "/lib/python2.7/site-packages/Lasagne-0.1.dev0-py2.7.egg/lasagne/layers/dnn.py", line 13, in
raise ImportError("dnn not available") # pragma: no cover
ImportError: dnn not available
can be solved with copying cudnn header and lib files to /usr/local/lib and /usr/local/include:
$ cd /cudnn-6.5-osx-v2/
$ ls
CUDNN_License.pdf cudnn.h libcudnn.dylib
INSTALL.txt libcudnn.6.5.dylib libcudnn_static.a
$ cp cudnn.h /usr/local/include/
$ cp *.dylib /usr/local/lib/
$ cp libcudnn_static.a /usr/local/lib/
$ cd /deep_q_rl/deep_q_rl
#and the running deep_q_rl looks like this:
$ ./run_nips.py --rom breakout
RL-Glue Version 3.04, Build 909
RL-Glue is listening for connections on port=4096
RL-Glue Python Experiment Codec Version: 2.1 (Build 738)
Connecting to 127.0.0.1 on port 4096...
RL-Glue :: Experiment connected.
A.L.E: Arcade Learning Environment (version 0.5.0)
[Powered by Stella]
Use -help for help screen.
Warning: couldn't load settings file: ./stellarc
Game console created:
ROM file: ../roms/breakout.bin
Cart Name: Breakout - Breakaway IV (1978) (Atari)
Cart MD5: f34f08e5eb96e500e851a80be3277a56
Display Format: AUTO-DETECT ==> NTSC
ROM Size: 2048
Bankswitch Type: AUTO-DETECT ==> 2K
Running ROM file…
Random Seed: Time
Game will be controlled through RL-Glue.
Initializing ALE RL-Glue …
RL-Glue :: Environment connected.
Using gpu device 0: GeForce GT 750M
RL-Glue Python Agent Codec Version: 2.1 (Build 738)
Connecting to 127.0.0.1 on port 4096…
Agent Codec Connected
RL-Glue :: Agent connected.
/Lasagne-0.1.dev0-py2.7.egg/lasagne/layers/helper.py:69: UserWarning: get_all_layers() has been changed to return layers in topological order. The former implementation is still available as get_all_layers_old(), but will be removed before the first release of Lasagne. To ignore this warning, use `warnings.filterwarnings(‘ignore’, ‘.*topo.*’)`.
warnings.warn(“get_all_layers() has been changed to return layers in ”
INFO:root:OPENING breakout_07-20-18-18_0p0002_0p95/results.csv
INFO:root:training epoch: 1 steps_left: 50000
INFO:root:steps/second: 100.20
Hope that this notes have made you able to train and successfully run an atari game agent.
P.S. If you were unable to run code and you still get ImportError: dnn not available error, you might want to debug theano/sandbox/cuda/dnn.py dnn_available() function.