by Kirill | Technical in , , ,

How to install Caffe on Mac (OS X Yosemite 10.10.4)

After famous Google Research group post about deep dream, they have released ipynb notebook to mess around with dream generation (available on github).

It requires

And there are a bunch of problems with installing caffe framework on Mac.

There is an official manual for installation with the specific instructions about OS X part.

In the general case you should be able to install Caffe on Mac without any problems. But in case you experience any, you can follow this guide.

Caffe requirements:

  • CUDA is required for GPU mode.
    This is the greatest problem, since CPU only version builds out-of-the-box. You need to download it from the official site and make sure you download version 7.0+. Also you may want to apply for cuDNN — deep learning framework, which may also speed up picture generation later. I’ve applied, but they are still reviewing my application, so I built caffe without cuDNN support so far.
  • BLAS via ATLAS, MKL, or OpenBLAS.
    Official Mac guide suggests that you might have BLAS installed, but I didn’t have it, so I used brew to get openblas (which also provides a speedup compared to BLAS)
  • Boost >= 1.55
  • OpenCV >= 2.4 including 3.0
    Downloading and building OpenCV from the official site was my primary mistake. Since I was building it with the default compiler, not the libstdc++, which works with CUDA
  • protobuf, glog, gflags
  • IO libraries hdf5, leveldb, snappy, lmdb

So, I uninstalled OpenBLAS and OpenCV, I built on my machine and made sure that I don’t have any out of date brew packages installed.

All these steps of removing previously built libraries and setting up all the dependancies are covered in the section “Errors I encountered during the build”, in the mean time the following section should cover the steps required to build Caffe on Mac.

If you encounter any errors, feel free to ask google or consult supplementary links.

How to build Caffe on Mac from scratch

At first we have to edit Makefile.config. My config (lines which differ from original):

# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
BLAS := open
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas

# Homebrew puts openblas in a directory that is not on the standard search path
#BLAS_INCLUDE := $(shell brew –prefix openblas)/include
#BLAS_LIB := $(shell brew –prefix openblas)/lib
BLAS_INCLUDE := /usr/local/Cellar/openblas/0.2.14_1/include
BLAS_LIB := /usr/local/Cellar/openblas/0.2.14_1/lib

# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app

# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
PYTHON_INCLUDE := /usr/include/python2.7 \
/usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it’s in root.
ANACONDA_HOME := $(HOME)/anaconda
PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python2.7 \
$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
PYTHON_LIB := $(ANACONDA_HOME)/lib
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib

After editing Makefile.config you should be able to install Caffe:

brew update
brew tap homebrew/science
for x in snappy leveldb gflags glog szip hdf5 lmdb homebrew/science/opencv; do brew uninstall $x; brew install --fresh -vd $x; done
brew uninstall --force protobuf; brew install --with-python --fresh -vd protobuf
brew uninstall boost boost-python; brew install --fresh -vd boost boost-python
#we have to export this to make sure we include all the cuda and anaconda resources
export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:$HOME/anaconda/lib:/usr/local/lib:/usr/lib:$DYLD_FALLBACK_LIBRARY_PATH
export CPLUS_INCLUDE_PATH=$HOME/anaconda/include/python2.7/:
#using -j8 key to enable parallel building. Number after j is the number of the CPUs available, in my case it's 8
make all -j8
make test -j8
make runtest -j8
make pycaffe
#to run caffe we need to export path to the built caffe:
export PYTHONPATH=/Users/kupa/Desktop/caffe-master/python/:$PYTHONPATH

Errors I encountered during the build


$ rm -rf /usr/local/Cellar && brew prune

If you don’t want to remove all packets, you can at first try removing only those, which are required by caffe:

$ for x in snappy leveldb gflags glog szip hdf5 lmdb homebrew/science/opencv; do brew uninstall $x; brew install --fresh -vd $x; done

In my case after removing them, I had also to link a couple of packages:

$ rm /usr/local/bin/f2py
$ brew link numpy
$ brew install opencv
$ rm -r /usr/local/include/opencv2/
$ rm -r /usr/local/share/OpenCV/
$ brew link opencv

Though, build failed:

$ make all -j8
****
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: ** [.build_release/lib/libcaffe.so] Error 1

By the way, before deleting everything I was getting annoying ‘-pthread’ error:

$ make all -j8
*****
clang: warning: argument unused during compilation: '-pthread'
ld: warning: directory not found for option '-L/opt/intel/mkl/lib/intel64'
Undefined symbols for architecture x86_64:
"leveldb::DB::Open(leveldb::Options const&, std::string const&, leveldb::DB*)", referenced from:
caffe::db::LevelDB::Open(std::string const&, caffe::db::Mode) in db.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: ** [.build_release/lib/libcaffe.so] Error 1

That was due to using wrong protobuf version. I reinstalled protobuf and boost (as suggested here):

$ brew uninstall --force protobuf; brew install --with-python --fresh -vd protobuf
$ brew uninstall boost boost-python; brew install --fresh -vd boost boost-python

This also helps if you have following error:

$ make all -j8
PROTOC src/caffe/proto/caffe.proto
make: protoc: No such file or directory
make: *** [.build_release/src/caffe/proto/caffe.pb.cc] Error 1
make: *** Waiting for unfinished jobs...

There is a possibility that you encounter this output:

./include/caffe/util/mkl_alternate.hpp:11:10: fatal error: 'cblas.h' file not found
#include
^
1 error generated.
make: *** [.build_release/src/caffe/layer_factory.o] Error 1

Which can be dealt with:

$ brew uninstall openblas; brew install --fresh -vd openblas
...
This formula is keg-only, which means it was not symlinked into /usr/local.

OS X already provides this software and installing another version in
parallel can cause all kinds of trouble.

Generally there are no consequences of this for you. If you build your
own software and it requires this formula, you’ll need to add to your
build variables:

LDFLAGS: -L/usr/local/opt/openblas/lib
CPPFLAGS: -I/usr/local/opt/openblas/include

==> Finishing up
Changing dylib ID of /usr/local/Cellar/openblas/0.2.14_1/lib/libopenblasp-r0.2.14.dylib
from @@HOMEBREW_PREFIX@@/opt/openblas/lib/libopenblasp-r0.2.14.dylib
to /usr/local/opt/openblas/lib/libopenblasp-r0.2.14.dylib
Changing install name in /usr/local/Cellar/openblas/0.2.14_1/lib/libopenblasp-r0.2.14.dylib
from @@HOMEBREW_PREFIX@@/lib/gcc/5/libgfortran.3.dylib
to /usr/local/lib/gcc/5/libgfortran.3.dylib
Changing install name in /usr/local/Cellar/openblas/0.2.14_1/lib/libopenblasp-r0.2.14.dylib
from @@HOMEBREW_PREFIX@@/lib/gcc/5/libquadmath.0.dylib
to /usr/local/lib/gcc/5/libquadmath.0.dylib
Changing install name in /usr/local/Cellar/openblas/0.2.14_1/lib/libopenblasp-r0.2.14.dylib
from @@HOMEBREW_PREFIX@@/lib/gcc/5/libgcc_s.1.dylib
to /usr/local/lib/gcc/5/libgcc_s.1.dylib
Changing dylib ID of /usr/local/Cellar/openblas/0.2.14_1/lib/libopenblasp-r0.2.14.dylib
from /usr/local/opt/openblas/lib/libopenblasp-r0.2.14.dylib
to /usr/local/opt/openblas/lib/libopenblasp-r0.2.14.dylib

I gave the output for installing openblas because it might be needed to alter Makefile.config or export global vars:

$ export LDFLAGS=/usr/local/opt/openblas/lib:$LDFLAGS
$ export CPPFLAGS=/usr/local/opt/openblas/include:$CPPFLAGS

Then I encountede an error:

$ make all -j8
CXX src/caffe/blob.cpp
In file included from src/caffe/blob.cpp:4:
In file included from ./include/caffe/blob.hpp:8:
./include/caffe/common.hpp:6:10: fatal error: 'glog/logging.h' file not found

Which resulted in installing glog

$ rm -r /usr/local/lib/cmake
$ brew link gflags
$ brew install glog

Then there was an Error limit:

$ brew install glog
*****

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__functional_03(218): error: unary_function is not a template

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../include/c++/v1/__functional_03(218): error: not a class or struct name

 Error limit reached.
 100 errors detected in the compilation of "/var/folders/ct/3_xtyk5n427_pvx3bfdf5jdm0000gn/T//tmpxft_00002db4_00000000-16_absval_layer.compute_50.cpp1.ii".
 Compilation terminated.
 make: *** [.build_release/cuda/src/caffe/layers/absval_layer.o] Error 1
 
 Which was fixed according to the found solution:
 
 $ cd Library/Formula/
 $ cp boost.rb boost_backup.rb
 $ cp boost-python.rb boost-python_backup.rb
 $ wget https://raw.githubusercontent.com/Homebrew/homebrew/6fd6a9b6b2f56139a44dd689d30b7168ac13effb/Library/Formula/boost.rb
 $ mv boost.rb.1 boost.rb
 $ wget https://raw.githubusercontent.com/Homebrew/homebrew/3141234b3473717e87f3958d4916fe0ada0baba9/Library/Formula/boost-python.rb
 $ mv boost-python.rb.1 boost-python.rb
 $ brew uninstall --force boost
 $ brew install boost

After that I was actually able to build a caffe with:


$ make clean
$ make all -j8
$ make test -j8

But I forgot to export the fallback paths and got error:

$ make runtest
.build_release/tools/caffe
dyld: Library not loaded: @rpath/libcudart.7.0.dylib
Referenced from: /Users/kupa/Desktop/caffe-master/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5

and got fixed it according to the original manual:

$ export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib:/Users/kupa/anaconda/lib:/usr/local/lib:/usr/lib:/usr/local/Cellar/hdf5/:/usr/local/Cellar/:$DYLD_FALLBACK_LIBRARY_PATH

Then, if you followed the guide you should be able to run $make runtest -j8

$ make runtest -j8
*********
[----------] Global test environment tear-down
[==========] 1356 tests from 214 test cases ran. (260250 ms total)
[ PASSED ] 1356 tests.

YOU HAVE 2 DISABLED TESTS

$ make pycaffe
CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp
touch python/caffe/proto/__init__.py
PROTOC (python) src/caffe/proto/caffe.proto

Now, all you left to do is to export correct python path and run $ python -c “import caffe” to test if everything is ok:

$ export PYTHONPATH=$(installed dir)/caffe-master/python/

So now, if you’ve done everything right you should be able to proceed with testing to the google/deepdream github repo.

Useful links

  1. Official caffe manual
  2. Caffe OS X installation manual
  3. Official CUDA download page
  4. Great post about installing caffe
  5. Caffe incompatible with Boost 1.58
  6. Another installing report
  7. One more post about successful experience
  8. Error importing caffe
  9. Error dyld: Library not loaded: libhdf5_hl.7.dylib on github. And one more.
  10. Post which didn’t work for me (but might for you)
  11. Protobuf error
  12. ‘-pthread’ error.
  13. More about protobuf error
  14. Great article about installing caffe on mac.

Share the post "How to install Caffe on Mac (OS X Yosemite 10.10.4)"

  • Facebook
  • Twitter
 

Kirill