Turning Photographs into Wireframes, VisualSFM on CentOS6

DL390 G5For the purposes of performing a highly detailed capture of various historical monuments, which are local to me, I purchased a big fat server with oodles of processing capability. The system itself is a 2.83GHz Dual Quad Core Xeon HP Proliant DL390 G5 – not a bad bit of iron from Bargain Hardware. It’s loud, it’s warm but it’s very very quick. Added to that a couple of 1Tb 2.5″ disks and a nVidia 210 with 1Gb of graphics memory. Once you’ve got this kind of hardware you’ve got to put it to good use. First thing for my project was to build up an appropriate workflow for using VisualSFM and Meshlab on the machine. These are tools which perform a “photogrammetric” reconstruction. VisualSFM is difficult to get working, It is dependent on packages which each have their dependencies, it’s also poorly documented generally so I wanted to ensure that my struggle helps out others. CentOS 6 was my OS of choice, I usually use this on a server as it’s pretty low maintenance once it’s configured and support stretches on and on. After a day or so of software builds (btw make -j8 is fantastic) I managed to get it working. I’ll share the binary/source RPMs that I’ve rebuilt and links to the other bits of the entire workflow. Along with a guide on how it is done.

Stages

Each part of this process is difficult and requires some expertise in Linux. If you identify any shortfall in my description please let me know via the comments.

  1. Install all available build dependencies from various repositories/sources
  2. Build SiftGPU to run on nvidia Cuda6
  3. Install Multicore Bundle Adjustment
  4. Download and hack a working PMVS2 and CMVS
  5. Build/Install VisualSFM

Before we begin we’re going to create a place to do the building. We’ll create a folder called VSFM in your home directory, so wherever you see ~/ or /home/karl/ replace it with the appropriate folder on your system.

$ cd ~
$ mkdir VSFM
$ cd VSF

Stage 1 – General Dependencies

Repositories

First you’ll want to install the release repositories for a bunch of dependencies, foremost is the el repo.

# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
# rpm -Uvh http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm

next up, get Cuda 6 installed which can be done by running the following commands, if you don’t install Cuda it will likely continue to work without using the GPU on nvidia systems.

# rpm -Uvh http://developer.download.nvidia.com/compute/cuda/repos/rhel6/x86_64/cuda-repo-rhel6-6.0-37.x86_64.rpm
# yum install cuda-license-6-0-6 cuda-extra-libs-6-0-6 cuda-command-line-tools-6-0-6 cuda-drivers-331 cuda-headers-6-0-6 cuda-misc-6-0-6 cuda-samples-6-0-6 cuda-visual-tools-6-0-6 cuda-core-libs-6-0-6 cuda-toolkit-6-0-6 cuda-core-6-0-6 cuda-documentation-6-0-6 cuda-6-0-6

Other requirements

# yum install gtk2-devel glew-devel boost-devel atlas-sse3-devel atlas-sse2-devel atlas-sse-devel atlas-devel GraphicsMagick cminpack-devel libgfortran metis-devel freeglut-devel gsl-devel

Pre-built RPMs

DevIL is required by SiftGPU, the original project is dormant but there is an effort to fix what has been left behind called ResIL but unfortunately I haven’t been able to get it to build, instead I rebuilt this DevIL RPM

Graclus 1.2

Follow this link to the homepage (Graclus 1.2). You must fill out your email address and a link to the download will be sent to you. Download and unzip it in the ~/VSFM folder. Then edit ~/VSFM/graclus1.2/Makefile.in and set

-DNUMBITS=64

Then build it

$ cd ~/VSFM/graclus1.2
$ make

Stage 2 – Build SiftGPU

Sift is the technique used to match “points” the way that this works is points are picked out which are features which can be translated in 3 dimensions. These features can be identifiable with changes in their scale/rotation in 3D space, because of this Sift will extract and match those features for you in the process and by doing this with a lot of photos, provides the necessary data to do the next part of the puzzle, which is positioning the cameras (Bundle adjustment) and projecting the point cloud (PMVS2 & CMVS). We’re using SiftGPU because it’s fast and works with VisualSFM.

$ cd ~/VSFM
$ wget http://wwwx.cs.unc.edu/~ccwu/cgi-bin/siftgpu.cgi -O SiftGPU.zip 
$ unzip SiftGPU.zip 
$ cd SiftGPU 
$ make

Stage 3 – Build Multicore bundle adjustment

Bundle adjustment helps us locate the exact position and operational settings of the camera, for this task we’re using the multicore bundle adjustment tool PBA. This tool can use both the GPU and CPU cores simultaneously. Multicore Bundle Adjustment described in it’s manual:

is a parallel-accelerated implementation of bundle adjustment for multicore CPU and GPU. By restructuring the non-linear optimization problem, the overall computation becomes dominated by series of simple matrix-vector operations. The matrix-vector operations are then parallelized with a combination of multi-threading and SIMD (Single Instruction Multiple Data). Additionally, the problem restructuring enables tremendous memory saving by computing Jacobians on- the-fly during matrix-vector multiplication. I prefer to call it PBA (Parallel Bundle Adjustment), which corresponds to the class interface ParallelBA.

– Changchang Wu

$ cd ~/VSFM
$ wget http://grail.cs.washington.edu/projects/mcba/pba_v1.0.5.zip -O pba.zip
$ unzip pba.zip

Edit pba/src/pba/SparseBundleCU.h and pba/src/pba/pba.h and add the following line

#include <stdlib.h>

Build multicore bundle adjust with

$ make

Stage 4 – Download and hack a working PMVS2 and CMVS

PMVS2 reconstructs a point cloud from a collection of images, but if it were to start processing hundreds of images your computer would likely run out of memory so CMVS is used to identify clusters of images which can be used to generate patches of the point cloud. These patches will then be aligned to the same zero co-ordinate therefore aligning when displayed simultaneously. These tools are used to There’s some trick to getting this to work.

Download and build PMVS2 (315Mb), there’s an issue with PMVS2 where mylapack.o isn’t easily rebuildable and there’s an issue with running the binary, so the hack to get around this requires you backing up mylapack.o cleaning, reinstating it and building.

$ cd ~/VSFM
$ wget http://www.di.ens.fr/pmvs/pmvs-2.tar.gz
$ tar xf pmvs-2.tar.gz
$ cd pmvs-2/program/main/
$ cp mylapack.o mylapack.o.backup
$ make clean
$ cp mylapack.o.backup mylapack.o
$ make depend
$ make

Download and prepare CMVS (232Mb)

$ cd ~/VSFM
$ wget http://www.di.ens.fr/cmvs/cmvs-fix2.tar.gz 
$ tar xf cmvs-fix2.tar.gz 
$ cp pmvs-2/program/main/mylapack.o cmvs/program/main/

Edit ~/VSFM/cmvs/program/base/cmvs/bundle.cc, inserting the following lines at the beginning of the file

#include <vector>
#include <numeric>

Next edit ~/VSFM/cmvs/program/main/genOption.cc adding this include at the top

#include <stdlib.h>

Next you have to modify the makefile cmvs/program/main/Makefile

#Your INCLUDE path (e.g., -I/usr/include)
YOUR_INCLUDE_PATH =

#Your metis directory (contains header files under graclus1.2/metisLib/)
YOUR_INCLUDE_METIS_PATH = -I/home/karl/VSFM/graclus1.2/metisLib

#Your LDLIBRARY path (e.g., -L/usr/lib)
YOUR_LDLIB_PATH = -L/home/karl/VSFM/graclus1.2
$ cd ~/VSFM/cmvs/program/main
$ make

Stage 5 – Build/Install VisualSFM

VisualSFM is a UI for running CMVS, PMVS2, PBA and SiftGPU it’s pretty much a front end for those tools with a viewer component which allows you to see the various images, cameras and projected points. There are other UIs but VisualSFM has the point cloud visualisation built in, and some other neat features.

$ wget http://ccwu.me/vsfm/download/VisualSFM_linux_64bit.zip
$ unzip VisualSFM_linux_64bit.zip
$ cd vsfm
$ make
$ cp ~/VSFM/SiftGPU/bin/libsiftgpu.so ../vsfm/bin 
$ cp ~/VSFM/cmvs/program/main/cmvs ~/VSFM/vsfm/bin
$ cp ~/VSFM/cmvs/program/main/pmvs2 ~/VSFM/vsfm/bin
$ cp ~/VSFM/cmvs/program/main/genOption ~/VSFM/vsfm/bin

Create a script to launch it called ~/bin/vsfm

#!/bin/bash
export PATH=$PATH:$HOME/VSFM/vsfm/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/VSFM/vsfm/bin
VisualSFM

Next make the script executable

$ chmod +x ~/bin/vsfm

Create a desktop file ~/.local/share/applications/vsfm.desktop

#!/usr/bin/env xdg-open
[Desktop Entry]
Version=1.0
Type=Application
Terminal=false
Icon[en_US]=gnome-panel-launcher
Name[en_US]=VisualSFM
Exec=~/bin/vsfm
Name=VisualSFM
Icon=gnome-panel-launcher

Now VisualSFM should appear somewhere in your menu.

Test Data Sets

After spending a lot of time online I found that good test data sets were pretty hard to come by for a decent SfM reconstruction. I’ve included here a church located in Seaham County Durham, and an example of a small object.

Test Data Downloads

Tips for reconstructing large objects

  • Try to get lots of correlating photos which have different sky patterns
  • Take as many photos as possible, this goes without saying but it’s important to collect lots and lots of pictures if you want good reconstructions.
  • Take photos from lots of angles and elevations the more you have the better.
  • Look for landmark points in the peripheral which can easily be matched as well as matchable foreground patterns
  • Take at least 3 photos of any detailed elements, from 3 distinct but matchable angles.
  • Use a calibration device of some kind, I’ve included a simple cube as a printable net which can be used to produce a rigid cube object for calibration purposes.

Tips for reconstructing small objects

  • Use consistent lighting, using a flash will not work unless the flash is always directed from the same position.
  • Take as many photos as possible
  • Take photos from lots of angles and elevations the more you have the better.
  • Take photos from underneath and inside any holes if there’s detail you want to capture.
  • Take at least 3 photos of any detailed elements, from 3 distinct but matchable angles.
  • Use a base of some kind to improve calibration. I’ve included an SVG pattern which can be printed onto a large piece of paper (A3+) and laid flat to improve the object alignment.

Results

The total time on producing this reconstruction was about 12-14 hours of compute time on my Proliant. Most reconstructions won’t be quite as detailed as this one and there are certain architectural objects which will produce very good results with only a handful of pictures.

Following on

This is the first article in a series about 3D reconstruction processes, I have a bunch more tools to investigate. At each point in the series I’m going to make sure that anything I’m using is at least mirrored here. A few things have just vanished from the internet unfortunately, but through various means I’ve managed to recover parts and will try and bring them together.

Some topics I’m still planning on investigating include

  • Mesh cleaning – using meshlab to clean point clouds and generate textures, I’ll be writing a post about this shortly.
  • MicMac – this is apparently a more user friendly tool with some excellent features, I finally managed to get a copy of the package a day or so ago and will be trying it out as soon as I can.
  • Insight3D – this tool is distributed from sourceforge, however for parts of it theres a requirement for OpenCV 2.4 which unfortunately CentOS is a little behind on and I haven’t managed to rebuild a package yet, but I’m working on it.
  • PPT – Python Photogrammetry Toolbox – these tools are included in archeos but they’re useful for so many different applications so I’ll be making them available to download hopefully as RPM packages shortly.
  • OpenSourcePhotogrammetry/OpenMVS this project seems poorly documented at present but very promising.
  • Using Kinect as a point cloud scanner, this would have to be done at night and we’d need a decent power source to run it.
  • Aligning cameras with location support with dense point clouds and obtaining accurate geo-location information, we can actually increase the resolution of GPS down to extremely small amounts with this method as we won’t only have the satellite information from one location but many multiple locations.

Eventually I’ll hopefully be able to do a few things automatically, essentially providing a group of photographs and get back a point cloud, and a location of the model on a 3D map. Once all of the tools are constructed it will be a matter of capturing tens of thousands of photographs of parts of Newcastle-upon-Tyne and accumulating dozens if not hundreds of historical photographs too. The end reconstruction will hopefully be of use to various people for historical and architectural research.

References

Update

Some of the URL’s have changed over time, I’ve updated them accordingly. I’m considering writing a script to do all of this on an RPM based system. I’ve had to switch over to fedora 21 after something broke on my CentOS install (I think I’d started an upgrade or something).

[auction-affiliate tool=”lister”]