Notes on AWS AMI Setup

This is an informal notes page. I am using it to keep track of configuration steps while I work towards scheduling and running big RenderToolbox3 jobs on AWS.

AWS AMI Config.

We had an AMI with Matlab installed and configured to use a private Matlab license server. From this AMI I installed what we need for RenderToolbox3.

I needed more disk and RAM than a t2.micro in order to install and build everything. I went with an m4.large and 50GB disk. Other instance types should work, too.

I launched the instance with our IAM Role ecsInstanceRole, which is granted access to policies AmazonSQSFullAccess, AmazonS3FullAccess, and AmazonSNSFullAccess. These will be useful when we want to read and write data at S3 buckets, via yas3fs. See Notes on YAS3FS Setup.

I used our security groups default and all-ssh, so that I can ssh in, and so that the instance can access our Matlab license server.

General Dependencies

sudo apt-get install git docker.io libboost-all-dev openexr cmake pkg-config

Assimp

cd ~
git clone https://github.com/assimp/assimp.git
cd assimp
cmake CMakeLists.txt -G 'Unix Makefiles'
make
sudo make install
sudo ldconfig

Version Hell: libstdc++

I pointed Matlab at the system installed libstdc++.so.6 instead of the version that's bundled with Matlab. This prevents Matlab from loading the old version. This allows us to use Matlab with mex-functions and shared libraries that were built with newer versions of libstdc++, like mexximp and libassimp. libstdc++ contains symbols for reverse compatibility, so I don't expect this to be a problem.

sudo ln -sf /usr/lib/x86_64-linux-gnu/libstdc++.so.6 /usr/local/MATLAB/R2015b/sys/os/glnxa64/libstdc++.so.6

Docker group

Add the ubuntu user to the docker group so that we can call Docker from Matlab.

sudo groupadd docker
sudo usermod -aG docker $USER

Then log out and back in.

Pull Images

Pull Docker images ahead of time so they're cached in the AMI.

docker pull ninjaben/exrtools-docker
docker pull ninjaben/mitsuba-rgb
docker pull ninjaben/mitsuba-spectral
docker pull ninjaben/pbrt-v2-spectral-docker

ToolboxToolbox

cd ~
git clone https://github.com/ToolboxHub/ToolboxToolbox.git
mkdir ~/MATLAB
cp ~/ToolboxToolbox/sampleStartup.m ~/MATLAB/startup.m
matlab -nodisplay -r "userpath(fullfile(getenv('HOME'), 'MATLAB'));exit"

RenderToolbox3

Cause local hooks to be created, so that we can edit them.

matlab -nodisplay -r "tbUse('VirtualWorldColorConstancy');exit"

Edit local hooks with local configuration. Most of the defaults are fine. I changed the working folders to the folder that we expect to use when mounting S3 buckets. See Notes on YAS3FS Setup. nano ~/localToolboxHooks/RenderToolbox3.m

%% Matlab preferences for RenderToolbox3 default hints.
myFolder = fullfile('/home', 'ubuntu', 'render-toolbox-working', 'render_toolbox');
renderToolbox3.workingFolder = myFolder;

nano ~/localToolboxHooks/VirtualScenes.m

% where to save recipe archives
setpref(prefName, 'recipesFolder', ...
    fullfile('/home', 'ubuntu', 'render-toolbox-working', 'virtual_scenes', 'recipe_archives'));

% where to put the RenderToolbox3 working folder
setpref(prefName, 'workingFolder', ...
    fullfile('/home', 'ubuntu', 'render-toolbox-working', 'virtual_scenes', 'working'));

nano ~/localToolboxHooks/VirtualWorldColorConstancy.m

%% Set up some parameters for portability
projectName = 'ToyVirtualWorld';
dataDirRoot = fullfile('/home', 'ubuntu', 'render-toolbox-working', 'VirtualWorldColorConstancy');

Render Test

matlab -nodisplay -r "tbUse('VirtualWorldColorConstancy');rtbTestInstallation();exit"

Save the AMI

I saved an AMI for this instance. I called it RTB3. Now it should be easy to fire up an instance that can use RenderToolbox3.