Building VisIt+OSPRay/SWR


Alok Hota, ahota@vols.utk.edu

The following instructions walk you through the steps of building an OSPRay-enabled instance of VisIt. Additionally, you can use OpenSWR (via Mesa) with VisIt, though it requires a separate download as VisIt's Mesa version does not include SWR yet. I tried to include any common problems encountered along with troubleshooting steps to resolve them.

NOTE: These instructions are for Linux machine running on a Xeon CPU or a KNL Xeon Phi. I have not tested with any other OS and don't know how well or even if anything would work. I have confirmed that this works on CentOS, so any RHEL-based distro should be fine. Ubuntu and similar distros should work, but I haven't had the chance to test it. We have tested with Sandy Bridge, Ivy Bridge, and Haswell Xeon CPUs. Single- or dual-socket is fine. Some steps in the process differ on KNL; I tried to make sure to mention the differences when applicable.

I am currently working on a build/binary for newer versions of VisIt, OSPRay, and SWR. Newer versions of VisIt (possibly 2.13+) will be incorporating VTK 7, which has OSPRay support built in. After that transition, the visit_ospray source will not be necessary.

There are some things you need to have before building.


Getting VisIt and OSPRay

To build from source, follow the directions below. If you would like the binary, click here.

To build from source, you will need to download the following. Right now, VisIt+OSPRay/SWR is version-locked to the versions listed below.

Okay, let's get started. Choose a directory under which you are going to do your build.

NOTE: Make sure you have enough space in your disk quota, you will need about 8 GB to finish this process.

I'm going to build it under my home directory, /home/alok.

UNIX> pwd
/home/alok

We will start with VisIt.

UNIX> pwd
/home/alok
UNIX> mkdir visit
UNIX> cp build_visit2_10_0 visit/
UNIX> cd visit
UNIX> chmod +x build_visit2_10_0
UNIX> ./build_visit2_10_0 --qt --vtk --cmake --silo --netcdf --hdf5 --szip --boost --makeflags -j4

This will get the whole process started. Many packages will be downloaded. This process should be fully automatic at this point. If you run into issues, please refer to section 2.1. This process will probably last around 2 hours. In the example above, the -j4 indicates 4 threads will be used during the build process. Feel free to use more than 4 threads if you have the cores available.

Next, we will build OSPRay.

UNIX> pwd
/home/alok
UNIX> mkdir ospray
UNIX> cp v1.1.0.tar.gz ospray/
UNIX> cp ispc-v1.9.1-linux.tar.gz ospray/
UNIX> cd ospray
UNIX> tar xf v1.1.0.tar.gz
UNIX> tar xf ispc-v1.9.1-linux.tar.gz
UNIX> mkdir build

The build directory is where the build process will take place.

UNIX> cd build
UNIX> cmake -DOSPRAY_APPS_BENCHMARK=OFF -DOSPRAY_APPS_ENABLE_SCRIPTING=OFF -DOSPRAY_APPS_GLUTVIEWER=OFF -DOSPRAY_APPS_PARAVIEW_TFN_CVT=OFF -DOSPRAY_APPS_QTVIEWER=OFF -DOSPRAY_BUILD_ISA=AVX2 -DOSPRAY_BUILD_MIC_SUPPORT=OFF -DOSPRAY_BUILD_MPI_DEVICE=OFF -DOSPRAY_USE_EXTERNAL_EMBREE=OFF ../ospray-1.1.0
UNIX> make -j 4

In this example, the ISA being used is AVX2, e.g. on a Haswell/Broadwell Xeon. If you are building on KNL, change the -DOSPRAY_BUILD_ISA=AVX2 option to AVX512. For other variations, please refer to Section 3.

Assuming everything has gone perfectly until this point, the next step is to build the OSPRay-enabled VisIt. The source for OSPRay-enabled VisIt is available on a VisIt development branch, shown below.

UNIX> pwd
/home/alok
UNIX> mkdir visit_ospray
UNIX> cd visit_ospray
UNIX> svn co http://visit.ilight.com/svn/visit/branches/hrchilds/OSPRay/src/
UNIX> mkdir build

Again, the build directory is where the build process will occur.

UNIX> cd build
UNIX> cp ../../visit/<hostname>.cmake .
... you need to modify this cmake file according to Section 4 below
UNIX> cmake ../src
UNIX> make -j 4

Assuming the universe aligns in your favor, you are done. To run your OSPRay-enabled VisIt build:

UNIX> cd bin/
UNIX> ./visit -gui -ospray

At this point, your directory should look like the following:

     -- /home/alok/
        |-- ospray/
        |   |-- build/
        |   |-- ospray-1.1.0/
        |   `-- ispc-v1.9.1-linux/
        |-- visit/
        `-- visit_ospray/
            |-- build/
            |   `-- bin/
            |       `-- visit*
            `-- src/
        

In this example, /home/alok/visit_ospray/build/bin/visit is the OSPRay-enabled application executable.

If you're ready to run a quick test, go straight to Section 6. Of course many things could have happened during this process. The remainder of this document contains more detailed information and troubleshooting steps for the process.

About the above components

Use VisIt's build_visit script to get all necessary libraries and to build a working copy of VisIt. The script will install everything into one place (called the VISIT_HOME directory) which makes everything much easier. At the time of writing, 2.10 is the most recent version confirmed to work. This process should also work with 2.8.1 and 2.9.x. Versions 2.11+ are not yet guaranteed to work.

OSPRay source is available in the different releases on their GitHub repository. The most recent version confirmed to work is 1.1.0. Higher versions are not yet confirmed or guaranteed to work. Lower versions will not work as the code uses features not present until 1.1.0.

OSPRay requires ISPC, which is a SIMD compiler. At the time of writing, the highest available version is 1.9.1, which is known to work with OSPRay 1.1.0.

The VisIt+OSPRay source is a modified version of the VisIt code that includes some piping to call OSPRay for rendering. It is currently available as a branch in VisIt's svn repository. Run svn co <the above URL> in /home/alok/visit_ospray/ to get the code.

Notes

In general I prefer out-of-source builds to keep things clean, hence the separate build and src directories in the example structure above. The build_visit script unfortunately does an in-source build by default. This might be configurable by setting a VISIT_BUILD_DIR environment variable. I only recently found this option and have not tested it, however.

Building VisIt (part 1)

Using the build_visit script

First the vanilla VisIt instance needs to be installed using the build_visit script downloaded in the last step. The script will download several tar files and create build directories for each library it needs. You shouldn't need to worry about these. The two directories it creates that are important are visit and visit2.10 (or whichever version you use). The former will contain all installations of the libraries and is referred to as the VISIT_HOME directory. The latter will contain a src directory with the source code and compiled binaries.

It is NOT recommended to use system libraries, even for things like Qt or Python. I have found that system installations are often missing certain options that will break the VisIt build process. I suggest saving yourself the headache and letting VisIt download and install the necessary libraries.

Run the build script with:

./build_visit --qt --vtk --cmake --silo --netcdf --hdf5 --szip --boost --makeflags -j4

If you want to specify a compiler, use the --cc and --cxx flags to specify a C and C++ compiler, respectively. Feel free to specify more or fewer threads for make (the -j4).

This will ask you to agree with Qt's license at the beginning. Building and installing will take some time, but in general I have found this command to be fairly reliable.

Notes

In the directory containing the build_visit script, /home/alok/visit in this case, there will be a cmake file named <hostname>.cmake. This contains the settings for the VISIT_HOME directory mentioned above. You will need this when building the OSPRay-enabled instance of Visit.

Building OSPRay

Untar the ISPC binaries. I like to leave them in the ispc directory that the tar file contains, and keep it in the ospray directory alongside build and ospray-1.1.0.

Go to the build directory. You can run the CMake command given at the beginning of this document. An alternative to this is using ccmake, which provides a terminal interface to the CMake process.

To use ccmake, run ccmake ../ospray-1.1.0. In ccmake, press c to configure. OSPRay may not find the ISPC executable. The executable is located within the ispc-v1.9.1-linux directory from the tar file. I suggest giving an absolute path. Set the following variables:

NOTE: Warnings about OSPRay not finding Embree installed on your system are okay. It will use a pre-packaged Embree bundled with OSPRay 1.1.0. You don't need to download or build Embree.

NOTE: If you are building on KNL and need AVX512, you will have to use the Intel compiler.

You can turn off the different example applications. If you would like to verify OSPRay was built correctly, you can leave the volume viewer application enabled. You will need to download a readable dataset to test with, such as the ones at the bottom of this page.

Finally, run make -j 4 for however many threads you like. This may take some time, especially when building and linking the different AVX libraries. Once compilation is done, make sure that you have the files libembree.so and libospray.so.

Troubleshooting

If you don't have a libembree.so shared library after compiling, double check that you set OSPRAY_USE_EXTERNAL_EMBREE to OFF.

Building VisIt (part 2)

Copy the <hostname>.cmake file from the vanilla VisIt build into the visit_ospray/src/config-site directory. You will need to modify it to add four CMake configuration variables:

VISIT_OPTION_DEFAULT(OSPRAY_DIR /home/alok/ospray/src) VISIT_OPTION_DEFAULT(OSPRAY_CMAKE_DIR /home/alok/ospray/src/cmake) VISIT_OPTION_DEFAULT(OSPRAY_BUILD_DIR /home/alok/ospray/build) VISIT_OPTION_DEFAULT(VISIT_OSPRAY ON TYPE BOOL)

I usually place these before the database reader plugin libraries. Of course, replace the directories above with the actual ones you are using. Note that this CMake file has settings for the VISIT_HOME directory. This allows the VisIt+OSPRay build to use the libraries that were installed in the vanilla build.

Change to the visit_ospray build directory and run cmake ../src, then run make with the desired number of threads. This may take some time. If the compile is successful, you should be able to go to the bin/ directory and run the following:

./visit -gui -ospray [-o /your/data/file/here]

Open a volume of your choice and render it. Currently, if the VisIt volume rendering method is set to Splatting, it will use OSPRay instead. OSPRay will also work when rendering a mesh, e.g. with a Pseudocolor plot or a Contour plot.

Troubleshooting

If your compiler gives an error stating it cannot find vtkMultiProcessController.h, you can safely comment out the line that includes it. You can resume compiling after making this change without cleaning.

If you are getting errors about unresolved symbols, you can try modifying a line in your <hostname>.cmake file. Find the two lines setting C/C++ compiler flags. They are usually near the top and commented. In the flags given, remove the -fvisibility flag from both the C and C++ flags. This keeps symbols from being marked as hidden. Rebuild VisIt after making this change.

Notes

Some of the annotations in VisIt do not currently work well with OSPRay. This includes the color map legend, dataset information, and user information. Bounding box and axis/ticks work, as shown in the examples below.

You may sometimes see some white/black lines in the bottom left corner. These are artifacts of the annotations and triad attempting to be rendered. I'm hoping to fix this soon.

On our dual-socket Haswell machine, we typically see 1-2 fps for large volumes when rendering with OSPRay. Note that this is with 8 samples per pixel in OSPRay. A lower sample rate results in faster rendering at the expense of image quality. The frame rate is expected to increase when VisIt moves to VTK 7.x; development for that is underway.

Using Mesa/OpenSWR

VisIt already uses Mesa, but includes a version that does not have OpenSWR built in. The newer version is a drop-in replacement, making it easy to get OpenSWR's performance. If you were able to get Mesa through your package manager, you can skip building. If you downloaded the source, first make sure you have the dependencies listed here. Building Mesa is a simple process using configure. To specifically use OpenSWR, run

configure --disable-dri --disable-egl --enable-xlib-glx --with-gallium-drivers=swrast,swr --prefix=/home/alok/mesa-13.0.0/install/

This command is suggested by OpenSWR. Then, run make and make install.

To use Mesa/OpwnSWR, set the following environment variables before running VisIt.

export LD_LIBRARY_PATH=/home/alok/mesa-13.0.0/install/lib:$LD_LIBRARY_PATH
export GALLIUM_DRIVER=swr

In our tests, Mesa/OpenSWR 13.0 performs better than the Mesa baseline in VisIt's rendering benchmarks. The baseline Mesa tests had a 94.38% pass rate, compared to Mesa/OpenSWR's 94.44%.

Testing

To do a simple test, we can volume render an iron protein dataset. I've uploaded the dataset and a Python script that can automate some steps in VisIt.

UNIX> pwd
/home/alok
UNIX> mkdir examples
UNIX> cd examples
UNIX> wget http://web.eecs.utk.edu/~ahota/ironProt.vtk
UNIX> wget http://web.eecs.utk.edu/~ahota/ospray_vol_test.py
UNIX> /home/alok/visit-ospray/build/bin/visit -gui -ospray -s ospray_vol_test.py

This will launch the OSPRay-enabled VisIt built above and run the Python script. The script simply loads the ironProt dataset, adds a Volume plot, and renders it. The volume render will be done with OSPRay.

Examples

The following show examples of VisIt rendering with OSPRay and Mesa/SWR using various different plots and on different platforms.

VisIt 2.10 with OSPRay 1.1.0 on dual-socket Haswell Xeon


VisIt 2.10 with OSPRay 1.1.2 on KNL
VisIt 2.9.2 with Mesa 17 release candidate 2 (includes SWR) on dual-socket Haswell Xeon