Building VisIt+OSPRay/SWR

I now have a pre-built binary of VisIt 2.10 with OSPRay 1.1.0 available here.

The following instructions walk you through the steps of building an OSPRay-enabled instance of VisIt. Additionally, you can use OpenSWR (via Mesa) with VisIt, though it requires a separate download as VisIt's Mesa version does not include SWR yet. I tried to include any common problems encountered along with troubleshooting steps to resolve them.

NOTE: These instructions are for Linux machine running on a Xeon CPU or a KNL Xeon Phi. I have not tested with any other OS and don't know how well or even if anything would work. I have confirmed that this works on CentOS, so any RHEL-based distro should be fine. Ubuntu and similar distros should work, but I haven't had the chance to test it. We have tested with Sandy Bridge, Ivy Bridge, and Haswell Xeon CPUs. Single- or dual-socket is fine. Some steps in the process differ on KNL; I tried to make sure to mention the differences when applicable.

I am currently working on a build/binary for newer versions of VisIt, OSPRay, and SWR.


Getting VisIt and OSPRay

Things you will need

There are several things to download to prepare for building everything.

Software you should have installed ahead of time:

This list contains the things you will need to download. See the explanations below for version numbers and reasons for downloading them.

As usual, it's good to stay organized, especially when dealing with multiple projects. To keep the instructions easy to read, I will use the following directory structure:

     -- /home/alok/
        |-- ospray/
        |   |-- build/
        |   |-- src/
        |   |   `-- v1.1.0.tar.gz
        |   `-- ispc--v1.9.1--linux.tar.gz
        |-- visit/
        |   `-- build_visit2_10_0
        |-- visit_ospray/
        |   |-- build/
        |   `- src/
        `-- mesa_swr/
            |-- install/
            `-- mesa-13.0.2.tar.gz
        
Note the locations where the downloaded items are kept.

About the above components

Use VisIt's build_visit script to get all necessary libraries and to build a working copy of VisIt. The script will install everything into one place (called the VISIT_HOME directory) which makes everything much easier. At the time of writing, 2.10 is the most recent version confirmed to work. This process should also work with 2.8.1 and 2.9.x. Versions 2.11+ are not yet guaranteed to work.

OSPRay source is available in the different releases on their GitHub repository. The most recent version confirmed to work is 1.1.0. Higher versions are not yet confirmed or guaranteed to work. Lower versions will not work as the code uses features not present until 1.1.0.

OSPRay requires ISPC, which is a SIMD compiler. At the time of writing, the highest available version is 1.9.1, which is known to work with OSPRay 1.1.0.

The VisIt+OSPRay source is a modified version of the VisIt code that includes some piping to call OSPRay for rendering. It is currently available as a branch in VisIt's svn repository. Run svn co <the above URL> in /home/alok/visit_ospray/ to get the code.

Notes

In general I prefer out-of-source builds to keep things clean, hence the separate build and src directories in the example structure above. The build_visit script unfortunately does an in-source build by default. This might be configurable by setting a VISIT_BUILD_DIR environment variable. I only recently found this option and have not tested it, however.

Building VisIt (part 1)

Using the build_visit script

First the vanilla VisIt instance needs to be installed using the build_visit script downloaded in the last step. The script will download several tar files and create build directories for each library it needs. You shouldn't need to worry about these. The two directories it creates that are important are visit and visit2.10 (or whichever version you use). The former will contain all installations of the libraries and is referred to as the VISIT_HOME directory. The latter will contain a src directory with the source code and compiled binaries.

It is NOT recommended to use system libraries, even for things like Qt or Python. I have found that system installations are often missing certain options that will break the VisIt build process. I suggest saving yourself the headache and letting VisIt download and install the necessary libraries.

Run the build script with:

./build_visit --qt --vtk --cmake --silo --netcdf --hdf5 --szip --boost --makeflags -j4

If you want to specify a compiler, use the --cc and --cxx flags to specify a C and C++ compiler, respectively. Feel free to specify more or fewer threads for make (the -j4).

This will ask you to agree with Qt's license at the beginning. Building and installing will take some time, but in general I have found this command to be fairly reliable.

Notes

In the directory containing the build_visit script, /home/alok/visit in this case, there will be a cmake file named <hostname>.cmake. This contains the settings for the VISIT_HOME directory mentioned above. You will need this when building the OSPRay-enabled instance of Visit.

Building OSPRay

Untar the ISPC binaries. I like to leave them in the ispc directory that the tar file contains, and keep it in the ospray directory alongside build and src. Untar the source code and move all the contents of OSPRay-1.1.0 into your src directory (i.e. move them up one level). Go to the build directory and run ccmake ../src to configure OSPRay. In ccmake, press c to configure. OSPRay may not find the ISPC executable. The executable is located within the ispc directory from the tar file. I suggest giving an absolute path. Set the following variables:

You can turn off the different example applications. If you would like to verify OSPRay was built correctly, you can leave the volume viewer application enabled. You will need to download a readable dataset to test with, such as the ones at the bottom of this page.

Finally, run make -j 4 for however many threads you like. This may take some time, especially when building and linking the different AVX libraries. Once compilation is done, make sure that you have the files libembree.so and libospray.so.

Troubleshooting

If you don't have a libembree.so shared library after compiling, double check that you set OSPRAY_USE_EXTERNAL_EMBREE to OFF.

Building VisIt (part 2)

Copy the <hostname>.cmake file from the vanilla VisIt build into the visit_ospray/src/config-site directory. You will need to modify it to add four CMake configuration variables:

    VISIT_OPTION_DEFAULT(OSPRAY_DIR /home/alok/ospray/src)
    VISIT_OPTION_DEFAULT(OSPRAY_CMAKE_DIR /home/alok/ospray/src/cmake)
    VISIT_OPTION_DEFAULT(OSPRAY_BUILD_DIR /home/alok/ospray/build)
    VISIT_OPTION_DEFAULT(VISIT_OSPRAY ON TYPE BOOL)
        
I usually place these before the database reader plugin libraries. Of course, replace the directories above with the actual ones you are using. Note that this CMake file has settings for the VISIT_HOME directory. This allows the VisIt+OSPRay build to use the libraries that were installed in the vanilla build.

Change to the visit_ospray build directory and run cmake ../src, then run make with the desired number of threads. This may take some time. If the compile is successful, you should be able to go to the bin/ directory and run the following:

./visit -gui -ospray [-o /your/data/file/here]

Open a volume of your choice and render it. Currently, if the VisIt volume rendering method is set to Splatting, it will use OSPRay instead. OSPRay will also work when rendering a mesh, e.g. with a Pseudocolor plot or a Contour plot.

Troubleshooting

If your compiler gives an error stating it cannot find vtkMultiProcessController.h, you can safely comment out the line that includes it. You can resume compiling after making this change without cleaning.

If you are getting errors about unresolved symbols, you can try modifying a line in your <hostname>.cmake file. Find the two lines setting C/C++ compiler flags. They are usually near the top and commented. In the flags given, remove the -fvisibility flag from both the C and C++ flags. This keeps symbols from being marked as hidden. Rebuild VisIt after making this change.

Notes

Some of the annotations in VisIt do not currently work well with OSPRay. This includes the color map legend, dataset information, and user information. Bounding box and axis/ticks work, as shown in the examples below.

You may sometimes see some white/black lines in the bottom left corner. These are artifacts of the annotations and triad attempting to be rendered. I'm hoping to fix this soon.

On our dual-socket Haswell machine, we typically see 1-2 fps for large volumes when rendering with OSPRay. Note that this is with 8 samples per pixel in OSPRay. A lower sample rate results in faster rendering at the expense of image quality. The frame rate is expected to increase when VisIt moves to VTK 7.x; development for that is underway.

Using Mesa/OpenSWR

VisIt already uses Mesa, but includes a version that does not have OpenSWR built in. The newer version is a drop-in replacement, making it easy to get OpenSWR's performance. If you were able to get Mesa through your package manager, you can skip building. If you downloaded the source, first make sure you have the dependencies listed here. Building Mesa is a simple process using configure. To specifically use OpenSWR, run

configure --disable-dri --disable-egl --enable-xlib-glx --with-gallium-drivers=swrast,swr --prefix=/home/alok/mesa_swr/install/

This command is suggested by OpenSWR. Then, run make and make install.

To use Mesa/OpwnSWR, set the following environment variables before running VisIt.

export LD_LIBRARY_PATH=/home/alok/mesa_swr/install/lib:$LD_LIBRARY_PATH
export GALLIUM_DRIVER=swr

In our tests, Mesa/OpenSWR 13.0 performs better than the Mesa baseline in VisIt's rendering benchmarks. The baseline Mesa tests had a 94.38% pass rate, compared to Mesa/OpenSWR's 94.44%.

Examples

The following show examples of VisIt rendering with OSPRay and Mesa/SWR using various different plots and on different platforms.

VisIt 2.10 with OSPRay 1.1.0 on dual-socket Haswell Xeon


VisIt 2.10 with OSPRay 1.1.2 on KNL
VisIt 2.9.2 with Mesa 17 release candidate 2 (includes SWR) on dual-socket Haswell Xeon