CS 594-4

Scientific Computing for Engineers:  Spring 2012 – 3 Credits  

This class is part of the Interdisciplinary Graduate Minor in Computational Science. See IGMCS for details.

Wednesdays from 1:30 – 4:15, Room 233 Claxton

Prof. Jack Dongarra with help from Profs. George Bosilca, Jakub Kurzak, Shirley Moore, Stan Tomov, and Vince Weaver

Email: dongarra@eecs.utk.edu

Phone: 865-974-8295

Office hours: Wednesday 11:00 - 1:00, or by appointment

TA:  Blake Haugen, bhaugen@utk.edu

TAs Office : 352 Claxton

 

TAs Office Hours: Wednesdays 10:00 – 12:00 or by appointment

 

There will be four major aspects of the course:

  • Part I will start with current trends in high-end computing systems and environments, and continue with a practical short description on parallel programming with MPI, OpenMP, and Pthreads.

 

  • Part II will illustrate the modeling of problems from physics and engineering in terms of partial differential equations (PDEs), and their numerical discretization using finite difference, finite element, and spectral approximation.

 

  • Part III will be on solvers: both iterative for the solution of sparse problems of part II, and direct for dense matrix problems.  Algorithmic and practical implementation aspects will be covered.

 

  • Finally in Part IV, various software tools will be surveyed and used.  This will include PETSc, Sca/LAPACK, MATLAB, and some tools and techniques for scientific debugging and performance analysis.

 

The grade would be based on homework, a midterm project, a final project, and a final project presentation. Topics for the final project would be flexible according to the student's major area of research.

 

 

Class Roster

If your name is not on the list or some information is incorrect, please send mail to the TA:

 

First Name

Last Name

Email

Department

Chongxiao

Cao

ccao1@utk.edu

CS

Ondrej

Chvala

ochvala@utk.edu

Nuc. Eng.

MohammadReza

Ghanbari

mghanbar@utk.edu

CBE

Ronald

Hagan

hagan@eecs.utk.edu

CS

Jungi

Jeong

jjeong7@utk.edu

CS

Matt

Johnson

mrj@eecs.utk.edu

EECS

Vijay

Joshi

vjoshi1@utk.edu

CS

Khairul

Kabir

kkabir@eecs.utk.edu

CS

John

Lewis

lewisj@alum.rpi.edu

Non-Degree

Xue

Li

xli44@utk.edu

EECS

Max

Mueller

mmuelle4@gmail.com

EECS

Hadi

Nafar

mnafarse@utk.edu

CBE

John

Nelson

jnelso37@utk.edu

CS

Hanieh

Niroomand

snirooma@utk.edu

CBE

Amir

Saadat

asaadat@utk.edu

CBE

Christopher

Stahl

chrstahl89@yahoo.com

Non-Degree

Clay

Taylor

ctaylo56@utk.edu

EECS

Nick

Vinson

nvinson2@utk.edu

CS

Kai

Wang

kwang11@utk.edu

EECS

Stephen

Wood

swood@utk.edu

ESE

Wei

Wu

wwu12@utk.edu

CS

Lonnie

Yu

lyn6@utk.edu

CS

Chunli

Zhang

zhangchunli.pku@gmail.com

Physics

 

 

 

 

 


 

 

 

 

 

Book for the Class:

The Sourcebook of Parallel Computing, Edited by Jack Dongarra, Ian Foster, Geoffrey Fox, William Gropp, Ken Kennedy, Linda Torczon, Andy White, October 2002, 760 pages, ISBN 1-55860-871-0, Morgan Kaufmann Publishers.

 

 

 

 

 

 

 

 

 

 

 

 

 

Lecture Notes: (Tentative outline of the class)

 

  1. January 11 (Dr. Dongarra)

Class Introduction

Introduction to High Performance Computing

Read Chapter 1, 2, and 9

Homework 1 (due January 25, 2012, due date extended to Friday January 27th)

Notes on Homework#1

Tar file of timer

 

  1. January 18 (Dr. Dongarra)

Introduction continued

Dense Linear Algebra

Homework 2 (due February 1, 2012)

Read Chapter 3

Read Chapter 20

 

  1. January 25 (Dr. Bosilca)

Parallel programming paradigms and their performances

Homework 3 (due February 8, 2012)

Read Chapter 21

 

  1.  February 1 (Dr. Bosilca)

Message Passing Interface (MPI)

Read Chapter 11

 

  1. February 8 (Dr. Bosilca)

Message Passing Interface (MPI)

MPI Dynamic Processes

Homework 4 (due February 29, 2012)

 

  1. February 15 (Dr. Weaver)

Performance Evaluation and Tuning

Homework 5 (due February 29, 2012)

Tar file for HW5

 

  1. February 22 (Dr. Bosilca)

MPI-pthreads

RPC

Homework 6 (due March 8th, 2012)

Read Chapter 9

HPC Performance Issues and Systems

Read Chapter 3

 

  1. February 29 (Dr. Moore)

OpenMP

Hybrid MPI/OpenMP programming

Homework 7 (due March 14th, 2012) 

Tar file for HW7

Domain Decomposition Exercise

 

  1. March 7 (Dr. Moore)

Partitioned Global Address Space (PGAS) languages

Homework 8 (due March 28, 2012)

Tar file for HW8

 

  1.  March 14 (Dr. Kurzak)

Accelerators

Quiz

 

March 23 – Spring Break

 

11.  March 28 (Dr. Tomov)

Projection and its importance in scientific computing

GPUs

Homework 9 (due April 11, 2012)

chol_qr_it.m

 

  1.  April 4 (Dr. Tomov)

Discretization of PDEs and Tools for the Parallel Solution of the Resulting Syst

Mesh generation and load balancing

Homework  10 (due April 18, 2012)

hw10.tar.gz

 

  1. April 11 (Dr. Tomov)

Sparse Matrices and Optimized Parallel Implementations

NVIDIA's Compute Unified Device Architecture (CUDA)

Homework 11 (due April 25, 2012)

Read Chapter 20 and 21

 

  1. April 18  (Dr. Tomov)

Iterative Methods in Linear Algebra (Part 1)

Iterative Methods in Linear Algebra (Part 2)

 

Video of  CUDA -- "Better Performance at Lower Occupancy", V. Volkov

Video of OpenCL -- "What is OpenCL"

 

15.  April 25   (No class)

Read Chapter 20

Baileys paper on 12 ways to fool

 

  1. May 2th (Dr. Dongarra)

Class Final reports

The project is to describe and demonstrate what you have learned in class.

The idea is to take an application and implement it on a parallel computer.

Describe what the application is and why this is of importance.

You should describe the parallel implementation, look at the performance,

perhaps compare it to another implementation if possible.

You should write this up in a report, 10-15 pages, and in class you will have

20 minutes to make a presentation.

 

9:00

Chongxiao

Cao

9:20

MohammadReza

Ghanbari

9:40

Ronald

Hagan

10:00

Blake

Haugen

10:20

Jungi

Jeong

Break

11:00

Matt

Johnson

11:20

Vijay

Joshi

11:40

Khairul

Kabir

Lunch

1:00

John

Lewis

1:20

Xue

Li

1:40

Max

Mueller

2:00

Hadi

Nafar

2:20

Amir

Saadat

Break

3:00

Nick

Vinson

3:20

Kai

Wang

3:40

Wei

Wu

4:00

Lonnie

Yu

 

 

 

 

Here are some ideas for projects:

o   Projects and additional projects.

 


Additional Reading Materials


Message Passing Systems

Several implementations of the MPI standard are available today. The most widely used open source MPI implementations are Open MPI and MPICH.

Here is the link to the MPI Forum.


Other useful reference material

    Here are pointers to specs on various processors:

http://www.cpu-world.com/CPUs/index.html

http://www.cpu-world.com/sspec/index.html

http://processorfinder.intel.com

 

             Introduction to message passing systems and parallel computing


J.J. Dongarra, G.E. Fagg, R. Hempl and D. Walker, Chapter in Wiley Encyclopedia of Electrical and Electronics Engineering, October 1999 ( postscript version )

 

``Message Passing Interfaces'', Special issue of Parallel Computing, vol 20(4), April 1994.

 

Ian Foster, Designing and Building Parallel Programs, see http://www-unix.mcs.anl.gov/dbpp/

 

Alice Koniges, ed., Industrial Strength Parallel Computing, ISBN1-55860-540-1, Morgan Kaufmann Publishers, San Francisco, 2000.

 

Ananth Gramma et al., Introduction to Parallel Computing, 2nd edition, Pearson Education Limited, 2003.

 

Michael Quinn, Parallel Programming: Theory and Practice, McGraw-Hill, 1993

 

David E. Culler & Jaswinder Pal Singh, Parallel Computer Architecture, Morgan Kaufmann, 1998, see http://www.cs.berkeley.edu/%7Eculler/book.alpha/index.html

 

George Almasi and Allan Gottlieb, Highly Parallel Computing, Addison Wesley, 1993

 

Matthew Sottile, Timothy Mattson, and Craig Rasmussen, Introduction to Concurrency in Programming Languages, Chapman & Hall, 2010

 

             Other relevant books

 

       Stephen Chapman, Fortran 95/2003 for Scientists and Engineers, McGraw-Hill, 2007

 

       Stephen Chapman, MATLAB Programming for Engineers, Thompson, 2007

 

       Barbara Chapman, Gabriele Jost, Ruud van der Pas, and David J. Kuck, Using OpenMP: Portable Shared Memory Paralllel Programming, MIT Press, 2007

 

      Tarek El-Ghazawi, William Carlson, Thomas Sterling, Katherine Yelick, UPC: Distributed Shared Memory Programming, John Wiley & Sons, 2005

 

       David Bailey, Robert Lucas, Samuel Williams, eds., Performance Tuning of Scientific Applications, Chapman & Hall, 2010

 


Message Passing Standards

``MPI - The Complete Reference, Volume 1, The MPI-1 Core, Second Edition'',
by Marc Snir, Steve Otto, Steven Huss-Lederman, David Walker, Jack Dongarra, MIT Press, September 1998, ISDN 0-262-69215-5.

 

``MPI: The Complete Reference - 2nd Edition: Volume 2 - The MPI-2 Extensions'',
by William Gropp, Steven Huss-Lederman, Andrew Lumsdaine, Ewing Lusk, Bill Nitzberg, William Saphir, and Marc Snir, published by The MIT Press, September, 1998; ISBN 0-262-57123-4.

 

 MPI-2.1 Standard, September 2008

 PDF format:  http://www.mpi-forum.org/docs/mpi21-report.pdf

 Hardcover:  https://fs.hlrs.de/projects/par/mpi//mpi21/

 

 MPI-2.2 Standard, September 2009

 PDF format: http://www.mpi-forum.org/docs/mpi-2.2/mpi22-report.pdf

 Hardcover: https://fs.hlrs.de/projects/par/mpi//mpi22/

 

 


On-line Documentation and Information about Machines

             Overview of Recent Supercomputers, Aad J. van der Steen and Jack J. Dongarra, 2007.

             TOP500 Supercomputer Sites

             Green 500 List of Energy –Efficient Supercomputers

 

 

Other Scientific Computing Information Sites

            Netlib Repository at UTK/ORNL

             BLAS Quick Reference Card

             LAPACK

             ScaLAPACK

             GAMS - Guide to Available Math Software

             Fortran Standards Working Group

             Message Passing Interface (MPI) Forum

             OpenMP

             Unified Parallel C

             DOD High Performance Computing Modernization Program

             DOE Accelerated Strategic Computing Initiative (ASC)

             NSF TeraGrid

             AIST Parallel and High Performance Application Software Exchange (in Japan)

                           (includes information on parallel computing conferences and journals)

             HPCwire

             Supercomputing Online

 

Related On-line Books/Textbooks

   Templates for the Solution of Linear Systems: Building Blocks for Iterative Methods, SIAM Publication, Philadelphia, 1994. 

    LAPACK Users' Guide (Second Edition), SIAM Publications, Philadelphia, 1995.

    Using MPI: Portable Parallel Programming with the Message-Passing Interface by W. Gropp, E. Lusk, and A. Skjellum

     Parallel Computing Works, by G. Fox, R. Williams, and P. Messina (Morgan Kaufmann Publishers)

     Designing and Building Parallel Programs. A dead-tree version of this book is available by Addison-Wesley.

             Introduction to High-Performance Scientific Computing, by Victor Eijkhout with Edmond Chow, Robert Van De Geijn, February 2010

             Introduction to Parallel Computing, by Blaise Barney

 

Performance Analysis Tools Websites

            PAPI

            PerfSuite

            TAU

            Vampir

            Scalasca

            HPCToolkit

            PerfExpert

            mpiP

            ompP

            Open|Speedshop

            IPM

            Eclipse Parallel Tools Platform

Other Online Software and Documentation

  Matlab documentation is available from several sources, most notably by typing ``help'' into the Matlab command window. A primer (for version 4.0/4.1 of Matlab, not too different from the current version) is available in either postscript or pdf.

 SuperLU is a fast implementations of sparse Gaussian elimination for sequential and parallel computers, respectively.

  Sources of test matrices for sparse matrix algorithms

  Matrix Market

  University of Florida Sparse Matrix Collection

  Templates for the solution of linear systems, a collection of iterative methods, with advice on which ones to use. The web site includes on-line versions of the book (in html and postscript) as well as software.

  Templates for the Solution of Algebraic Eigenvalue Problems is a survey of algorithms and software for solving eigenvalue problems. The web site points to an html version of the book, as well as software.

  Updated survey of sparse direct linear equation solvers, by Xiaoye Li

  MGNet is a repository for information and software for Multigrid and Domain Decomposition methods, which are widely used methods for solving linear systems arising from PDEs.

  Resources for Parallel and High Performance Computing

  ACTS (Advanced CompuTational Software) is a set of software tools that make it easier for programmers to write high performance scientific applications for parallel computers.

  PETSc: Portable, Extensible, Toolkit for Scientific Computation 

  Issues related to Computer Arithmetic and Error Analysis

  Efficient software for very high precision floating point arithmetic

  Notes on IEEE Floating Point Arithmetic, by Prof. W. Kahan

  Other notes on arithmetic, error analysis, etc. by Prof. W. Kahan

  Report on arithmetic error that cause the Ariane 5 Rocket Crash  Video of the explosion

  The IEEE floating point standard is currently being updated. To find out what issues the standard committee is considering, look here.

 
 
Jack Dongarra
4/24/2012