Inside the Message Passing Interface: Creating Fast Communication Libraries

by Alexander Supalov

Overview

A hands-on guide to writing a Message Passing Interface, this book takes the reader on a tour across major MPI implementations, best optimization techniques, application relevant usage hints, and a historical retrospective of the MPI world, all based on a quarter of a century spent inside MPI. Readers will learn to write MPI implementations from scratch, and to design and optimize communication mechanisms using pragmatic subsetting as the guiding principle. Inside the Message Passing Interface also covers MPI quirks and tricks to achieve best performance.

Dr. Alexander Supalov created the Intel Cluster Tools product line, including the Intel MP Library that he designed and led between 2003 and 2015. He invented the common MPICH ABI and also guided Intel efforts in the MPI Forum during the development of the MPI-2.1, MPI-2.2, and MPI-3 standards. Before that, Alexander designed new finite-element mesh-generation methods, contributing to the PARMACS and PARASOL interfaces, and developed the first full MPI-2 and IMPI implementations in the world. He graduated from the Moscow Institute of Physics and Technology in 1990, and earned his PhD in applied mathematics at the Institute of Numerical Mathematics of the Russian Academy of Sciences in 1995. Alexander holds 26 patents (more pending worldwide).

Product Details

ISBN-13:	9781501506789
Publisher:	De G Press
Publication date:	09/24/2018
Sold by:	Barnes & Noble
Format:	eBook
Pages:	384
File size:	20 MB
Note:	This product may take a few minutes to download.
Age Range:	18 Years

About the Author

Dr. Alexander Supalov, Supalov HPC, Germany

Table of Contents

1. Introduction – Learn what expects you inside the book

1. What this book is about
2. Who should read this book
3. Notation and conventions
4. How to read this book

2. Overview

1. Parallel computer

1. Intraprocessor parallelism
2. Interprocessor parallelism
3. Exercises

2. MPI standard

1. MPI history
2. Related standards
3. Exercises

3. MPI subsetting

1. Motivation
2. Typical examples
3. Implementation practice
4. Exercises

3. Shared memory – Learn how to create a simple MPI subset capable of basic blocking point-to-point and collective operations over shared memory

1. Subset definition

1. General assumptions
2. Blocking point-to-point communication
3. Blocking collective operations
4. Exercises

2. Communication mechanisms

1. Basic communication
2. Intraprocess performance
3. Interprocess performance
4. Exercises

3. Startup and termination

1. Process creation

1. Two processes
2. More processes

2. Connection establishment
3. Process termination
4. Exercises

4. Blocking point-to-point communication

1. Limited message length

1. Blocking protocol

2. Unlimited message length

1. Double buffering
2. Eager protocol
3. Rendezvous protocol

3. Exercises

5. Blocking collective operations

1. Naive algorithms
2. Barrier
3. Broadcast
4. Reduce and Allreduce
5. Exercises

4. Sockets – Learn how to create an MPI subset capable of all point-to-point and blocking collective operations over Ethernet and other IP capable networks

1. Subset definition

1. General assumptions
2. Blocking point-to-point communication
3. Nonblocking point-to-point operations
4. Blocking collective operations
5. Exercises

2. Communication mechanisms

1. Basic communication
2. Intranode performance
3. Internode performance
4. Exercises

3. Synchronous progress engine

1. Communication establishment
2. Data transfer
3. Exercises

4. Startup and termination

1. Process creation

1. Startup command
2. Process daemon
3. Out-of-band communication
4. Host name resolution

2. Connection establishment

1. At startup (eager)
2. On request (lazy)

3. Process termination
4. Exercises

5. Blocking point-to-point communication

1. Source and tag matching
2. Unexpected messages
3. Exercises

6. Nonblocking point-to-point communication

1. Request management
2. Exercises

7. Blocking collective operations

1. Communication context
2. Basic algorithms

1. Tree based algorithms
2. Circular algorithms
3. Hypercube algorithms

3. Exercises

5. OFA libfabrics – Learn how to create an MPI subset capable of all point-to-point and collective operations over InfiniBand and upcoming future networks

1. Subset definition

1. General assumptions
2. Point-to-point operations
3. Collective operations
4. Exercises

2. Communication mechanisms

1. Basic communication
2. Intranode performance
3. Internode performance
4. Exercises

3. Startup and termination

1. Process creation
2. Credential exchange
3. Connection establishment
4. Process termination
5. Exercises

4. Point-to-point communication

1. Blocking communication
2. Nonblocking communication
3. Exercises

5. Collective operations

1. Advanced algorithms
2. Blocking operations
3. Nonblocking operations
4. Exercises

6. Advanced features – Learn how to add advanced MPI features including but not limited to heterogeneity, one-sided communication, file I/O, and language bindings

1. Communication modes

1. Standard
2. Buffered
3. Synchronous

2. Heterogeneity

1. Basic datatypes
2. Simple datatypes
3. Derived datatypes
4. Exercises

3. Groups, communicators, topologies

1. Group management
2. Communicator management
3. Process topologies
4. Exercises

4. One-sided communication

1. Mapped implementation
2. Native implementation
3. Exercises

5. File I/O

1. Standard I/O
2. MPI file I/O
3. Exercises

6. Language bindings

1. Fortran
2. C++
3. Java
4. Python
5. Exercises

7. Optimization – Learn how to optimize MPI internally by using advanced implementation techniques and available special hardware

1. Direct data transfer

1. Direct memory access
2. Remote direct memory access
3. Exercises

2. Threads

1. Thread support level
2. Threads as MPI processes
3. Shared memory extensions
4. Exercises

3. Multiple fabrics

1. Synchronous progress engine
2. Asynchronous progress engine
3. Hybrid progress engine
4. Exercises

4. Dedicated hardware

1. Synchronization
2. Special memory
3. Auxiliary networks
4. Exercises

8. Look ahead – Learn to recognize MPI advantages and drawbacks to better assess its future

1. MPI axioms

1. Reliable data transfer
2. Ordered message delivery
3. Dense process rank sequence
4. Exercises

2. MPI-4 en route

1. Fault tolerance
2. Exercises

3. Beyond MPI

1. Exascale challenge
2. Exercises

9. References – Learn about books that may further extend your knowledge

Appendices

1. MPI Families – Learn about major MPI implementation families, their genesis, architecture and relative performance

1. MPICH

1. Genesis
2. Architecture
3. Details

1. MPICH
2. MVAPICH
3. Intel MPI
4. …

4. Exercises

2. OpenMPI

1. Genesis
2. Architecture
3. Details
4. Exercises

3. Comparison

1. Market
2. Features
3. Performance
4. Exercises
1. Alternative interfaces – Learn about other popular interfaces that are used to implement MPI

1. DAPL

1. …
2. Exercises

2. SHMEM

1. …
2. Exercises

3. GasNET

1. …
2. Exercises

4. Portals

1. …
2. Exercises

2. Solutions to all exercises – Learn how to answer all those questions