Presenting the theoretical and practical aspects of parallel numerical algorithms, this book illustrates how iterative algorithms can solve numerical problems and discusses their implementation on parallel and distributed computers. It reviews basic convergence results of iterative algorithms and explores the advantages of asynchronism, such as reducing the effect of bottlenecks and the synchronization penalty. The authors also analyze numerous experiments in various computational contexts for linear and nonlinear numerical problems. The book contains several exercises at the end of most chapters as well as an appendix of mathematical results.