Title: Efficient Gather-Scatter Operations in NEK5000 using PGAS
Speaker: Niclas Jansson (KTH Stockholm)
Abstract: Gather-scatter operations are one of the key communication kernels used in the computation fluid dynamics (CFD) application Nek5000 for fetching data dependencies (gather), and spreading results to other nodes (scatter). The current implementation used in Nek5000 is the Gather-Scatter library, GS, which utilises different communication strategies: nearest neighbour exchange, message aggregation, and collectives, to efficiently perform communication on a given platform. GS is implemented using non-blocking, two-sided message passing via MPI and the library has proven to scale well to hundreds of thousands of cores. However, the necessity to match sending and receiving messages in the two-sided communication abstraction can quickly
increase latency and synchronisation costs for very fine grained parallelism, in particular for the unstructured communication patterns created by unstructured CFD problems. ExaGS is are-implementation of the Gather-Scatter library, with the intent
to use the best available programming model for a given architecture. We present our current implementation of ExaGS, based on the one-sided programming model provided by the Partitioned Global Address Space (PGAS) abstraction, using Unified Parallel C (UPC). Using a lock-free design with efficient point-to-point synchronisation primitives, ExaGS is able to reduce communication latency compared to the current two-sided MPI implementation. A detailed description of the library and implemented algorithms are given, together with a performance study of ExaGS when used together with Nek5000, and its co-design benchmarking application Nekbone.
Title: Enabling Exascale Fluid Dynamics: Adaptive Mesh Refinement
Speaker: Philipp Schlatter (KTH Stockholm)
Abstract: The complex nature of turbulent fluid flows implies that the computational resources needed to accurately model problems of industrial and academic relevance is virtually unbounded. Computational Fluid Dynamics (CFD) is therefore a natural driver for exascale computing and has the potential for substantial societal impact, like reduced energy consumption, alternative sources of energy, improved health care, and improved climate models. Extreme-scale CFD poses several cross disciplinary challenges e.g. algorithmic issues in scalable solver design, handling of extreme sized data with compression and in-situ analysis, resilience and energy awareness in both hardware and algorithm design. The wide range of topics makes exascale CFD relevant to a wider HPC audience, extending outside the traditional fluid dynamics community. This talk will summarise the work within the EU funded Horizon 2020 project ExaFLOW, which brings together leading CFD experts and users from both industry and academica: Partners include KTH Royal Institute of Technology (Sweden), Imperial College (UK), University of Southampton (UK), University of Edinburgh (UK), University of Stuttgart (Germany), and industrial partners McLaren (UK) and ACSC (Germany). Special focus is on adaptive mesh refinement (AMR), which is identified as one of the key aspects of large-scale, exascale simulations in CFD: The solution of such problems is a-priori unknown, such that the mesh structure necessarily needs to be solution-dependent, and adjust during
the progress of a simulation. In this talk we focus on AMR implemented in Nek5000, a code aimed at direct numerical simulations, based on the spectral element method (SEM). The two main ingredients for AMR are tools for automatic mesh refinement and error estimators which allow for optimal error control. New capabilities in Nek5000 enable the use of the h-refinement method for mesh adaptation, where selected elements are
split via quadtree (2D) or octree (3D) structures. Two methods are considred for estimating the error. The first method is local and based on the spectral properties of the solution on each element. These so-called spectral error indicators come with a low overhead and are easily implemented but they provide only a local measure of the error. The second method, on the other hand, is goal-oriented and takes into account both the local properties of the solution and the global dependence of the error in a functional of interest (such as drag or lift) via the resolution of an adjoint problem. These so-called adjoint error estimators are based on a similar work done within the framework
of the finite element method. AMR capabilities are demonstrated in Nek5000 and applied to simple steady and unsteady test cases, such as the flow past a cylinder and the
lid-driven cavity, in two and three dimensions. The definition and implementation of both error estimation methods are presented and discussed. The results obtained with both methods are compared and are shown to efficiently reduce the number of degrees of freedom required to reach a given tolerance on the solution compared to conforming refinement. Moreover, the gains in terms of mesh generation, accuracy and computational cost are discussed by analyzing the convergence of some functional of interest and the evolution of the mesh as refinement proceeds.