GRTeclyn and MHDuet

Numerical Relativity with AMReX

Miren Radia

University of Cambridge

Wednesday 18 June 2025

Introduction

Who are we?

  • I am a Research Software Engineer (RSE) at the University of Cambridge.
  • Previously, I completed a PhD in numerical relativity.
  • I am involved with several numerical relativity codes/projects/collaborations:

GRTeclyn

Katy Clough

Eugene Lim

Juliana Kwan

Paul Shellard

Pau Figueras

Ericka Florio

MHDuet

Miguel Bezares

Carlos Palenzuela

UKNR CCP

Eugene Lim

Katy Clough

Mark Hannam

Geraint Pratten

Patricia Schmidt

Predecessor code: GRChombo

  • Built on the Chombo libraries for “fully-adaptive” block-structured AMR
  • Written in C++14
  • Explicit vectorization through C++ templates to achieve good performance
  • Hybrid MPI + OpenMP parallelization
  • Intended for more exotic problems than most other NR codes:
    • Cosmology: Inflation, cosmic strings, etc.
    • Boson stars/oscillatons/axion stars
    • Modified gravity
    • Higher-dimensional BHs
    • Much more

Predecessor code: MHDuet (SAMRAI version)

  • Automatically generated code using the Simflowny infrastructure
  • Built on SAMRAI
  • MPI parallelism
  • GRMHD with tabulated EoSs and high resolution shock capturing
  • Large eddy simulation techniques with the gradient sub-grid scale model
  • Neutrino transport
  • Mostly used for problems involving compact object binaries:
    • Boson stars
    • Neutron stars
    • Black holes
  • Often used to investigate the behaviour of the magnetic field in these problems

A visualization of the formation of magnetically dominated jets in hypermassive neutron stars produced by a binary merger simulated using MHDueT.

Visualization of four different cases of the coalescence of unequal-mass highly-compact boson stars

Common methods

  • Finite difference discretization
    • Cell-centred (GRChombo)
    • Node-centred (MHDuet)
  • Block-structured AMR
    • Can emulate “moving boxes” for compact object binaries
    • Generally something more adaptive for more exotic spacetimes
    • Can fix some of the coarser grids to avoid regridding noise near GW extraction spheres
    • Arbitrary refinement criterion
  • Method of lines
    • Fourth order Runge-Kutta time integration
    • Fourth or sixth-order spatial discretization
  • Kreiss-Oliger dissipation
  • Boundary conditions:
    • Fourth order interpolation at finer level boundaries from coarser levels
    • Outgoing radiation (Sommerfeld)
    • Symmetric/reflective
    • Extrapolating
  • CCZ4 evolution system
    • Enforcement of algebraic constraints (GRChombo)
    • Damping of algebraic constraints (MHDuet)
  • Moving puncture gauge conditions

Why AMReX?

  • Mature and performant cross-vendor (Nvidia/AMD/Intel) GPU support
  • Similar block-structured AMR capabilities to Chombo1/SAMRAI
  • AMReX previously received significant support as part of the US Exascale Computing Program (ECP).
    • There are many AMReX applications across diverse research areas.
    • Large user community (Slack workspace/GitHub discussions)
  • AMReX is now an established project in the High Performance Software Foundation.
  • Helpful and very active development team:
    • In particular, Weiqun Zhang has been instrumental in helping us to get started.

GRTeclyn

The GRTeclyn code

  • GRTeclyn is an in-development port of GRChombo to AMReX.
  • “Teclyn” is Welsh for “tool”.
  • Features:
    • Uses AMReX’s Amr framework where the management of inter-level operations is handled by AMReX (similar to GRChombo approach)
    • Higher-order spatial interpolation between coarser and finer levels than GRChombo
    • Black-hole binary evolution
    • Designed to be flexible and easy to adapt to new problems
    • Familiar structure for existing GRChombo users
    • Tested and runs well on Nvidia, AMD and Intel GPUs

A bar chart showing the mean walltime taken to evolve a single timestep on various different GPUs and a CPU. The results are as follows: Intel Xeon Platinum 8480 CPU (480s), Nvidia GH200 GPU (56s), AMD MI300X GPU (76s), AMD MI210 GPU (358s), Nvidia A100 GPU (105s) and Intel PVC 1550 (176s).

Recent development

  • Black-hole puncture tracking:
    • More complicated than GRChombo due to data being GPU-resident
    • Uses AMReX’s particle infrastructure (also on GPUs)
    • Typically 2 BHs so just 2 puncture particles so maybe overkill
    • Good experience for future particle use
  • AMD MI300A hackathon:
    • Lots of variables so high register pressure in RHS kernels
    • Legacy complicated data structure from GRChombo increases this
    • Refactoring led to a ~2x speedup in RHS kernels (~40% overall) on AMD MI300A APUs.
  • Support for evolving scalar matter

A plot showing the trajectories of the two punctures in a binary black-hole simulation. The punctures spiral inwards and complete ~10 orbits before merging

A bar chart showing the mean walltime to run a benchmark with the BinaryBH example on an AMD MI300A APU. There are 2 bars, one for before refactoring (103s) and one for after refactoring (60s)

MHDuet

The MHDuet code

  • Support has been added to Simflowny to allow it to generate MHDuet with AMReX as the AMR backend instead of SAMRAI
  • Features
    • Unlike GRTeclyn it uses AMReX’s AmrCore framework so more parts are implemented on the MHDuet side.
    • Custom RK4 time integration
    • Fifth order polynomial interpolation from coarse to fine
    • Uses FUKA/LORENE for compact binary initial data
    • Supports most of the GRMHD features from the SAMRAI version
    • Tested and runs well on Nvidia GPUs

Strong scaling plots for the MHD code with 9 fixed mesh refinement levels

Recent development

  • Addition of nodal syncing when filling boundary nodes to prevent instabilities (only appeared on GPUs).
  • Refactoring of custom higher-order extrapolating BCs to speed up these loops ~3000x on Nvidia A100 GPUs.
  • Reimplementation of reduction calculation to use AMReX ParReduce functions rather than custom code ported from SAMRAI which was neither GPU-offloaded nor threaded/threadsafe.
  • Dependency build scripts are now just Spack environments rather than a collection of custom shell scripts.

Future work

Particles

  • Extend “diagnostic” particles in GRTeclyn to allow high-order interpolation of state variables at arbitrary points in the domain for use with
    • GW extraction on coordinate spheres
    • Apparent horizon finding (~the boundary of black holes)
    • Matter flux around coordinate spheres
  • Add physical particles in GRTeclyn to model collisionless dark matter particles.
  • Add geodesic tracer particles to investigate behaviour around compact objects.
  • MHDueT already uses particles in their old SAMRAI version to model neutrino transport in high-density, hot fluid matter.
    • Port this feature to AMReX version.

An AI generated visualization of a sphere of particles enclosing a black-hole binary emitting gravitational waves.

UKNR CCP Benchmarking

  • The creation of a UKNR CCP would provide long-term funding for our community including shared cross-community code development.
  • As part of the technical component of the initial grant for the creation of this CCP, we will undertake the following:
    • Benchmark and compare the performance of existing NR codes in use by UK researchers, and assess their readiness for future systems.
    • Identify opportunities to make data formats more interoperable and allow codes to share simulation/initial data.
    • Investigate the possibility of developing common [analysis] tools for NR codes.

Any questions?