Numerical Relativity with AMReX

AMReX Community Breakout, HPSF Conference 2025

Miren Radia

University of Cambridge

May 7, 2025

Introduction

Who are we?

  • I am a Research Software Engineer (RSE) in Research Computing Services, University of Cambridge.
  • Previously, I completed a PhD in numerical relativity.
  • I am involved with several numerical relativity codes/projects/collaborations:

GRTeclyn

Katy Clough

Eugene Lim

Juliana Kwan

Paul Shellard

Pau Figueras

Ericka Florio

MHDueT

Miguel Bezares

Carlos Palenzuela

UKNR CCP

Eugene Lim

Katy Clough

Mark Hannam

Geraint Pratten

Patricia Schmidt

What is Numerical Relativity (NR)?

  • Numerical Relativity is an area of research that uses numerical methods to find and investigate solution to Einstein’s equation of General Relativity (and other modified theories of gravity),

    \[ \underbrace{G_{ab}}_{\text{curvature}} + \underbrace{\Lambda g_{ab}}_{\text{cosmological const.}} = \underbrace{8\pi T_{ab}}_{\text{EM tensor}}. \]

  • We recast the equations into a set of coupled non-linear hyperbolic PDEs with elliptic constraints that we solve for the initial data.

  • Common problems include compact object binaries (black holes, neutron stars, more exotic objects) but NR is also used to study cosmological problems and perform “numerical experiments” for fundamental physics.

  • We are often interested in the gravitational waves (GWs) emitted in a simulation, particularly in the astrophysical compact binary case.

A plot showing a mode of the strain (spacetime distortion) calculated from a binary black-hole simulation

Common methods

  • Discretization:
    • Finite differences
      • Cell-centered
      • Vertex/node-centered
    • Finite volume (for fluids)
    • Spectral methods
    • Discontinuous Galerkin (DG) methods
  • Finite-difference mesh refinement:
    • Block-structured AMR
      • “Moving boxes” for compact object binaries
      • More adaptive for more exotic spacetimes
    • Oct-tree AMR
  • Method of lines
    • Fourth order Runge-Kutta time integration
    • Fourth or sixth-order spatial discretization
  • Kreiss-Oliger dissipation
  • Boundary conditions:
    • Outgoing radiation (Sommerfeld)
    • Symmetric/reflective
    • Extrapolating
    • Multipatch approaches: separate (e.g. spherical) outer grids

NR codes

There are many NR codes/frameworks including (but by no means limited to):

Oct-tree AMR

[Pseudo-]spectral/DG codes

Other

Block-structured AMR

\({\dagger}\) denotes codes/frameworks that are either transitioning to or developing support for using AMReX.

GRTeclyn

The GRTeclyn code

  • GRTeclyn is an in-development port of GRChombo (previously built on the Chombo libraries) to AMReX.
  • “Teclyn” is Welsh for “tool”.
  • Features
    • Amr based
    • Cell-centered but finite-difference rather than finite-volume
      • Fourth-order cell-centered interpolator from coarse to fine (CellQuartic)
      • Fourth-order interpolation from fine to coarse
    • Fourth-order Runge-Kutta time integration
    • Currently, just BinaryBH example which evolves black-hole binaries.
    • Designed to be flexible and easy to adapt to new problems.

A bar chart showing the mean walltime taken to evolve a single timestep on various different GPUs and a CPU. The results are as follows: Intel Xeon Platinum 8480 CPU (480s), Nvidia GH200 GPU (56s), AMD MI300X GPU (76s), AMD MI210 GPU (358s), Nvidia A100 GPU (105s) and Intel PVC 1550 (176s).

Recent development

  • Black-hole puncture tracking:
    • Allows us to track the “centers” of black hole during the simulation.
    • Uses AMReX’s particle infrastructure.
    • Typically 2 BHs so just 2 puncture particles so maybe overkill.
    • Good experience for future particle use.
  • AMD MI300A hackathon:
    • Lots of variables so high register pressure in RHS kernels.
    • Legacy complicated data structure from GRChombo increases this.
    • Refactoring led to a ~2x speedup in RHS kernels (~40% overall) on AMD MI300A APUs.
  • Currently working on adding support for basic matter.

A plot showing the trajectories of the two punctures in a binary black-hole simulation. The punctures spiral inwards and complete ~10 orbits before merging

A bar chart showing the mean walltime to run a benchmark with the BinaryBH example on an AMD MI300A APU. There are 2 bars, one for before refactoring (103s) and one for after refactoring (60s)

MHDueT

The MHDueT code

  • MHDueT is a code that is automatically generated using the Simflowny infrastructure.
  • Previously used the SAMRAI library.
  • Features
    • AmrCore based.
    • Mostly node-centered
      • Fifth order polynomial interpolation from coarse to fine
    • Custom fourth order Runga-Kutta time integration
    • Supports Magnetohydrodynamics (MHD)
      • High resolution shock capturing with WENO reconstruction
      • Arbitrary EOS via HDF5 tabulated input
    • Can simulate binaries of boson stars, neutron stars, black holes and more.
    • Auto-generated using Simflowny infrastructure to enable minimal user programming.

MHDueT logo

A visualization of the formation of magnetically dominated jets in hypermassive neutron stars produced by a binary merger simulated using MHDueT.

Recent development

  • Addition of nodal syncing when filling boundary nodes to prevent instabilities (only appeared on GPUs).
  • Refactoring of custom higher-order extrapolating BCs to speed up these loops ~3000x on Nvidia A100 GPUs.
  • Reimplementation of reduction calculation to use AMReX ParReduce functions rather than custom code ported from SAMRAI which was neither GPU-offloaded nor threaded/threadsafe.
  • Dependency build scripts are now just Spack environments rather than a collection of custom shell scripts.

Future work

Particles

  • Extend “diagnostic” particles in GRTeclyn to allow high-order interpolation of state variables at arbitrary points in the domain for use with
    • GW extraction on coordinate spheres
    • Apparent horizon finding (~the boundary of black holes)
    • Matter flux around coordinate spheres
  • Add physical particles in GRTeclyn to model collisionless dark matter particles.
  • Add geodesic tracer particles to investigate behaviour around compact objects.
  • MHDueT already uses particles in their old SAMRAI version to model neutrino transport in high-density, hot fluid matter.
    • Port this feature to AMReX version.

An AI generated visualization of a sphere of particles enclosing a black-hole binary emitting gravitational waves.

UKNR Collaborative Computational Project

  • We have been awarded a grant to build a roadmap for the creation of a UK Numerical Relativity Collaborative Computational Project/Community (CCP) as part of the Computational Science Centre for Research Communities (CoSeC) programme.
  • This would provide long-term funding for our scientific community including shared cross-community code development.
  • As part of the technical branch of this grant we will:
    • Benchmark the performance of existing NR codes in use by UK researchers, and assess their readiness for future systems.
    • Identify opportunities to make data formats more interoperable and allow codes to share simulation/initial data.
    • Investigate the possibility of developing common [analysis] tools for NR codes.

Any questions?