BlackHoles: Redistribute particles before writing plotfile

SmallDataIO: Fix repeated renaming of APPEND files

Black Holes: Port puncture tracking and enable tagging around the punctures

PR #109 introduced a bug where APPEND files written by `SmallDataIO` are
renamed every time we try to append to them. We only want to rename old
files at the first step.

This fixes #113.

SmallDataIO: Rename old files instead of overwriting

These are currently only used for the punctures.

Switch the type of the device puncture coords array to
amrex::Gpu::DeviceVector. Also move the reduction over levels onto the
GPU.

Note that this only disables the writing of the .dat text file and
doesn't affect the checkpoint or plotfile functions.

This controls the frequency of writing out the punctures.

We lost this in the port to AMReX. I think this is implemented a bit
better than in GRChombo since we use the step number rather than the
time to calculate it.

Note that these checks are sensitive to the grid parameters and may fail
if the configuration means that proper nesting conditions prevent cells
from being refined that were tagged by the puncture tagger.

We could just use post_regrid in derived classes but let's do this for
consistency.

This allows us to add a parameter file to the args that we pass to
amrex::Initialize in tests that require it.

This test sets 1 component of the shift to a constant value and then
tracks two "punctures" which should just move in the opposite direction
to the shift. This is checked for as we know how they should move.

There is puncture tagging in the test but it's not currently checked.

Unfortunately, for the moment, we do have a common
SimulationParameters.hpp file for all the tests.

These can be visualized with ParaView along with the rest of the
plotfile.

Also remove m_is_writing_plotfile as it is no longer used.

This may happen when using bitant symmetry with reflective BCs and
numerical error may lead to the particles just leaving the valid domain
by a very small amount. In this case, AMReX invalidates the particles
and removes them. We want to avoid this hence this change.

Since AMReX might modify the particle id (e.g. if it is invalidated), we
should store the index of the puncture separately in a way that AMReX
won't change.

This requires moving around some of the initialization of the
PunctureTracker class. Also, rather than having a combined
ChiPunctureExtractionTaggingCriterion class, we switch to using the
existing ChiExtractionTaggingCriterion class and then using a separate
PunctureTracker class.

Following AMReX-Codes/amrex#4098 we no longer need to make an alias
MultiFab and use cic_interpolate.

Also make m_puncture_tracker a private member of BHAMR.

Also get the filename and output path from the ParmParse table directly
(in a more AMReX-like way) rather than requiring it to be passed in the
constructor to PunctureTracker.

Move the initialization of PunctureTracker to the BHAMR constructor.

These changes include
* Making PunctureTracker a template class with the template parameter
  being the number of punctures. This allows us to make
  m_puncture_coords a fixed size array.
* Moving the setting of initial_puncture_coords to
  start_from_initial_punctures() rather than initialize() as it's
  irrelevant in the restart case.
* Making PunctureTracker obtain the restart_time from it's GRAMR pointer
  rather than it being needed to be passed to track().
* Moving the updating of m_puncture_coords from the particle locations
  to a separate function which we now also call after restarting from
  the checkpoint file.
* Other minor changes

This is in an effort to make member functions single-purpose and remove
redundant variables.

This removes the need to convert between a 2-index object and a linear
object in the PunctureTracker.

Also remove the tracking of which process has each puncture. We don't
need this info as we set the puncture coords to 0 by default and then
sum reduce over all procs.

This means we only have to do MPI collectives at the end of
execute_tracking rather than in redistribute to figure out which
process has the punctures.

CUDA doesn't allow device lambdas in class member functions that are
private.

This seems to work unlike invoking linear_interpolate_to_particle
directly.

MPI_AllGather doesn't like the send and receive buffers being the same
and we need to redistribute particles before writing to a checkpoint as
they might have moved to a different process.

We know we want to switch to tagging cells based on the puncture
locations but for now this is making debugging the punctures annoying.