Actually getting the GPU working required patching the Linux kernel to include the open-source AMDGPU driver, which includes Arm support and provides decent support for the RX 460 (Geerling says the card and its Polaris architecture were chosen because they were new enough to be practically useful and to be supported by the AMDGPU driver, old enough that driver support is pretty mature, and because the card is cheap and uses PCIe 3.0). Nvidia's GPUs generally aren't really an option for projects like this because the open source drivers lag far behind the ones available for Radeon GPUs.
Once various kernel patches were applied and the kernel was recompiled, installing AMD's graphics firmware got both graphics output and 3D acceleration working more or less normally.
Despite their age and relative graphical simplicity, running Doom 3 or Tux Racer on the Pi 5's GPU is a tall order, even at 1080p. The RX 460 was able to run both at 4K, albeit with some settings reduced; Geerling also said that the card rendered the Pi operating system's UI smoothly at 4k (the Pi's integrated GPU does support 4K output, but things get framey quickly in our experience, especially when using multiple monitors).
Though a qualified success, anything this hacky is likely to have at least some software problems; Geerling noted that graphics acceleration in the Chromium browser and GPU-accelerated video encoding and decoding support weren't working properly.
Most Pi owners aren't going to want to run out and recreate this setup themselves, but it is interesting to see progress when it comes to using dedicated GPUs with Arm CPUs. So far, Arm chips across all major software ecosystems—including Windows, macOS, and Android—have mostly been restricted to using their own integrated GPUs. But if Arm processors are really going to compete with Intel's and AMD's in every PC market segment, we'll eventually need to see better support for external graphics chips.