Austin's own Advanced Micro Devices (AMD) has most generously donated a number of GPU-enabled servers to UT.
While it is still true that AMD GPUs do not support as many 3rd party applications as NVIDIA, they do support many popular Machine Learning (ML) applications such as TensorFlow, PyTorch, and AlphaFold, and Molecular Dynamics (MD) applications such as GROMACS, all of which are installed and ready for use.
Two BRCF research pods have AMD GPU servers available: the Hopefog and Livestong PODs. Their use is restricted to the groups who own those pods. See Livestrong and Hopefog pod AMD servers for specific information.
The BRCF's AMD GPU pod is available for instructional use and for research use for qualifying UT-Austin affiliated PIs. Allocations are granted to groups who will only perform certain GPU-enabled workflows, not for general computation. To request an allocation, contact us at rctf-support@utexas.edu, and provide the UT EIDs of those who should be granted access.
ROCm is AMD's equivalent to the CUDA framework. ROCm is open source, while CUDA is proprietary.
We have multiple versions of the ROCm framework installed in the /opt directory, designated by a version number extension (e.g. /opt/rocm-5.7.2, /opt/rocm-5.2.3). The default version is the one pointed to by the /opt/rocm symbolic link, which is generally the latest version supported on the specific server.
Livestrong and Hopefog pod AMD GPU servers have MI-50 GPUs (livecomp02/03, hfogcomp02/03), which are now End-Of-Life (no longer supported) according to AMD. As of May 2024, the highest ROCm version supported for the MI-50 GPUs is rocm-5.7.2, which is the last minor version in the ROCm 5.x series. ROCm 5.7.2 is the default for these MI-50 servers, but a lower ROCm version may be selected.
AMD GPU pod servers have MI-100 GPUs (amdgcomp01/02/03), which support the newer ROCm 6.x series. As of July 2025, the ROCm default for these MI-100 servers is currently rocm-6.3.1.
To specify a particular ROCm version other than the default, set the ROCM_PATH environment variable; for example:
export ROCM_PATH=/opt/rocm-5.1.3 |
You may also need to adjust your LD_LIBRARY_PATH as follows:
export LD_LIBRARY_PATH="/opt/rocm-5.1.3/hip/lib:$LD_LIBRARY_PATH" |
The AlphaFold2 protein structure solving software is available on all AMD GPU servers.
The /stor/scratch/AlphaFold directory has the large required database, under the data.4 sub-directory. There is also an AMD example script /stor/scratch/AlphaFold/alphafold_example_amd.sh and an alphafold_example_nvidia.sh script if the POD also has NVIDIA GPUs, (e.g. the Hopefog pod).
On AMD GPU servers, AlphaFold is implemented by a run_alphafold.py Python script inside a Docker image, See the run_alphafold_rocm.sh and run_multimer_rocm.sh scripts under /stor/scratch/AlphaFold for a complete list of options to that script.
AlphaFold requires a number of databases in order to run and several versions of these databases can be found under /stor/scratch/AlphaFold/:
AMD GPU-enabled version of the Molecular Dynamics (MD) GROMACS program is available on all AMD GPU servers, and a CPU-only version is installed also.
The /stor/scratch/GROMACS directory has several useful resources:
All pod compute servers have 3 main Python environments, which are all managed separately (see About Python and JupyterHub server for more information about these environments):
The status of AMD-GPU-enabled versions of TensorFlow and PyTorch working each environment is as follows:
| POD | GPU-enabled PyTorch | GPU-enabled TensorFlow |
|---|---|---|
| AMD GPU |
|
|
| Livestrong |
|
|
| Hopefog |
|
|
The AMD-GPU-specific version of TensorFlow, tensorflow-rocm, is installed on all AMD GPU servers. However, while the MI-100 GPU servers on the AMD GPU pod support the latest tensorflow-rocm version (which requires ROCm 6.3), the MI-50 GPU servers on Livestrong and Hopefog pods requires an older version, tensorflow-rocm 2.13.0.570. Unfortunately the older version does not support Python versions higher than Python 3.10, and both our command-line and JupyterHub use Python 3.12. This is why AMD-GPU-enabled TensorFlow is not supported in JupyterHub on the Livestrong and Hopefog pod MI-50 AMD GPU servers.
On Livestrong and Hopefog pod MI-50 AMD GPU servers, the older version of TensorFlow can still be run from the command line, but is implemented in a globally-accessible miniconda3 environment. So on these MI-50 servers, you must activate the TensorFlow AMD conda first before running any TensorFlow code:
source /stor/scratch/GPU_info/activate_tensorflow_amd_conda |
Once you're done running TensorFlow code, use conda deactivate to exit the conda environment.
On the AMD GPU pod, you can install your own local version of tensorflow-rocm with pip3, e.g.:
pip3 install tensorflow-rocm==2.15.1 |
See https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/3rd-party/tensorflow-install.html for a table showing TensorFlow/ROCm version compatibilities.
If your custom TensorFlow version requires a different ROCm version, you will need to adjust your ROCM_PATH and LD_LIBRARY_PATH, e.g.:
export ROCM_PATH=/opt/rocm-6.2.1 export LD_LIBRARY_PATH="/opt/rocm-6.2.1/hip/lib:$LD_LIBRARY_PATH" |
Two Python scripts are located in /stor/scratch/GPU_info that can be used to ensure you have access to the server's GPUs from TensorFlow or PyTorch. You can run them from the command as shown below:
If GPUs are available and accessible, the output generated should indicate they are being used.
Since there's no batch system on BRCF POD compute servers, it is important for users to monitor their resource usage and that of other users in order to share resources appropriately.
ROCm GPU-enabling framework
Best starting places:
Training Guides