The Neuroscience Institute (NI) and the Center for the Neural Basis of Cognition (CNBC) at Carnegie Mellon University maintain a dedicated computing infrastructure to support the research needs of faculty, staff, and students. These facilities offer secure access to advanced computing equipment, high-speed networking, and a range of software tools essential for modern neuroscience research.
Our server infrastructure is housed in a climate-controlled machine room in the Mellon Institute, with systems protected by redundant power supplies and uninterruptible power sources (UPS), ensuring availability during power outages. Central to our storage capabilities is a Dell PowerEdge R740XD server running Ubuntu Linux, providing 160TB of enterprise-grade disk space configured with RAID 6 and the ZFS file system. This setup includes automatic snapshotting and offsite replication to safeguard research data. A separate 100TB ZFS-based system is dedicated to the MICrONS project (funded by IARPA) and follows similar data protection practices.
For high-performance computing (HPC), the NI leverages a two-rack GPU/CPU cluster (called the MIND cluster) located in the School of Computer Science (SCS) data center. This secure, professionally managed facility is monitored 24/7 by SCS experts who work closely with our NI computing team. The MIND cluster currently consists of 19 compute nodes (3 CPU-only and 16 GPU-accelerated), offering 452 physical CPU cores, 904 logical CPUs, over 6TB of RAM, and 76 GPUs spanning multiple generations and models (including NVIDIA L40S, A5000, RTX 3090, Titan RTX, Titan X, and Titan Xp).
In the spring of 2025, the cluster underwent a series of major infrastructure upgrades to expand capacity and improve performance. A new high-performance GPU node was added, featuring 1TB of RAM and eight NVIDIA L40S GPUs, each with 48GB of memory, along with ultra-fast NVMe SSD storage for improved data throughput. At the same time, all of the CPU-only nodes were replaced with newer, more powerful AMD EPYC-based systems, offering a combined total of 144 physical cores and nearly 3TB of RAM. To increase data transfer speeds across the system, the InfiniBand networking was upgraded to a 100G full bi-directional EDR fabric.
Additional improvements included the installation of 8TB SSD scratch drives on seven GPU nodes, providing fast local storage for temporary data during intensive computations. The RAID storage infrastructure was also expanded and modernized with new 12Gb/s SAS enclosures and additional drive bays, allowing for future scalability. A new head node and a dedicated login node were brought online, and the primary NAS storage server was upgraded with increased memory and processing power. The old NAS server was repurposed as a backup system, replacing legacy hardware dating back to 2012. Finally, the operating system across all nodes was upgraded from CentOS 7 to Springdale 8 (based on CentOS 8), ensuring continued support and compatibility with the latest software environments.
The cluster is managed using SLURM (for job scheduling), Ansible (for system configuration), and Hydra (for user environment setup). Two data partitions use ZFS with RAID 6 for high reliability and are backed up incrementally to ZFS replicated system with some smaller spaces backed up to a secure tape backup system managed by SCS.
Researchers access the cluster via SSH, and available software includes a broad suite of neuroscience and data analysis tools: AFNI, Freesurfer, MNE, SPM, Matlab, R, DSI Studio, TensorFlow, and others. Singularity and Anaconda are also installed, allowing users to customize and isolate their computing environments.
The Computing Center operates as a 100% recharge facility which means user fees cover operational costs and staff support. However, capital improvements (like new hardware purchases) are typically funded by individual researchers through grants or startup funds. As hardware ages and new needs emerge, we depend on users to continue investing in infrastructure to maintain cutting-edge performance.
The computing environment is supported by David Pane, NI’s Manager of Computing, who maintains documentation, provides user support, and manages the overall operation of the facilities. He is available to help with software requests, technical issues, and general guidance related to the computing at the Carnegie Mellon University Neuroscience Institute.