Appendix. Comparison of old and new TSUBAME¶
Appendix.1. Architecture¶
| TSUBAME3.0 | TSUBAME4.0 | |
|---|---|---|
| Theoretical peak double precision performance | 12.5PFLOPS | 66.8PFLOPS |
| Theoretical peak half precision performance | 47.2PFLOPS | 952PFLOPS |
| Total main memory | 135TiB | 180TiB |
| Total HDD capacity | 15.9PB | 44.2PB |
| Total SSD capacity | 327TB | |
| Compute nodes | 540 | 240 |
| Total cores | 15,120 | 46,080 |
| Total GPUs | 2,160 | 960 |
| Interconnect | Intel Omni-Path HFI 100Gbps | InfiniBand NDR200 200Gbps |
| Interenet | SINET5 100Gbps | SINET6 100Gbps |
Appendix.2. Compute nodes¶
| TSUBAME3.0 | TSUBAME4.0 | |
|---|---|---|
| Computing Unit | Compute node HPE SGI ICE-XA 540 nodes | Compute node HPE Cray XD665 240 nodes |
| Components (per node) | ||
| CPU | Intel Xeon E5-2680 v4 2.4GHz x 2 Socket | AMD EPYC 9654 2.4GHz x 2 Socket |
| Cores/Threads | 14cores / 28threads x 2CPU | 96cores / 192threads x 2CPU |
| Memory | 256GiB | 768GiB (DDR5-4800) |
| GPU | NVIDIA TESLA P100 for NVlink-Optimized Servers x 4 | NVIDIA H100 SXM5 94GB HBM2e x 4 |
| SSD | 2TB | 1.92TB NVMe U.2 SSD |
| Interconnect | Intel Omni-Path HFI 100Gbps x 4 | InfiniBand NDR200 200Gbps x 4 |
Appendix.3. Software¶
Appendix.3.1 System software¶
| TSUBAME3.0 (as of 2023.4.6) | TSUBAME4.0 | |
|---|---|---|
| OS | SUSE Linux Enterprise Server 12 SP5 | RedHat Enterprise Linux Server 9.3 |
| Job scheduler | Univa Grid Engine 8.6.11 | Altair Grid Engine 2023.1.1 |
| Compilers | GCC 4.8.5, 12.2.0 Intel Compiler 23.0.0 NVIDIA HPC SDK 22.2 |
GCC 11.4.1 Intel oneAPI compiler 2024.0 and MKL NVIDIA HPC SDK 24.1 AOCC 4.1.0 |
| MPI | Intel MPI 21.8.0 SGI MPT 2.16 OpenMPI 3.1.4 |
Intel MPI 2021.11 OpeMPI 5.0.2 |
| CUDA library | 11.0.3 (12.1.0) | 12.3.2 |
| CUDA driver | 450.172.01 | 545.23.08 |
| OmniPath driver (OPA) | 10.10.3.1.1 | |
| InfiniBand driver (OFED) | 23.10-1.1.9 |
Appendix.3.2 Commercial application¶
| Software | Description | TSUBAME3.0 | TSUBAME4.0 |
|---|---|---|---|
| ANSYS | Analysis Software | ○ | ○ |
| ABAQUS | Analysis Software | ○ | ○ |
| ABACUS CAE | Analysis Software | ○ | ○ |
| MSC Nastran | Analysis Software | ○ | |
| MSC Patran | Analysis Software | ○ | |
| Gaussian | Quantum chemistry calculation program | ○ | ○ |
| GaussView | Quantum chemistry calculation program Pre-Post tool | ○ | ○ |
| AMBER | Molecular dynamics calculation program | ○ | ○ |
| Materials Studio | Chemical Simulation Software | ○ | ○ |
| Discovery Studio | Chemical Simulation Software | ○ | ○ |
| Mathematica | Mathematical Processing Software | ○ | ○ |
| Maple | Mathematical Processing Software | ○ | |
| AVS/Express | Visualization | ○ | |
| AVS/Express PCE | Visualization | ○ | |
| LS-DYNA | Analysis Software | ○ | ○ Included in ANSYS |
| COMSOL | Analysis Software | ○ | ○ |
| Schrodinger | Chemical Simulation Software | ○ | ○ |
| MATLAB | Numerical calculation software | ○ | ○ |
| VASP | Quantum molecular dynamics calculation program | ○ | ○ |
| Linaro forge(ex:Arm Forge) | Debugger | ○ | ○ |
| Intel Compiler | Compiler | ○ | ○ oneAPI |
| PGI Compiler | Compiler | ○ | ○ NVIDIA HPC SDK |
Appendix.3.3 Freeware¶
| Software | Description | TSUBAME3.0 | TSUBAME4.0 |
|---|---|---|---|
| GAMESS | Solver Simulator | ○ | ○ |
| Tinker | Solver Simulator | ○ | ○ |
| GROMACS | Solver Simulator | ○ | ○ |
| LAMMPS | Solver Simulator | ○ | ○ |
| NAMD | Solver Simulator | ○ | ○ |
| QUANTUM ESPRESSO | Solver Simulator | ○ | ○ |
| CP2K | Solver Simulator | ○ | ○ |
| OpenFOAM | Solver Simulator, Visualization | ○ | ○ |
| CuDNN | GPU library | ○ | ○ |
| NCCL | GPU library | ○ | ○ |
| Caffe | DeepLearning framework | ○ | |
| Chainer | DeepLearning framework | ○ | |
| TensorFlow | DeepLearning framework | ○ | ○ |
| DeePMD-kit | MD DeepLearning framework | ○ | ○ |
| R | Interpriter (Rmpi,rpud) | ○ | ○ |
| clang | Compiler | ○ | ○ AOCC |
| Apache Hadoop | Distributed Data Processing Tools | ○ | ○ |
| POV-Ray | Visualization | ○ | ○ |
| ParaView | Visualization | ○ | ○ |
| VisIt | Visualization | ○ | ○ |
| turbovnc | Remote GUI(X11) | ○ | ○ |
| gnuplot | Data grapher/visualization | ○ | ○ |
| Tgif | Image Viewing and Editing | ○ | ○ |
| GIMP | Image Viewing and Editing | ○ | ○ |
| ImageMagick | Image Viewing and Editing | ○ | ○ |
| TeX Live | TeX distribution | ○ | ○ |
| Java SDK | Development tool | ○ | ○ |
| PETSc | Linear system solvers, libraries | ○ | ○ |
| FFTW | Fast Fourier Transform Library | ○ | ○ |
| DMTCP | Checkpoint Restart | ○ | ○ |
| Singularity | Linux container for HPC | ○ | ○ Apptainer |
| Open OnDemand | Web portal site for HPC | ○ |
Appendx.4. Storage¶
| TSUBAME3.0 | Storage | Mount point | Capacity | Filesystem |
|---|---|---|---|---|
| Home directory Shared application deployment (HDD) |
/home /apps |
40TB | GPFS+cNFS | |
| Hight-speed storage area 1 (HDD) | /gs/hs0 | 4.8PB | Lustre | |
| Hight-speed storage area 2 (HDD) | /gs/hs1 | 4.8PB | Lustre | |
| Hight-speed storage area 3 (HDD) | /gs/hs2 | 4.8PB | Lustre | |
| Local scratch area (SSD) | /scr | 1.9TB/node | xfs | |
| Shared scratch area | /beeond | Depends on compute node | BeeGFS |
| TSUBAME4.0 | Storage | Mount point | Capacity | Filesystem |
|---|---|---|---|---|
| High-speed storage area (SSD) |
/gs/fs | 372TB | Lustre | |
| Large-scale (Big) storage area Shared application deployment Home directory Work directory (HDD) |
/gs/bs /apps /home /work |
44.2PB | Lustre | |
| Local scratch area (SSD) | /local | 1.62TiB/node | xfs |