High-Performance Computing

The Gauß-IT-Zentrum operates local high-performance computers to cover the basic needs of university members for HPC capacity and functionality. For requirements beyond this, it operates additional high-performance computers for the university institutes, which are financed by the institutes using them. For this purpose, the Gauss IT Center carries out the installation and operation of the computers and provides user support within the same framework as for the systems procured itself.

In order to obtain the greatest possible benefit from the procured systems, the individual systems are to be included in a local grid network in order to absorb additional overflow capacity and balance out usage peaks.

Use by the user

• Contact the Gauß-IT-Zentrum as soon as funding for HLR procurement is forthcoming, preferably prior to application.
•  Arrangement and planning of the installation in the machine hall Campus North.

Boundary conditions and requirements

• The computers are located in the north campus machine room and are subject to the conditions of the infrastructure there.
• The assumption of follow-up costs (electricity, etc.) must be clarified with GB3 in each individual case.
• The personnel capacities required for the support are to be reimbursed by the institute.
• The details of the service are to be coordinated with the institute.
• Free computing capacities can be allocated to other users by the Gauß-IT-Zentrum.
• The system should be integrated into a local grid network.

• Performance: 244.5 TFLOPS (CPU) + 339.2 TFLOPS (GPU)
• Number of nodes: 304 compute nodes, 8 FAT nodes, 8 GPU nodes, 6 visualization nodes
• Storage (BeeGFS): ~576TB
• Bandwidth: 10 GBit/s Ethernet + 58 GBit/s Infiniband

The application and procurement of the DFG large-scale equipment was lead-managed by the Institute for Computational Modeling in Civil Engineering (IRMB) with the support of the Gauss IT Center. Furthermore, the Institutes of Joining and Welding Technology, Scientific Computing, Fluid Mechanics, Flight Propulsion and Fluid Machinery, Particle Technology, Design Engineering, Applied Mechanics, Medicinal and Pharmaceutical Chemistry, Thermodynamics, and Energy and Systems Process Engineering were involved in the application and financing of the system. The administration of the system is carried out jointly by IRMB and Gauss IT Center. In addition, the GITZ supports and advises the users in the further development of their parallel research software.

To apply for a user account for the Phoenix, fill out the corresponding form and send it scanned as a pdf via email to phoenix-support@tu-braunschweig.de. Please also explicitly include the user ID and e-mail address in the e-mail.

User Application

"This work used the supercomputer Phoenix and was supported by the Gauß-IT-Zentrum of the University of Braunschweig (GITZ). We are grateful to the GITZ supercomputer staff, especially [name of the person]."

Inspired by: https://www.hlrn.de/doc/display/PUB/Acknowledge+HLRN

Contact

For problems, questions, suggestions for improvement and general support you can reach us at: phoenix-support(at)tu-braunschweig.de.

Ticketsystem

Your requests are recorded directly as a ticket in our central ticket system (OTRS) and can then be processed by our employees. You can track the status of your tickets here.

In our Doku-Wiki you can find more information about using the cluster. An english version is avaiable at the end of this site.

Server/Node (Details)

This is a detailed list of all nodes and servers installed in the Phoenix.

2x administration server: (s[01-02] / XCAT + Clustware + NFS (Pacemaker) ):

• CPU: 2x CPU INTEL Xeon E5-2640v4
• RAM: 8x DDR4-2400 8GB ECC (Samsung M393A1G43EB1-CRC)
• Hard drives: 2x Seagate 2TB ST2000NM0033 (System / RAID1), 2x Samsung SSD SM863 240GB (/var/log / RAID1)
• Motherboard: Supermicro X10DRH-C (Geh. Supermicro 825TQ-R740LPB)
• Interconnect: eth1 onboard 1GB (corosync), eth2 10 GB Ethernet (Internes Clusternetz), eth3 10 GB Ethernet (Außenanbindung GITZ), ipmi dediziert, ib0 Intel Omni-Path HFI Adapter 100 Series (Infinibandnetzwerk)

1x External RAID-System for administration server:

• Model: Infortrend ESDS-1012RC
• Hard drives: 2x HDD Seagate ST4000NM0115 4TB (Controller Software RAID 1)

2x Access nodes: (login[01-02]):

• CPU: 2 x CPU INTEL Xeon E5-2640v4
• RAM: 8 x DDR4-2400 8GB ECC (Samsung M393A1G43EB1-CRC)
• Hard drives: 2x Micron SSD 5100 240GB 2,5" ( System / RAID1), 2x HDD Toshiba 1TB MG03ACA100 (/scratch / RAID1)
• Motherboard: Supermicro X10DRT-H (Supermicro BB 6028TR-D72R)
• Interconnect: eth0 onboard 1GB Ethernet (service),  eth1 onboard 1GB Ethernet (ungenutzt), eth2 10 GB Ethernet (External connection GITZ), ipmi dediziert, ib0 Intel Omni-Path HFI Adapter 100 Series (Infinibandnetwork)

2x Metadaten-Server: (mds[01-02])

• CPU: 1x INTEL Xeon E5-1630v4
• RAM: 8x DDR4-2133 16GB (Samsung M393A2G40DB0-CPB)
• Hard drives: 2x Samsung SSD SM863 240GB (system / RAID 1), 11x Samsung SSD SM863A 480GB (/beegfs/meta / Raid 10 + HS)
• Motherboard: Supermicro X10SRH-CLN4F (Supermicro BB 5028R-E1CR12L)
• Interconnect: eth0 onboard 1GB Ethernet (service), eth1 onboard 1GB Ethernet (unused), ipmi dedicated, ib0 Intel Omni-Path HFI Adapter 100 Series (Infinibandnetzwerk)

2x Object-Storage-Server: (ods[01-02] Pacemaker)

• CPU: 2x CPU INTEL Xeon E5-2630v4
• RAM: 8x DDR4-2400 32GB (Samsung M393A4K40BB1-CRC)
• Hard drives: 2x Samsung SSD SM863 240GB (system / RAID 1)
• Motherboard: Supermicro X10DRT-H (Supermicro BB 6028TR-D72R)
• Interconnect: eth0 onboard 1GB Ethernet (service), eth1 onboard 1GB Ethernet (corosync), ib0 Intel Omni-Path HFI Adapter 100 Series (Infinibandnetwork)

2x MEGWARE Object-Storage-Target:

• Model: HGST JBOD 4U60 Storage
• Hard drives: 60x 6TB SAS 7200 (ZFS 4×14 Raidz2 + 4 HS)

304x Rechenknoten: (node[001-304]):

• CPU: 2x CPU INTEL Xeon E5-2640v4
• RAM: 8 x DDR4-2400 8GB
• Hard drives: diskless
• Motherboard: Intel S2600KPR (Module HNS2600KPR / Geh H2000G)
• Interconnect: eth0 onboard 1GB Ethernet (service), eth1 onboard 1GB Ethernet (ungenutzt), ib0 Intel Omni-Path HFI Adapter 100 Series (Infinibandnetwork)

8x fat nodes(fat[001-008]):

• CPU: 2x CPU INTEL Xeon E5-2640v4, RAM8 x DDR4-2400 32GB
• Hard drives: diskless
• Motherboard. Intel S2600KPR (Module HNS2600KPR / Geh H2000G)
• Interconnect: eth0 onboard 1GB Ethernet (service), eth1 onboard 1GB Ethernet (unused), ipmi dedicted, ib0 Intel Omni-Path HFI Adapter 100 Series (Infinibandnetwork)

8x gpu nodes (gpu[01-08]):

• CPU: 2x CPU INTEL Xeon E5-2640v4
• RAM: 8 x DDR4-2400 8GB (Samsung M393A1G40EB1-CRC)
• Hard drives: diskless
• Motherboard: Intel X10DGQ (Supermicro BB 1028GQ-TXR)
• GPU: 4x NVIDIA Tesla P100 16GB HBM2
• Interconnect: eth0 onboard 1GB Ethernet (service), eth1 onboard 1GB Ethernet (unused), ipmi dedicated, ib0 Intel Omni-Path HFI Adapter 100 Series (Infinibandnetwork)

6x Visualization Nodes (vis[01-06]):

• CPU: 2x CPU INTEL Xeon E5-2640v4
• RAM:16x DDR4-2400 32GB
• Hard drives: 1x HDD Seagate ST4000NM0115 4TB (system)
• Motherboard: SuperMicro X10DRU-i+ (Supermicro BB 6028U-TR4+)
• GPU: GK PNY Geforce GTX1080 8GB
• Interconnect: eth0 onboard 1GB Ethernet (service), eth1 onboard 1GB Ethernet (unused), ipmi dedicated, ib0 Intel Omni-Path HFI Adapter 100 Series (Infinibandnetwork)

A detailed list of installed software can be found on the corresponding page in our manuals.

Since the installed software changes regularly in terms of both number and version, the list may not be complete.

You are welcome to send us a request for this to phoenix-support(at)tu-bs.de.