To bring cosmological simulations into the petascale era, we are collaborating with the University of Chicago on the development of PetaART. The Adaptive Refinement Tree (ART) code, a cell-based AMR simuation package, uses a combination of multi-level particle-mesh and shock-capturing Eulerian methods for simulating the evolution of dark matter and gas, respectively. The project involves active participation of an inter-discplinary group from the field of cosmology at the University of Chicago (led by Andrey Kravtsov, Nick Gnedin, and Douglas Rudd) and computer science at Illinois Institute of Technology (led by Zhiling Lan).

The work at Illinois Institute of Technology, in collaboration with the Univ. of Chicago, contains five major research activities: (1) performance analysis of the ART code, (2) development of performance emulator for load balancing study, (3) development of parallel IO, and (4) development of hierarchical task mapping. Task 1 aims to study the performance and scalability issues of the code; Task 2 enables us to study various load balancing schemes without direct, time-consuming code implementation; Task 3 improves parallel IO performance of the cell-based AMR cosmology simulations; Task 4 allows us to improve communication of cosmology simulations by exploiting application communication pattern and the architectural properties of multiprocessor systems. The integrated education activity is to broaden the participation by underrepresented groups, enhance the CS curriculum, and help train the future-generation scientific computing workforce.

The petascale cosmological simulations resulting from this work will lead to major breakthroughs in our understanding of galaxy formation and will provide critical theoretical support for forthcoming large observational surveys designed to probe the matter and energy content of our universe and constrain properties of the mysterious dark matter and dark energy.

Team at Illinois Tech:

Zhiling Lan (faculty)
Jingjin Wu (Ph.D. student, graduated in July 2013)
Yongen Yu (Master student, expected to graduate in Dec. 2013)
Eduardo Berrocal (Ph.D. student)
Li Yu (Ph.D. student)
Xu Yang (Ph.D. student)

Team at UChicago:

A. Kravtsov (Univ. of Chicago)
N. Gnedin (Univ. of Chicago and Fermilab)
D. Rudd (Univ. of Chicago)
Roberto Gonzalez (Univ. of Chicago)

Software Tools:

(Software) LibProfil - a light-weight user-transparent communication profiler. Link
(Software) TCIO - a transparent collective I/O library.
(Software) TOPOMap - a topology aware task mapping library. Link

Key Publications:

J. Wu, Performance Analysis and Optimization of Large-Scale Scientific Applications, Ph.D. Thesis, Illinois Institute of Technology, 2013.
Y. Yu, J. Wu, Z. Lan, D. Rudd, N. Gnedin, and A. Kravtsov, “A Transparent Collective I/O Implementation”, Proc. of IPDPS’13, 2013. [PDF]
J. Wu, X. Xiong, Z. Lan, J. Wang, “Analytical Task Mapping of Scientific Applications on 3D Network Topologies” (research poster), Design Automation Conference (DAC), 2013.
J. Wu, Z. Lan, X. Xiong, N. Gnedin, and A. Kravtsov, “Hierarchical Task Mapping of Cell-based AMR Cosmology Simulations”, Proc. of SC’12, 2012. [PDF]
Y. Yu, D. Rudd, Z. Lan, N. Gnedin, A. Kravtsov, and J. Wu, “Improving Parallel IO Performance of Cell-based AMR Cosmology Applications”, Proc. of IPDPS’12, 2012. [PDF]
J. Wu, R. Gonzalez, Z. Lan, N. Gnedin, A. Kravtsov, D. Rudd, and Y. Yu, “Performance Emulation of Cell-based AMR Cosmology Simulations”, Proc. of Cluster’11, 2011. [PDF]
“Performance Emulation of the Cell-based AMR Cosmology Simulation Code - ART”, DOE SciDAC’11 research poster, 2011. [PDF]

This work is supported by US National Science Foundation.