2015
Heitmann, Katrin; Frontiere, Nicholas; Sewell, Christopher; Habib, Salman; Pope, Adrian; Finkel, Hal; Rizzi, Silvio; Insley, Joe; Bhattacharya, Suman
The Q Continuum Simulation: Harnessing the Power of GPU Accelerated Supercomputers Journal Article
In: 2015, (LA-UR-15-28271).
Abstract | Links | BibTeX | Tags: cosmology, gpu, n-body
@article{Heitmann:2015a,
title = {The Q Continuum Simulation: Harnessing the Power of GPU Accelerated Supercomputers},
author = {Katrin Heitmann and Nicholas Frontiere and Christopher Sewell and Salman Habib and Adrian Pope and Hal Finkel and Silvio Rizzi and Joe Insley and Suman Bhattacharya},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/11/TheQContinuumSimulationHarnessingThePowerOfGPUAcceleratedSupercomputers2.pdf},
year = {2015},
date = {2015-01-01},
publisher = {To appear in The Astrophysical Journal},
abstract = {Modeling large-scale sky survey observations is a key driver for the continuing development of high resolution, large-volume, cosmological simulations. We report the first results from the 'Q Continuum' cosmological N-body simulation run carried out on the GPU-accelerated supercomputer Titan. The simulation encompasses a volume of (1300 Mpc)^3 and evolves more than half a trillion particles, leading to a particle mass resolution of ~1.5 X 10^8 M_sun. At this mass resolution, the Q Continuum run is currently the largest cosmology simulation available. It enables the construction of detailed synthetic sky catalogs, encompassing different modeling methodologies, including semi-analytic modeling and sub-halo abundance matching in a large, cosmological volume. Here we describe the simulation and outputs in detail and present first results for a range of cosmological statistics, such as mass power spectra, halo mass functions, and halo mass-concentration relations for different epochs. We also provide details on challenges connected to running a simulation on almost 90% of Titan, one of the fastest supercomputers in the world, including our usage of Titan's GPU accelerators.},
note = {LA-UR-15-28271},
keywords = {cosmology, gpu, n-body},
pubstate = {published},
tppubtype = {article}
}
Modeling large-scale sky survey observations is a key driver for the continuing development of high resolution, large-volume, cosmological simulations. We report the first results from the 'Q Continuum' cosmological N-body simulation run carried out on the GPU-accelerated supercomputer Titan. The simulation encompasses a volume of (1300 Mpc)^3 and evolves more than half a trillion particles, leading to a particle mass resolution of ~1.5 X 10^8 M_sun. At this mass resolution, the Q Continuum run is currently the largest cosmology simulation available. It enables the construction of detailed synthetic sky catalogs, encompassing different modeling methodologies, including semi-analytic modeling and sub-halo abundance matching in a large, cosmological volume. Here we describe the simulation and outputs in detail and present first results for a range of cosmological statistics, such as mass power spectra, halo mass functions, and halo mass-concentration relations for different epochs. We also provide details on challenges connected to running a simulation on almost 90% of Titan, one of the fastest supercomputers in the world, including our usage of Titan's GPU accelerators.
Sewell, Christopher; Lo, Li-Ta; Heitmann, Katrin; Habib, Salman; Ahrens, James
Utilizing Many-Core Accelerators for Halo and Center Finding within a Cosmology Simulation Proceedings Article
In: Proceedings of the IEEE Symposium on Large Data Analysis and Visualization, IEEE Press, Chicago, Illinois, 2015, (LA-UR-15-22202).
Abstract | Links | BibTeX | Tags: cosmology, halo finding, many-core, Programming Techniques
@inproceedings{Sewell:2015a,
title = {Utilizing Many-Core Accelerators for Halo and Center Finding within a Cosmology Simulation},
author = {Christopher Sewell and Li-Ta Lo and Katrin Heitmann and Salman Habib and James Ahrens},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/UtilizingMany-CoreAcceleratorsForHaloAndCenterFindingWithinACosmologySimulation.pdf},
year = {2015},
date = {2015-01-01},
booktitle = {Proceedings of the IEEE Symposium on Large Data Analysis and Visualization},
publisher = {IEEE Press},
address = {Chicago, Illinois},
series = {LDAV '15},
abstract = {Efficiently finding and computing statistics about “halos” (regions of high density) are essential analysis steps for N-body cosmology simulations. However, in state-of-the-art simulation codes, these analysis operators do not currently take advantage of the shared- memory data-parallelism available on multi-core and many-core ar- chitectures. The Hybrid / Hardware Accelerated Cosmology Code (HACC) is designed as an MPI+X code, but the analysis operators are parallelized only among MPI ranks, because of the difficulty in porting different X implementations (e.g., OpenMP, CUDA) across all architectures on which it is run. In this paper, we present portable data-parallel algorithms for several variations of halo find- ing and halo center finding algorithms. These are implemented with the PISTON component of the VTK-m framework, which uses Nvidia’s Thrust library to construct data-parallel algorithms that al- low a single implementation to be compiled to multiple backends to target a variety of multi-core and many-core architectures. Fi- nally, we compare the performance of our halo and center find- ing algorithms against the original HACC implementations on the Moonlight, Stampede, and Titan supercomputers. The portability of Thrust allowed the same code to run efficiently on each of these architectures. On Titan, the performance improvements using our code have enabled halo analysis to be performed on a very large data set (81923 particles across 16,384 nodes of Titan) for which analysis using only the existing CPU algorithms was not feasible.},
note = {LA-UR-15-22202},
keywords = {cosmology, halo finding, many-core, Programming Techniques},
pubstate = {published},
tppubtype = {inproceedings}
}
Efficiently finding and computing statistics about “halos” (regions of high density) are essential analysis steps for N-body cosmology simulations. However, in state-of-the-art simulation codes, these analysis operators do not currently take advantage of the shared- memory data-parallelism available on multi-core and many-core ar- chitectures. The Hybrid / Hardware Accelerated Cosmology Code (HACC) is designed as an MPI+X code, but the analysis operators are parallelized only among MPI ranks, because of the difficulty in porting different X implementations (e.g., OpenMP, CUDA) across all architectures on which it is run. In this paper, we present portable data-parallel algorithms for several variations of halo find- ing and halo center finding algorithms. These are implemented with the PISTON component of the VTK-m framework, which uses Nvidia’s Thrust library to construct data-parallel algorithms that al- low a single implementation to be compiled to multiple backends to target a variety of multi-core and many-core architectures. Fi- nally, we compare the performance of our halo and center find- ing algorithms against the original HACC implementations on the Moonlight, Stampede, and Titan supercomputers. The portability of Thrust allowed the same code to run efficiently on each of these architectures. On Titan, the performance improvements using our code have enabled halo analysis to be performed on a very large data set (81923 particles across 16,384 nodes of Titan) for which analysis using only the existing CPU algorithms was not feasible.
2014
Sewell, Christopher
Streaming Data-Parallel Algorithms Enable Cosmology Data Analysis for Large Halos Presentation
31.12.2014, (LA-UR-14-29638).
Abstract | Links | BibTeX | Tags: cosmology, data parallel
@misc{Sewell2014,
title = {Streaming Data-Parallel Algorithms Enable Cosmology Data Analysis for Large Halos},
author = {Christopher Sewell },
url = {http://datascience.dsscale.org/wp-content/uploads/2016/08/Streaming_Data-Parallel_Algorithms_Enable_Cosmology_Data_Analysis_for_Large_Halos.pdf},
year = {2014},
date = {2014-12-31},
abstract = {This presentation given by Christopher Sewell describes how streaming data-parallel algorithms have enabled cosmology data analysis for large halos.},
note = {LA-UR-14-29638},
keywords = {cosmology, data parallel},
pubstate = {published},
tppubtype = {presentation}
}
This presentation given by Christopher Sewell describes how streaming data-parallel algorithms have enabled cosmology data analysis for large halos.
Sewell, Christopher; Ahrens, James; Patchett, John
New Data-parallel Algorithms Accelerate Cosmology Data Analysis on GPUs Presentation
30.06.2014, (LA-UR-14-22054).
Abstract | Links | BibTeX | Tags: cosmo, cosmology, data parallel, gpu
@misc{Sewell2014b,
title = {New Data-parallel Algorithms Accelerate Cosmology Data Analysis on GPUs},
author = {Christopher Sewell and James Ahrens and John Patchett},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/08/New_Data-parallel_Algorithms_Accelerate_Cosmology_Data_Analysis_on_GPUs.pdf},
year = {2014},
date = {2014-06-30},
abstract = {This presentation describes how new data-parallel algorithms have accelerated cosmology data analysis on GPUs.},
note = {LA-UR-14-22054},
keywords = {cosmo, cosmology, data parallel, gpu},
pubstate = {published},
tppubtype = {presentation}
}
This presentation describes how new data-parallel algorithms have accelerated cosmology data analysis on GPUs.
2013
Patchett, John
Applications of In Situ Visualization for Ocean, Cosmology, and Plasma Presentation
20.02.2013, (LA-UR-13-21112).
Abstract | Links | BibTeX | Tags: cosmology, in situ, oceanography simulation and modeling, plasma
@misc{Patchett2013b,
title = {Applications of In Situ Visualization for Ocean, Cosmology, and Plasma},
author = {John Patchett},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/08/Applications_of_In_Situ_Visualization_for_Ocean_Cosmology_and_Plasma.pdf},
year = {2013},
date = {2013-02-20},
abstract = {This is a five minute or less talk for the Office of Science SDAV All Hands Meeting on 2/20/2013. It describes our work with three domains of science: ocean modeling (POP), cosmology(HACC), and plasma(VPIC). In particular it presents work that was directly related to in situ analysis and our future work with these models under SDAV.},
note = {LA-UR-13-21112},
keywords = {cosmology, in situ, oceanography simulation and modeling, plasma},
pubstate = {published},
tppubtype = {presentation}
}
This is a five minute or less talk for the Office of Science SDAV All Hands Meeting on 2/20/2013. It describes our work with three domains of science: ocean modeling (POP), cosmology(HACC), and plasma(VPIC). In particular it presents work that was directly related to in situ analysis and our future work with these models under SDAV.
2011
Woodring, Jonathan; Heitmann, Katrin; Ahrens, James; Fasel, Patricia; Hsu, Chung-Hsing; Habib, Salman; Pope, Adrian
Analyzing and visualizing cosmological simulations with ParaView Journal Article
In: The Astrophysical Journal Supplement Series, vol. 195, no. 1, pp. 11, 2011, (LA-UR-10-06301).
Abstract | Links | BibTeX | Tags: cosmology, large-scale structure of universe, numerical methods, ParaView
@article{woodring2011analyzing,
title = {Analyzing and visualizing cosmological simulations with ParaView},
author = {Jonathan Woodring and Katrin Heitmann and James Ahrens and Patricia Fasel and Chung-Hsing Hsu and Salman Habib and Adrian Pope},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/AnalyzingAndVisualizingCosmologicalSimulationsWithParaView.pdf},
year = {2011},
date = {2011-01-01},
journal = {The Astrophysical Journal Supplement Series},
volume = {195},
number = {1},
pages = {11},
publisher = {IOP Publishing},
abstract = {The advent of large cosmological sky surveys – ushering in the era of precision cosmology – has been accompanied by ever larger cosmological simulations. The analysis of these simulations, which currently encompass tens of billions of particles and up to trillion particles in the near future, is often as daunting as carrying out the simulations in the first place. Therefore, the development of very efficient analysis tools combining qualitative and quantitative capabilities is a matter of some urgency. In this paper we introduce new analysis features implemented within ParaView, a parallel, open-source visualization toolkit, to analyze large N-body simulations. The new features include particle readers and a very efficient halo finder which identifies friends-of-friends halos and determines common halo properties. In combination with many other functionalities already existing within ParaView, such as histogram routines or interfaces to Python, this enhanced version enables fast, interactive, and convenient analyses of large cosmological simulations. In addition, development paths are available for future extensions.},
note = {LA-UR-10-06301},
keywords = {cosmology, large-scale structure of universe, numerical methods, ParaView},
pubstate = {published},
tppubtype = {article}
}
The advent of large cosmological sky surveys – ushering in the era of precision cosmology – has been accompanied by ever larger cosmological simulations. The analysis of these simulations, which currently encompass tens of billions of particles and up to trillion particles in the near future, is often as daunting as carrying out the simulations in the first place. Therefore, the development of very efficient analysis tools combining qualitative and quantitative capabilities is a matter of some urgency. In this paper we introduce new analysis features implemented within ParaView, a parallel, open-source visualization toolkit, to analyze large N-body simulations. The new features include particle readers and a very efficient halo finder which identifies friends-of-friends halos and determines common halo properties. In combination with many other functionalities already existing within ParaView, such as histogram routines or interfaces to Python, this enhanced version enables fast, interactive, and convenient analyses of large cosmological simulations. In addition, development paths are available for future extensions.
2009
Habib, Salman; Pope, Adrian; Lukic, Zarija; Daniel, David; Fasel, Patricia; Desai, Nehal; Heitmann, Katrin; Hsu, Chung-Hsing; Ankeny, Lee; Mark, Graham
Hybrid petacomputing meets cosmology: The Roadrunner Universe project Proceedings Article
In: Journal of Physics: Conference Series, pp. 012019, IOP Publishing 2009, (LA-UR-09-03785).
Abstract | Links | BibTeX | Tags: cosmology, petacomputing
@inproceedings{habib2009hybrid,
title = {Hybrid petacomputing meets cosmology: The Roadrunner Universe project},
author = {Salman Habib and Adrian Pope and Zarija Lukic and David Daniel and Patricia Fasel and Nehal Desai and Katrin Heitmann and Chung-Hsing Hsu and Lee Ankeny and Graham Mark},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/HybridPetacomputingMeetsCosmologyTheRoadrunnerUniverseProject.pdf},
year = {2009},
date = {2009-01-01},
booktitle = {Journal of Physics: Conference Series},
volume = {180},
number = {1},
pages = {012019},
organization = {IOP Publishing},
abstract = {Over the last two decades, critical observational advances in large-volume sky surveys carried out over a wide range of wavelengths, as well as over short time cadences, have revolutionized cosmology. Computational cosmology has emerged as an essential resource for providing detailed predictions for these observations, essential data for assisting in their design, and sophisticated tools for interpreting the final results.},
note = {LA-UR-09-03785},
keywords = {cosmology, petacomputing},
pubstate = {published},
tppubtype = {inproceedings}
}
Over the last two decades, critical observational advances in large-volume sky surveys carried out over a wide range of wavelengths, as well as over short time cadences, have revolutionized cosmology. Computational cosmology has emerged as an essential resource for providing detailed predictions for these observations, essential data for assisting in their design, and sophisticated tools for interpreting the final results.
2008
Anderson, Erik; Silva, Claudio T; Ahrens, James; Heitmann, Katrin; Habib, Salman
Provenance in comparative analysis: A study in cosmology Journal Article
In: Computing in Science & Engineering, vol. 10, no. 3, pp. 30–37, 2008, (LA-UR-08-02608).
Abstract | Links | BibTeX | Tags: Comparative Analysis, cosmology, Provenance
@article{anderson2008provenance,
title = {Provenance in comparative analysis: A study in cosmology},
author = {Erik Anderson and Claudio T Silva and James Ahrens and Katrin Heitmann and Salman Habib},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/ProvenanceInComparativeAnalysisAStudyInCosmology.pdf},
year = {2008},
date = {2008-01-01},
journal = {Computing in Science & Engineering},
volume = {10},
number = {3},
pages = {30--37},
publisher = {AIP Publishing},
abstract = {Provenance—the logging of information about how data came into being and how it was processed—is an essential aspect of managing large-scale simulation and data-intensive projects. Using a cosmology code comparison project as an example, this article presents how a provenance system can play a key role in such applications.},
note = {LA-UR-08-02608},
keywords = {Comparative Analysis, cosmology, Provenance},
pubstate = {published},
tppubtype = {article}
}
Provenance—the logging of information about how data came into being and how it was processed—is an essential aspect of managing large-scale simulation and data-intensive projects. Using a cosmology code comparison project as an example, this article presents how a provenance system can play a key role in such applications.
Heitmann, Katrin; Lukic, Zarija; Fasel, Patricia; Habib, Salman; Warren, Michael S.; White, Martin; Ahrens, James; Ankeny, Lee; Armstrong, Ryan; O’Shea, Brian; Ricker, Paul M.; Springel, Volker; Stadel, Joachim; Trac, Hy
The cosmic code comparison project Journal Article
In: Computational Science & Discovery, vol. 1, no. 1, pp. 015003, 2008, (LA-UR-07-1953).
Abstract | Links | BibTeX | Tags: cosmology
@article{heitmann2008cosmic,
title = {The cosmic code comparison project},
author = {Katrin Heitmann and Zarija Lukic and Patricia Fasel and Salman Habib and Michael S. Warren and Martin White and James Ahrens and Lee Ankeny and Ryan Armstrong and Brian O’Shea and Paul M. Ricker and Volker Springel and Joachim Stadel and Hy Trac},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/TheCosmicCodeComparisonProject.pdf},
year = {2008},
date = {2008-01-01},
journal = {Computational Science & Discovery},
volume = {1},
number = {1},
pages = {015003},
publisher = {IOP Publishing},
abstract = {Current and upcoming cosmological observations allow us to probe structures on smaller and smaller scales, entering highly nonlinear regimes. In order to obtain theoretical predictions in these regimes, large cosmological simulations have to be carried out. The promised high accuracy from observations make the simulation task very demanding: the simulations have to be at least as accurate as the observations. This requirement can only be ful lled by carrying out an extensive code validation program. The rst step of such a program is the comparison of diㄦent cosmology codes including gravitation interactions only. In this paper we extend a recently carried out code comparison project to include five more simulation codes. We restrict our analysis to a small cosmological volume which allows us to investigate properties of halos. For the matter power spectrum and the mass function, the previous results hold, with the codes agreeing at the 10% level over wide dynamic ranges. We extend our analysis to the comparison of halo profiles and investigate the halo count as a function of local density. We introduce and discuss ParaView as a exible analysis tool for cosmological simulations, the use of which immensely simpli es the code comparison task.},
note = {LA-UR-07-1953},
keywords = {cosmology},
pubstate = {published},
tppubtype = {article}
}
Current and upcoming cosmological observations allow us to probe structures on smaller and smaller scales, entering highly nonlinear regimes. In order to obtain theoretical predictions in these regimes, large cosmological simulations have to be carried out. The promised high accuracy from observations make the simulation task very demanding: the simulations have to be at least as accurate as the observations. This requirement can only be ful lled by carrying out an extensive code validation program. The rst step of such a program is the comparison of diㄦent cosmology codes including gravitation interactions only. In this paper we extend a recently carried out code comparison project to include five more simulation codes. We restrict our analysis to a small cosmological volume which allows us to investigate properties of halos. For the matter power spectrum and the mass function, the previous results hold, with the codes agreeing at the 10% level over wide dynamic ranges. We extend our analysis to the comparison of halo profiles and investigate the halo count as a function of local density. We introduce and discuss ParaView as a exible analysis tool for cosmological simulations, the use of which immensely simpli es the code comparison task.
2006
Ahrens, James; Heitmann, Katrin; Habib, Salman; Ankeny, Lee; McCormick, Patrick; Inman, Jeff; Armstrong, Ryan; Ma, Kwan-Liu
Quantitative and comparative visualization applied to cosmological simulations Proceedings Article
In: Journal of Physics: Conference Series, pp. 526, IOP Publishing 2006, (LA-UR-06-4416).
Abstract | Links | BibTeX | Tags: comparative visualization, cosmology, Quantitative visualization
@inproceedings{ahrens2006quantitative,
title = {Quantitative and comparative visualization applied to cosmological simulations},
author = {James Ahrens and Katrin Heitmann and Salman Habib and Lee Ankeny and Patrick McCormick and Jeff Inman and Ryan Armstrong and Kwan-Liu Ma},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/QuantitativeAndComparativeVisualizationAppliedToCosmologicalSimulations.pdf},
year = {2006},
date = {2006-01-01},
booktitle = {Journal of Physics: Conference Series},
volume = {46},
number = {1},
pages = {526},
organization = {IOP Publishing},
abstract = {Cosmological simulations follow the formation of nonlinear structure in dark and luminous matter. The associated simulation volumes and dynamic range are very large, making visualization both a necessary and challenging aspect of the analysis of these datasets. Our goal is to understand sources of inconsistency between different simulation codes that are started from the same initial conditions. Quantitative visualization supports the definition and reasoning about analytically defined features of interest. Comparative visualization supports the ability to visually study, side by side, multiple related visualizations of these simulations. For instance, a scientist can visually distinguish that there are fewer halos (localized lumps of tracer particles) in low-density regions for one simulation code out of a collection. This qualitative result will enable the scientist to develop a hypothesis, such as loss of halos in low-density regions due to limited resolution, to explain the inconsistency between the different simulations. Quantitative support then allows one to confirm or reject the hypothesis. If the hypothesis is rejected, this step may lead to new insights and a new hypothesis, not available from the purely qualitative analysis. We will present methods to significantly improve the scientific analysis process by incorporating quantitative analysis as the driver for visualization. Aspects of this work are included as part of two visualization tools, ParaView, an open-source large data visualization tool, and Scout, an analysis-language based, hardware-accelerated visualization tool.},
note = {LA-UR-06-4416},
keywords = {comparative visualization, cosmology, Quantitative visualization},
pubstate = {published},
tppubtype = {inproceedings}
}
Cosmological simulations follow the formation of nonlinear structure in dark and luminous matter. The associated simulation volumes and dynamic range are very large, making visualization both a necessary and challenging aspect of the analysis of these datasets. Our goal is to understand sources of inconsistency between different simulation codes that are started from the same initial conditions. Quantitative visualization supports the definition and reasoning about analytically defined features of interest. Comparative visualization supports the ability to visually study, side by side, multiple related visualizations of these simulations. For instance, a scientist can visually distinguish that there are fewer halos (localized lumps of tracer particles) in low-density regions for one simulation code out of a collection. This qualitative result will enable the scientist to develop a hypothesis, such as loss of halos in low-density regions due to limited resolution, to explain the inconsistency between the different simulations. Quantitative support then allows one to confirm or reject the hypothesis. If the hypothesis is rejected, this step may lead to new insights and a new hypothesis, not available from the purely qualitative analysis. We will present methods to significantly improve the scientific analysis process by incorporating quantitative analysis as the driver for visualization. Aspects of this work are included as part of two visualization tools, ParaView, an open-source large data visualization tool, and Scout, an analysis-language based, hardware-accelerated visualization tool.
: . .
1.
Heitmann, Katrin; Frontiere, Nicholas; Sewell, Christopher; Habib, Salman; Pope, Adrian; Finkel, Hal; Rizzi, Silvio; Insley, Joe; Bhattacharya, Suman
The Q Continuum Simulation: Harnessing the Power of GPU Accelerated Supercomputers Journal Article
In: 2015, (LA-UR-15-28271).
@article{Heitmann:2015a,
title = {The Q Continuum Simulation: Harnessing the Power of GPU Accelerated Supercomputers},
author = {Katrin Heitmann and Nicholas Frontiere and Christopher Sewell and Salman Habib and Adrian Pope and Hal Finkel and Silvio Rizzi and Joe Insley and Suman Bhattacharya},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/11/TheQContinuumSimulationHarnessingThePowerOfGPUAcceleratedSupercomputers2.pdf},
year = {2015},
date = {2015-01-01},
publisher = {To appear in The Astrophysical Journal},
abstract = {Modeling large-scale sky survey observations is a key driver for the continuing development of high resolution, large-volume, cosmological simulations. We report the first results from the 'Q Continuum' cosmological N-body simulation run carried out on the GPU-accelerated supercomputer Titan. The simulation encompasses a volume of (1300 Mpc)^3 and evolves more than half a trillion particles, leading to a particle mass resolution of ~1.5 X 10^8 M_sun. At this mass resolution, the Q Continuum run is currently the largest cosmology simulation available. It enables the construction of detailed synthetic sky catalogs, encompassing different modeling methodologies, including semi-analytic modeling and sub-halo abundance matching in a large, cosmological volume. Here we describe the simulation and outputs in detail and present first results for a range of cosmological statistics, such as mass power spectra, halo mass functions, and halo mass-concentration relations for different epochs. We also provide details on challenges connected to running a simulation on almost 90% of Titan, one of the fastest supercomputers in the world, including our usage of Titan's GPU accelerators.},
note = {LA-UR-15-28271},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Modeling large-scale sky survey observations is a key driver for the continuing development of high resolution, large-volume, cosmological simulations. We report the first results from the 'Q Continuum' cosmological N-body simulation run carried out on the GPU-accelerated supercomputer Titan. The simulation encompasses a volume of (1300 Mpc)^3 and evolves more than half a trillion particles, leading to a particle mass resolution of ~1.5 X 10^8 M_sun. At this mass resolution, the Q Continuum run is currently the largest cosmology simulation available. It enables the construction of detailed synthetic sky catalogs, encompassing different modeling methodologies, including semi-analytic modeling and sub-halo abundance matching in a large, cosmological volume. Here we describe the simulation and outputs in detail and present first results for a range of cosmological statistics, such as mass power spectra, halo mass functions, and halo mass-concentration relations for different epochs. We also provide details on challenges connected to running a simulation on almost 90% of Titan, one of the fastest supercomputers in the world, including our usage of Titan's GPU accelerators.
2.
Sewell, Christopher; Lo, Li-Ta; Heitmann, Katrin; Habib, Salman; Ahrens, James
Utilizing Many-Core Accelerators for Halo and Center Finding within a Cosmology Simulation Proceedings Article
In: Proceedings of the IEEE Symposium on Large Data Analysis and Visualization, IEEE Press, Chicago, Illinois, 2015, (LA-UR-15-22202).
@inproceedings{Sewell:2015a,
title = {Utilizing Many-Core Accelerators for Halo and Center Finding within a Cosmology Simulation},
author = {Christopher Sewell and Li-Ta Lo and Katrin Heitmann and Salman Habib and James Ahrens},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/UtilizingMany-CoreAcceleratorsForHaloAndCenterFindingWithinACosmologySimulation.pdf},
year = {2015},
date = {2015-01-01},
booktitle = {Proceedings of the IEEE Symposium on Large Data Analysis and Visualization},
publisher = {IEEE Press},
address = {Chicago, Illinois},
series = {LDAV '15},
abstract = {Efficiently finding and computing statistics about “halos” (regions of high density) are essential analysis steps for N-body cosmology simulations. However, in state-of-the-art simulation codes, these analysis operators do not currently take advantage of the shared- memory data-parallelism available on multi-core and many-core ar- chitectures. The Hybrid / Hardware Accelerated Cosmology Code (HACC) is designed as an MPI+X code, but the analysis operators are parallelized only among MPI ranks, because of the difficulty in porting different X implementations (e.g., OpenMP, CUDA) across all architectures on which it is run. In this paper, we present portable data-parallel algorithms for several variations of halo find- ing and halo center finding algorithms. These are implemented with the PISTON component of the VTK-m framework, which uses Nvidia’s Thrust library to construct data-parallel algorithms that al- low a single implementation to be compiled to multiple backends to target a variety of multi-core and many-core architectures. Fi- nally, we compare the performance of our halo and center find- ing algorithms against the original HACC implementations on the Moonlight, Stampede, and Titan supercomputers. The portability of Thrust allowed the same code to run efficiently on each of these architectures. On Titan, the performance improvements using our code have enabled halo analysis to be performed on a very large data set (81923 particles across 16,384 nodes of Titan) for which analysis using only the existing CPU algorithms was not feasible.},
note = {LA-UR-15-22202},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Efficiently finding and computing statistics about “halos” (regions of high density) are essential analysis steps for N-body cosmology simulations. However, in state-of-the-art simulation codes, these analysis operators do not currently take advantage of the shared- memory data-parallelism available on multi-core and many-core ar- chitectures. The Hybrid / Hardware Accelerated Cosmology Code (HACC) is designed as an MPI+X code, but the analysis operators are parallelized only among MPI ranks, because of the difficulty in porting different X implementations (e.g., OpenMP, CUDA) across all architectures on which it is run. In this paper, we present portable data-parallel algorithms for several variations of halo find- ing and halo center finding algorithms. These are implemented with the PISTON component of the VTK-m framework, which uses Nvidia’s Thrust library to construct data-parallel algorithms that al- low a single implementation to be compiled to multiple backends to target a variety of multi-core and many-core architectures. Fi- nally, we compare the performance of our halo and center find- ing algorithms against the original HACC implementations on the Moonlight, Stampede, and Titan supercomputers. The portability of Thrust allowed the same code to run efficiently on each of these architectures. On Titan, the performance improvements using our code have enabled halo analysis to be performed on a very large data set (81923 particles across 16,384 nodes of Titan) for which analysis using only the existing CPU algorithms was not feasible.
3.
Sewell, Christopher
Streaming Data-Parallel Algorithms Enable Cosmology Data Analysis for Large Halos Presentation
31.12.2014, (LA-UR-14-29638).
@misc{Sewell2014,
title = {Streaming Data-Parallel Algorithms Enable Cosmology Data Analysis for Large Halos},
author = {Christopher Sewell },
url = {http://datascience.dsscale.org/wp-content/uploads/2016/08/Streaming_Data-Parallel_Algorithms_Enable_Cosmology_Data_Analysis_for_Large_Halos.pdf},
year = {2014},
date = {2014-12-31},
abstract = {This presentation given by Christopher Sewell describes how streaming data-parallel algorithms have enabled cosmology data analysis for large halos.},
note = {LA-UR-14-29638},
keywords = {},
pubstate = {published},
tppubtype = {presentation}
}
This presentation given by Christopher Sewell describes how streaming data-parallel algorithms have enabled cosmology data analysis for large halos.
4.
Sewell, Christopher; Ahrens, James; Patchett, John
New Data-parallel Algorithms Accelerate Cosmology Data Analysis on GPUs Presentation
30.06.2014, (LA-UR-14-22054).
@misc{Sewell2014b,
title = {New Data-parallel Algorithms Accelerate Cosmology Data Analysis on GPUs},
author = {Christopher Sewell and James Ahrens and John Patchett},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/08/New_Data-parallel_Algorithms_Accelerate_Cosmology_Data_Analysis_on_GPUs.pdf},
year = {2014},
date = {2014-06-30},
abstract = {This presentation describes how new data-parallel algorithms have accelerated cosmology data analysis on GPUs.},
note = {LA-UR-14-22054},
keywords = {},
pubstate = {published},
tppubtype = {presentation}
}
This presentation describes how new data-parallel algorithms have accelerated cosmology data analysis on GPUs.
5.
Patchett, John
Applications of In Situ Visualization for Ocean, Cosmology, and Plasma Presentation
20.02.2013, (LA-UR-13-21112).
@misc{Patchett2013b,
title = {Applications of In Situ Visualization for Ocean, Cosmology, and Plasma},
author = {John Patchett},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/08/Applications_of_In_Situ_Visualization_for_Ocean_Cosmology_and_Plasma.pdf},
year = {2013},
date = {2013-02-20},
abstract = {This is a five minute or less talk for the Office of Science SDAV All Hands Meeting on 2/20/2013. It describes our work with three domains of science: ocean modeling (POP), cosmology(HACC), and plasma(VPIC). In particular it presents work that was directly related to in situ analysis and our future work with these models under SDAV.},
note = {LA-UR-13-21112},
keywords = {},
pubstate = {published},
tppubtype = {presentation}
}
This is a five minute or less talk for the Office of Science SDAV All Hands Meeting on 2/20/2013. It describes our work with three domains of science: ocean modeling (POP), cosmology(HACC), and plasma(VPIC). In particular it presents work that was directly related to in situ analysis and our future work with these models under SDAV.
6.
Woodring, Jonathan; Heitmann, Katrin; Ahrens, James; Fasel, Patricia; Hsu, Chung-Hsing; Habib, Salman; Pope, Adrian
Analyzing and visualizing cosmological simulations with ParaView Journal Article
In: The Astrophysical Journal Supplement Series, vol. 195, no. 1, pp. 11, 2011, (LA-UR-10-06301).
@article{woodring2011analyzing,
title = {Analyzing and visualizing cosmological simulations with ParaView},
author = {Jonathan Woodring and Katrin Heitmann and James Ahrens and Patricia Fasel and Chung-Hsing Hsu and Salman Habib and Adrian Pope},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/AnalyzingAndVisualizingCosmologicalSimulationsWithParaView.pdf},
year = {2011},
date = {2011-01-01},
journal = {The Astrophysical Journal Supplement Series},
volume = {195},
number = {1},
pages = {11},
publisher = {IOP Publishing},
abstract = {The advent of large cosmological sky surveys – ushering in the era of precision cosmology – has been accompanied by ever larger cosmological simulations. The analysis of these simulations, which currently encompass tens of billions of particles and up to trillion particles in the near future, is often as daunting as carrying out the simulations in the first place. Therefore, the development of very efficient analysis tools combining qualitative and quantitative capabilities is a matter of some urgency. In this paper we introduce new analysis features implemented within ParaView, a parallel, open-source visualization toolkit, to analyze large N-body simulations. The new features include particle readers and a very efficient halo finder which identifies friends-of-friends halos and determines common halo properties. In combination with many other functionalities already existing within ParaView, such as histogram routines or interfaces to Python, this enhanced version enables fast, interactive, and convenient analyses of large cosmological simulations. In addition, development paths are available for future extensions.},
note = {LA-UR-10-06301},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The advent of large cosmological sky surveys – ushering in the era of precision cosmology – has been accompanied by ever larger cosmological simulations. The analysis of these simulations, which currently encompass tens of billions of particles and up to trillion particles in the near future, is often as daunting as carrying out the simulations in the first place. Therefore, the development of very efficient analysis tools combining qualitative and quantitative capabilities is a matter of some urgency. In this paper we introduce new analysis features implemented within ParaView, a parallel, open-source visualization toolkit, to analyze large N-body simulations. The new features include particle readers and a very efficient halo finder which identifies friends-of-friends halos and determines common halo properties. In combination with many other functionalities already existing within ParaView, such as histogram routines or interfaces to Python, this enhanced version enables fast, interactive, and convenient analyses of large cosmological simulations. In addition, development paths are available for future extensions.
7.
Habib, Salman; Pope, Adrian; Lukic, Zarija; Daniel, David; Fasel, Patricia; Desai, Nehal; Heitmann, Katrin; Hsu, Chung-Hsing; Ankeny, Lee; Mark, Graham
Hybrid petacomputing meets cosmology: The Roadrunner Universe project Proceedings Article
In: Journal of Physics: Conference Series, pp. 012019, IOP Publishing 2009, (LA-UR-09-03785).
@inproceedings{habib2009hybrid,
title = {Hybrid petacomputing meets cosmology: The Roadrunner Universe project},
author = {Salman Habib and Adrian Pope and Zarija Lukic and David Daniel and Patricia Fasel and Nehal Desai and Katrin Heitmann and Chung-Hsing Hsu and Lee Ankeny and Graham Mark},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/HybridPetacomputingMeetsCosmologyTheRoadrunnerUniverseProject.pdf},
year = {2009},
date = {2009-01-01},
booktitle = {Journal of Physics: Conference Series},
volume = {180},
number = {1},
pages = {012019},
organization = {IOP Publishing},
abstract = {Over the last two decades, critical observational advances in large-volume sky surveys carried out over a wide range of wavelengths, as well as over short time cadences, have revolutionized cosmology. Computational cosmology has emerged as an essential resource for providing detailed predictions for these observations, essential data for assisting in their design, and sophisticated tools for interpreting the final results.},
note = {LA-UR-09-03785},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Over the last two decades, critical observational advances in large-volume sky surveys carried out over a wide range of wavelengths, as well as over short time cadences, have revolutionized cosmology. Computational cosmology has emerged as an essential resource for providing detailed predictions for these observations, essential data for assisting in their design, and sophisticated tools for interpreting the final results.
8.
Anderson, Erik; Silva, Claudio T; Ahrens, James; Heitmann, Katrin; Habib, Salman
Provenance in comparative analysis: A study in cosmology Journal Article
In: Computing in Science & Engineering, vol. 10, no. 3, pp. 30–37, 2008, (LA-UR-08-02608).
@article{anderson2008provenance,
title = {Provenance in comparative analysis: A study in cosmology},
author = {Erik Anderson and Claudio T Silva and James Ahrens and Katrin Heitmann and Salman Habib},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/ProvenanceInComparativeAnalysisAStudyInCosmology.pdf},
year = {2008},
date = {2008-01-01},
journal = {Computing in Science & Engineering},
volume = {10},
number = {3},
pages = {30--37},
publisher = {AIP Publishing},
abstract = {Provenance—the logging of information about how data came into being and how it was processed—is an essential aspect of managing large-scale simulation and data-intensive projects. Using a cosmology code comparison project as an example, this article presents how a provenance system can play a key role in such applications.},
note = {LA-UR-08-02608},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Provenance—the logging of information about how data came into being and how it was processed—is an essential aspect of managing large-scale simulation and data-intensive projects. Using a cosmology code comparison project as an example, this article presents how a provenance system can play a key role in such applications.
9.
Heitmann, Katrin; Lukic, Zarija; Fasel, Patricia; Habib, Salman; Warren, Michael S.; White, Martin; Ahrens, James; Ankeny, Lee; Armstrong, Ryan; O’Shea, Brian; Ricker, Paul M.; Springel, Volker; Stadel, Joachim; Trac, Hy
The cosmic code comparison project Journal Article
In: Computational Science & Discovery, vol. 1, no. 1, pp. 015003, 2008, (LA-UR-07-1953).
@article{heitmann2008cosmic,
title = {The cosmic code comparison project},
author = {Katrin Heitmann and Zarija Lukic and Patricia Fasel and Salman Habib and Michael S. Warren and Martin White and James Ahrens and Lee Ankeny and Ryan Armstrong and Brian O’Shea and Paul M. Ricker and Volker Springel and Joachim Stadel and Hy Trac},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/TheCosmicCodeComparisonProject.pdf},
year = {2008},
date = {2008-01-01},
journal = {Computational Science & Discovery},
volume = {1},
number = {1},
pages = {015003},
publisher = {IOP Publishing},
abstract = {Current and upcoming cosmological observations allow us to probe structures on smaller and smaller scales, entering highly nonlinear regimes. In order to obtain theoretical predictions in these regimes, large cosmological simulations have to be carried out. The promised high accuracy from observations make the simulation task very demanding: the simulations have to be at least as accurate as the observations. This requirement can only be ful lled by carrying out an extensive code validation program. The rst step of such a program is the comparison of diㄦent cosmology codes including gravitation interactions only. In this paper we extend a recently carried out code comparison project to include five more simulation codes. We restrict our analysis to a small cosmological volume which allows us to investigate properties of halos. For the matter power spectrum and the mass function, the previous results hold, with the codes agreeing at the 10% level over wide dynamic ranges. We extend our analysis to the comparison of halo profiles and investigate the halo count as a function of local density. We introduce and discuss ParaView as a exible analysis tool for cosmological simulations, the use of which immensely simpli es the code comparison task.},
note = {LA-UR-07-1953},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Current and upcoming cosmological observations allow us to probe structures on smaller and smaller scales, entering highly nonlinear regimes. In order to obtain theoretical predictions in these regimes, large cosmological simulations have to be carried out. The promised high accuracy from observations make the simulation task very demanding: the simulations have to be at least as accurate as the observations. This requirement can only be ful lled by carrying out an extensive code validation program. The rst step of such a program is the comparison of diㄦent cosmology codes including gravitation interactions only. In this paper we extend a recently carried out code comparison project to include five more simulation codes. We restrict our analysis to a small cosmological volume which allows us to investigate properties of halos. For the matter power spectrum and the mass function, the previous results hold, with the codes agreeing at the 10% level over wide dynamic ranges. We extend our analysis to the comparison of halo profiles and investigate the halo count as a function of local density. We introduce and discuss ParaView as a exible analysis tool for cosmological simulations, the use of which immensely simpli es the code comparison task.
10.
Ahrens, James; Heitmann, Katrin; Habib, Salman; Ankeny, Lee; McCormick, Patrick; Inman, Jeff; Armstrong, Ryan; Ma, Kwan-Liu
Quantitative and comparative visualization applied to cosmological simulations Proceedings Article
In: Journal of Physics: Conference Series, pp. 526, IOP Publishing 2006, (LA-UR-06-4416).
@inproceedings{ahrens2006quantitative,
title = {Quantitative and comparative visualization applied to cosmological simulations},
author = {James Ahrens and Katrin Heitmann and Salman Habib and Lee Ankeny and Patrick McCormick and Jeff Inman and Ryan Armstrong and Kwan-Liu Ma},
url = {http://datascience.dsscale.org/wp-content/uploads/2016/06/QuantitativeAndComparativeVisualizationAppliedToCosmologicalSimulations.pdf},
year = {2006},
date = {2006-01-01},
booktitle = {Journal of Physics: Conference Series},
volume = {46},
number = {1},
pages = {526},
organization = {IOP Publishing},
abstract = {Cosmological simulations follow the formation of nonlinear structure in dark and luminous matter. The associated simulation volumes and dynamic range are very large, making visualization both a necessary and challenging aspect of the analysis of these datasets. Our goal is to understand sources of inconsistency between different simulation codes that are started from the same initial conditions. Quantitative visualization supports the definition and reasoning about analytically defined features of interest. Comparative visualization supports the ability to visually study, side by side, multiple related visualizations of these simulations. For instance, a scientist can visually distinguish that there are fewer halos (localized lumps of tracer particles) in low-density regions for one simulation code out of a collection. This qualitative result will enable the scientist to develop a hypothesis, such as loss of halos in low-density regions due to limited resolution, to explain the inconsistency between the different simulations. Quantitative support then allows one to confirm or reject the hypothesis. If the hypothesis is rejected, this step may lead to new insights and a new hypothesis, not available from the purely qualitative analysis. We will present methods to significantly improve the scientific analysis process by incorporating quantitative analysis as the driver for visualization. Aspects of this work are included as part of two visualization tools, ParaView, an open-source large data visualization tool, and Scout, an analysis-language based, hardware-accelerated visualization tool.},
note = {LA-UR-06-4416},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Cosmological simulations follow the formation of nonlinear structure in dark and luminous matter. The associated simulation volumes and dynamic range are very large, making visualization both a necessary and challenging aspect of the analysis of these datasets. Our goal is to understand sources of inconsistency between different simulation codes that are started from the same initial conditions. Quantitative visualization supports the definition and reasoning about analytically defined features of interest. Comparative visualization supports the ability to visually study, side by side, multiple related visualizations of these simulations. For instance, a scientist can visually distinguish that there are fewer halos (localized lumps of tracer particles) in low-density regions for one simulation code out of a collection. This qualitative result will enable the scientist to develop a hypothesis, such as loss of halos in low-density regions due to limited resolution, to explain the inconsistency between the different simulations. Quantitative support then allows one to confirm or reject the hypothesis. If the hypothesis is rejected, this step may lead to new insights and a new hypothesis, not available from the purely qualitative analysis. We will present methods to significantly improve the scientific analysis process by incorporating quantitative analysis as the driver for visualization. Aspects of this work are included as part of two visualization tools, ParaView, an open-source large data visualization tool, and Scout, an analysis-language based, hardware-accelerated visualization tool.