hpc4neuro Python library
Background and Motivation
This project brings together a collection of utilities that have been factored out from different projects. In certain cases we need a specific functionality that does not appear to be available in existing packages. We therefore develop the required code in-house, and if it appears to be general enough, we add it to this collection in the hope that it may be useful to others.
Contributions so far
Our very first utility module is hpc4neuro.distribution. This module provides a high-level interface for data distribution with MPI, without the explicit need to write MPI code on the user's part. The primary motivation for creating this module was to hide the details of distributing training and validation data amongst MPI ranks when training deep artificial neural networks in a data-parallel fashion using Horovod. Even though Horovod hides the intricate details of distributed training, proper distribution of training and validation data is only possible via MPI programming. The hpc4neuro.distribution fills this gap.
Please visit the project website for further details.
We are currently in the process of adding further utilities for automated sectioning and batching of multi-dimensional data, as well as image processing. In the future, we hope to extend the hpc4neuro package with many more utilities, including utilities for simulation development.
Our collaboration partners
This project is being developed in-house. However, some of the earliest utilities have been factored out from code developed for the following collaborators:
- Fiber Architecture group of the INM-1
- Dr. rer. medic. Peter Pieperhoff from the INM-1
Related projects at the Simulation Lab Neuroscience
- Deep Learning Software for Supercomputers
- Deep Learning for segmentation of 3D-PLI images
- Deep Learning for Brain Extraction from MRI scans
- Tutorial: Getting started with Deep Learning on Supercomputers
SimLab Contact
Wouter Klijn
Team Leader Multi Scale Simulation and Design, Simulation and Data Lab Neuroscience (SDLN), Jülich Supercomputing Centre (JSC)
- Institute for Advanced Simulation (IAS)
- Jülich Supercomputing Centre (JSC)
Room 2001