Efficient Compression Systems for Next-Generation Scientific Datasets

The project starts from a clear challenge: scientific data production is growing much faster than current storage and transmission capabilities. Synchrotron facilities, DNA sequencing workflows, and space missions generate massive data volumes that require a revision of current standards.

ECS4NSD advances the state of the art through compression techniques tailored to each domain and computing platform, combining highly parallel approaches, machine learning-based coding, and compact data structures to enable efficient information extraction from long-term archives.

The project is led by Joan Serra-Sagristà (IP1) and Joan Bartrina-Rapesta (IP2).

Projecte contributions

General objectives

• O1. On-the-fly compression: new algorithms for scientific acquisition with high throughput and low complexity.
• O2. Long-term compression and information extraction: improved compression performance, quality, and efficient data access.
• O3. Internationalization and transfer: contributions to standards (CCSDS/HDF5), European collaborations, and industrial transfer.

Specific objectives

• O1.1 Hardware-tailored parallel algorithms (SIMD/GPU) for compression during data acquisition.
• O1.2 Multi-platform deployment (GPU, TPU, and low-power accelerators).
• O2.1 High-compression techniques using scalable entropy coding, RoI, and learned models.
• O2.2 New methodologies and compact structures for fast information extraction (genomics and crystallography).
• O3.1 Participation in international standardization (CCSDS and HDF5).
• O3.2 International collaborations and transfer of results to industrial partners.

WP1 - Macromolecular crystallography data

Coordination: Albert Castellví, Joan Bartrina Rapesta (IP2)

Design and integration of compression techniques for crystallography workflows, including on-the-fly coding, long-term archiving, and HDF5 integration.

WP2 - Computed tomography data

Coordination: Gabriel Jover Mañas, Joan Serra-Sagristà (IP1)

Study of tomography workflows, development and validation of compression techniques for long-term archiving, and deployment in ALBA workflows.

WP3 - Space data

Coordination: Ian Blanes García, Joan Bartrina Rapesta (IP2)

Compression for space data using machine learning models, low computational complexity, and advanced smart coding features.

WP4 - Deoxyribonucleic acid data

Coordination: Ivan Erill, Joan Serra-Sagristà (IP1)

Compact data structures and new algorithms for efficient representation, search, and analysis of genomic data.

WP5 - Internationalization and technology transfer

Coordination: Joan Bartrina Rapesta (IP2), Joan Serra-Sagristà (IP1)

Contributions to CCSDS and HDF5, participation in European proposals, and transfer of results to industrial products.

WP6 - Coordination and management

Coordination: Joan Bartrina Rapesta (IP2), Joan Serra-Sagristà (IP1)

Technical and financial management of the project, milestone tracking, and overall execution coordination.

Research team

• Joan Serra-Sagristà (IP1)
• Joan Bartrina Rapesta (IP2)
• Albert Castellví
• Gabriel Jover Mañas
• Ian Blanes García
• Ivan Erill Sagalés
• Victor Sanchez Silva

Work team

• David Alegre Alarza
• Natàlia Blasco Andreo
• Xavier Fernández Mellado
• Junya Ji
• Pau Quintas Torra
• Roger Ribas Nicolau
• Sebastià Mijares i Verdú
• Zuoxin Xi
• Michael Wesley Marcellin
• Manuel Isaac Martínez Torres
• Armando Pinho
• 2 PhD positions (to be hired)
• 3 technical positions (to be hired)

• Universitat Autònoma de Barcelona (UAB) - main coordinating institution of the project.
• ALBA Synchrotron - direct participation in crystallography/tomography data and demonstrator integration.
• University of Maryland, Baltimore County (USA) - participation in the bioinformatics/genomics line.
• University of Warwick (UK) - participation in computing and applied machine learning.
• University of Arizona (USA), Philips Medizin Systeme Böblingen GmbH (Germany), and University of Aveiro (Portugal) - international collaboration in the work team.
• Transfer and data ecosystem entities: ESA, CNES, NASA, OpenCosmos, Satellogic, Airbus, NCBI, and IEEC (depending on project lines and data sources).

For collaborations or technical inquiries about ECS4NSD, contact the DATAKOM Data Compression Lab.

Join us How to reach us