Chris Patrick writes that SLAC researchers have built a neural-network method for compressing large scientific datasets while preserving fine details that conventional compression erases. The approach uses wavelet analysis to separate data features by scale, then encodes each scale separately through a neural network, enabling 10- to 100-fold file size reductions and selective decompression of only the regions a researcher needs.
- Published in Nature Machine Intelligence (August 24, 2026)
- Motivated by upcoming LCLS upgrades that will generate nearly one terabyte of data per second
- Tested successfully on X-ray diffraction data, solar magnetic field measurements, and photographs
- Neural networks trained on Perlmutter at NERSC (Lawrence Berkeley National Lab)
- Co-developers include researchers from UC Davis and Carnegie Mellon University
- Published in Nature Machine Intelligence (August 24, 2026)
- Motivated by upcoming LCLS upgrades that will generate nearly one terabyte of data per second
- Tested successfully on X-ray diffraction data, solar magnetic field measurements, and photographs
- Neural networks trained on Perlmutter at NERSC (Lawrence Berkeley National Lab)
- Co-developers include researchers from UC Davis and Carnegie Mellon University
