Science Data Processor

What do we mean when we refer to Science Data Processor (SDP) in the SKA design?

The “Science Data Processor” (SDP) element will focus on the design of the computing hardware platforms, software, and algorithms needed to process science data from the correlator or non-imaging processor into science data products. The data rates involved in this will exceed that of the entire global internet traffic per day.

More on the Science Data Processor

The Science Data Processor will have to manage the vast amounts of data being generated by the telescopes. From spectral and continuum sky surveys, to more targeted observations of objects both near and far, the SDP will ingest the data, and move it through data reduction pipelines at staggering speeds, to then form data packages which will then be passed to the scientists, and in almost realtime, make decisions about noise that is not part of those delicate radio signals.

Unlike most previous telescopes the SDP element for the SKA will be in large part a pseudo-realtime component of the whole system. The reliability of the SDP element is intimately linked to the overall system performance in a way more analogous to, for example, correlators are in current telescopes. An integrated local monitor and control system with appropriate health monitoring feeding back into the Telescope Manager (TM) is a critical component of the SDP element.

The SDP Consortium

The SDP consortium will focus on the development of the computing hardware platforms, software, and algorithms needed to process science data from the correlator or non-imaging processor into science data products. The data rates involved in this will exceed anything in the field of science at this time.

The SDP’s computing platform will host the processing pipelines (taking the raw data from the telescopes and converting this in to usable data for scientists) and the data layer. The computing platform includes hardware, operating system, SKA-originated software, and third party software. There may be multiple configurations tuned to the different telescopes

The processing pipelines include all software for ingesting, calibrating, editing, imaging, cataloguing, and searching to produce science data products. The processing pipelines execute on the SDP, and have to be executed in near real time.

Main challenges

SDP is the part of the SKA system that will accept the data output from the correlator/beam-former and that will deliver final calibrated data products to scientists. To achieve the fidelity and high dynamic range required, the SDP system will have to employ sophisticated radio frequency interference mitigation, calibration and imaging algorithms, yet the sustained input
data rates will be in the 100’s of gigabits per second and will need to be processed in soft-realtime.

This means that a super-computer scale computing facility will need to be used for the processing of the data and that a fresh and carefully considered approach to software engineering, algorithms, and computing operations will need to be developed.

The SDP Consortium

The SDP Consortium is led by Professor Paul Alexander of the University of Cambridge, UK, but the consortium itself will be working with over 40 organisations around the globe, including many of the world’s greatest Universities.

Institutions involved in the SDP Consortium include:

Full partners

• Canadian Universities Collaboration, Canada
• Centre for High Performance Computing, South Africa
• Chinese Universities Collaboration, China
• Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia
• Forschungszentrum Jülich GmbH: JUELICH
• International Centre for Radio Astronomy Research (ICRAR), Australia
• iVEC, Australia
• Max Planck Institute for Radio Astronomy (MPifR), Germany
• Canadian Astronomy Data Centre (CADC), Canada
• Netherlands Institute for Radio Astronomy (ASTRON), The Netherlands
• New Zealand Alliance (NZA)
  • Auckland University of Technology
  • AUT, High Performance Computing Research Laboratory
  • Catalyst IT
  • Compucon
  • Open Parallel
• Science and Technology Facilities Council (STFC) Hartree, UK
• SKA South Africa, South Africa
• University of Manchester, UK
• University of Cambridge, UK
• Oxford University, UK
• University of Southampton, UK
• Victoria University of Wellington, New Zealand

Associate partners

• Barcelona Supercomputing Center, Spain
• Instituto de Astrofísica de Andalucía (IAA-CSIC), Spain
• Observatoire de Paris (Station de Radioastronomie de Nançay), France
• Portuguese ENGAGE SKA Consortium, Portugal
• Université d’Orléans, France
• University College London, UK

Additional contact information of people involved at each institution can be provided by the consortium lead Paul Alexander

Click on the map below to find out more about the institutes involved in the SDP Consortium

SKA Global Consortia