Science Data Processor
What do we mean when we refer to Science Data Processor (SDP) in the SKA design?
The “Science Data Processor” (SDP) element will focus on the design of the computing hardware platforms, software, and algorithms needed to process science data from the correlator or non-imaging processor into science data products. The data rates involved in this will exceed that of the entire global internet traffic per day.
More on the Science Data Processor
The Science Data Processor will have to manage the vast amounts of data being generated by the telescopes. From spectral and continuum sky surveys, to more targeted observations of objects both near and far, the SDP will ingest the data, and move it through data reduction pipelines at staggering speeds, to then form data packages which will then be passed to the scientists, and in almost realtime, make decisions about noise that is not part of those delicate radio signals.
Unlike most previous telescopes the SDP element for the SKA will be in large part a pseudo-realtime component of the whole system. The reliability of the SDP element is intimately linked to the overall system performance in a way more analogous to, for example, correlators are in current telescopes. An integrated local monitor and control system with appropriate health monitoring feeding back into the Telescope Manager (TM) is a critical component of the SDP element.
The SDP Consortium
The SDP consortium will focus on the development of the computing hardware platforms, software, and algorithms needed to process science data from the correlator or non-imaging processor into science data products. The data rates involved in this will exceed anything in the field of science at this time.
The SDP’s computing platform will host the processing pipelines (taking the raw data from the telescopes and converting this in to usable data for scientists) and the data layer. The computing platform includes hardware, operating system, SKA-originated software, and third party software. There may be multiple configurations tuned to the different telescopes
The processing pipelines include all software for ingesting, calibrating, editing, imaging, cataloguing, and searching to produce science data products. The processing pipelines execute on the SDP, and have to be executed in near real time.
Main challenges
SDP is the part of the SKA system that will accept the data output from the correlator/beam-former and that will deliver final calibrated data products to scientists. To achieve the fidelity and high dynamic range required, the SDP system will have to employ sophisticated radio frequency interference mitigation, calibration and imaging algorithms, yet the sustained input
data rates will be in the 100’s of gigabits per second and will need to be processed in soft-realtime.
This means that a super-computer scale computing facility will need to be used for the processing of the data and that a fresh and carefully considered approach to software engineering, algorithms, and computing operations will need to be developed.
The SDP Consortium
The SDP Consortium is led by Professor Paul Alexander of the University of Cambridge, UK, but the consortium itself will be working with over 40 organisations around the globe, including many of the world’s greatest Universities.
Institutions involved in the SDP Consortium include:
Full partners
- Canadian Universities Collaboration, Canada
- Centre for High Performance Computing, South Africa
- Chinese Universities Collaboration, China
- Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia
- Forschungszentrum Jülich GmbH: JUELICH
- International Centre for Radio Astronomy Research (ICRAR), Australia
- iVEC, Australia
- Max Planck Institute for Radio Astronomy (MPifR), Germany
- Canadian Astronomy Data Centre (CADC), Canada
- Netherlands Institute for Radio Astronomy (ASTRON), The Netherlands
- New Zealand Alliance (NZA)
- Science and Technology Facilities Council (STFC) Hartree, UK
- SKA South Africa, South Africa
- University of Manchester, UK
- University of Cambridge, UK
- Oxford University, UK
- University of Southampton, UK
- Victoria University of Wellington, New Zealand
Associate partners
- Barcelona Supercomputing Center, Spain
- Instituto de Astrofísica de Andalucía (IAA-CSIC), Spain
- Observatoire de Paris (Station de Radioastronomie de Nançay), France
- Portuguese ENGAGE SKA Consortium, Portugal
- Université d’Orléans, France
- University College London, UK
Additional contact information of people involved at each institution can be provided by the consortium lead Paul Alexander
Click on the map below to find out more about the institutes involved in the SDP Consortium
SKA Global Consortia
STFC Rutherford Appleton Laboratory
Part of the following consortia:
Open Parallel Ltd
Part of the following consortia:
University of Auckland
Part of the following consortia:
Massey University
Part of the following consortia:
AUT University
Part of the following consortia:
Canadian Astronomy Data Center
Part of the following consortia:
Oxford University
Part of the following consortia:
University of Cambridge
Part of the following consortia:
University of Manchester
Part of the following consortia:
Fundación Centro de Supercomputación (FCSCL)
Part of the following consortia:
Barcelona Supercomputing Center
Part of the following consortia:
Instituto de Astrofísica de Andalucía (IAA)
Part of the following consortia:
Council for Scientific and Industrial Research (CSIR)
Part of the following consortia:
SKA South Africa (Cape Town Office)
Part of the following consortia:
Instituto de Telecomunicações
Part of the following consortia:
Compucon New Zealand
Part of the following consortia:
Institute for Radio Astronomy and Space Research
Part of the following consortia:
Victoria University of Wellington
Part of the following consortia:
Netherlands Institute for Radio Astronomy (ASTRON)
Part of the following consortia:
University of British Columbia
Part of the following consortia:
International Centre for Radio Astronomy Research (ICRAR)
Part of the following consortia:
Commonwealth Scientific and Industrial Research Organisation (CSIRO)
Part of the following consortia: