Science Data Processor

What do we mean when we refer to Science Data Processor (SDP) in the SKA design?

SDP LOGO

The “Science Data Processor” (SDP) element will focus on the design of the computing hardware platforms, software, and algorithms needed to process science data from the correlator or non-imaging processor into science data products. The data rates involved in this will exceed that of the entire global internet traffic per day.

More on the Science Data Processor
The science data processor will have to manage the vast amounts of data being generated by the telescopes. From sky surveys, continuum surveys through to more targeted observations of objects both near and far. Taking the data, and moving it through data pipelines at staggering speeds, to then form data packages which will then be passed to the scientists, and in almost real-time, make decisions about noise that is not part of those delicate radio signals.

Unlike most previous telescopes the SDP element for the SKA will be in large part a pseudo
real-time part of the whole system. Reliability of the SDP element is intimately linked to
the overall system performance in a way more analogous to, for example, correlators are in
current telescopes. An integrated local monitor and control system with appropriate health
monitoring feeding back into the Telescope Manager is a critical component of the SDP
element.

The SDP Consortium
The SDP consortium will focus on the development of the computing hardware platforms, software, and algorithms needed to process science data from the correlator or non-imaging processor into science data products. The data rates involved in this will exceed anything in the field of science at this time.

The SDP’s computing platform will host the processing pipelines (taking the raw data from the telescopes and converting this in to usable data for scientists) and the data layer. The computing platform includes hardware, operating system, SKA-originated software, and third party software. There may be multiple configurations tuned to the different telescopes

The processing pipelines include all software for ingesting, calibrating, editing, imaging, cataloguing, and searching to produce science data products. The processing pipelines execute on the SDP, and have to be executed in near real time.

Main challenges
SDP is the part of the SKA system that will accept the data output from the correlator/beam-former and that will deliver final calibrated data products to scientists. To achieve the fidelity and high dynamic range required, the SDP system will have to employ sophisticated radio frequency interference mitigation, calibration and imaging algorithms, yet the sustained input
data rates will be in the 100’s of gigabits per second and will need to be processed in soft-realtime.

This means that a super-computer scale computing facility will need to be used for the processing of the data and that a fresh and carefully considered approach to software engineering, algorithms, and computing operations will need to be developed.

The SDP Consortium
The SDP Consortium is led by Professor Paul Alexander of the University of Cambridge, UK, but the consortium itself will be working with over 40 organisations around the globe, including many of the world’s greatest Universities.

Institutions involved in the SDP Consortium include:

Full partners

Associate partners

Additional contact information of people involved at each institution can be provided by the consortium lead Paul Alexander

Click on the map below to find out more about the institutes involved in the SDP Consortium

SKA Global Consortia

STFC Rutherford Appleton Laboratory

Part of the following consortia:

Central Signal Processor
Science Data Processor
Visit The Website

Callaghan Innovation

Part of the following consortia:

Science Data Processor
Visit The Website

Open Parallel Ltd

Part of the following consortia:

Central Signal Processor
Science Data Processor
Visit The Website

University of Auckland

Part of the following consortia:

Central Signal Processor
Science Data Processor

Massey University

Part of the following consortia:

Central Signal Processor
Science Data Processor
Visit The Website

AUT University

Part of the following consortia:

Central Signal Processor
Science Data Processor
Visit The Website

Canadian Astronomy Data Center

Part of the following consortia:

Science Data Processor
Visit The Website

University of Southampton

Part of the following consortia:

Science Data Processor
Visit The Website

University College London

Part of the following consortia:

Science Data Processor
Visit The Website

Oxford University

Part of the following consortia:

Central Signal Processor
Low-Frequency Aperture Array
Mid-Frequency Aperture Array
Science Data Processor
Visit The Website

University of Cambridge

Part of the following consortia:

Low-Frequency Aperture Array
Mid-Frequency Aperture Array
Science Data Processor
Visit The Website

University of Manchester

Part of the following consortia:

Central Signal Processor
Low-Frequency Aperture Array
Mid-Frequency Aperture Array
Science Data Processor
Signal and Data Transport
Visit The Website

STFC Daresbury Laboratory

Part of the following consortia:

Science Data Processor
Visit The Website

Fundación Centro de Supercomputación (FCSCL)

Part of the following consortia:

Science Data Processor
Visit The Website

Barcelona Supercomputing Center

Part of the following consortia:

Science Data Processor
Visit The Website

Instituto de Astrofísica de Andalucía (IAA)

Part of the following consortia:

Science Data Processor
Visit The Website

Council for Scientific and Industrial Research (CSIR)

Part of the following consortia:

Science Data Processor
Visit The Website

SKA South Africa (Cape Town Office)

Part of the following consortia:

Assembly, Integration & Verification
Central Signal Processor
Dish
Mid-Frequency Aperture Array
Science Data Processor
Signal and Data Transport
Telescope Manager
Visit The Website

University of Évora

Part of the following consortia:

Science Data Processor
Visit The Website

Instituto de Telecomunicações

Part of the following consortia:

Mid-Frequency Aperture Array
Science Data Processor
Signal and Data Transport
Telescope Manager
Visit The Website

GreenButton

Part of the following consortia:

Science Data Processor
Visit The Website

Compucon New Zealand

Part of the following consortia:

Central Signal Processor
Science Data Processor
Visit The Website

Institute for Radio Astronomy and Space Research

Part of the following consortia:

Science Data Processor
Visit The Website

University of Otogo

Part of the following consortia:

Science Data Processor
Visit The Website

Victoria University of Wellington

Part of the following consortia:

Science Data Processor
Visit The Website

Netherlands Institute for Radio Astronomy (ASTRON)

Part of the following consortia:

Assembly, Integration & Verification
Central Signal Processor
Low-Frequency Aperture Array
Mid-Frequency Aperture Array
Science Data Processor
Signal and Data Transport
Wideband Single Pixel Feeds
Visit The Website

Regensburg University

Part of the following consortia:

Science Data Processor
Visit The Website

University of Chile

Part of the following consortia:

Science Data Processor
Visit The Website

McGill University

Part of the following consortia:

Science Data Processor
Visit The Website

University of British Columbia

Part of the following consortia:

Science Data Processor
Visit The Website

University of Alberta

Part of the following consortia:

Science Data Processor
Visit The Website

University of Alberta

Part of the following consortia:

Science Data Processor
Visit The Website

RackForce

Part of the following consortia:

Science Data Processor
Visit The Website

Calgary Scientific Inc.

Part of the following consortia:

Science Data Processor
Visit The Website

IBM Canada Limited

Part of the following consortia:

Science Data Processor
Visit The Website

University of Calgary

Part of the following consortia:

Science Data Processor
Visit The Website

University of Melbourne

Part of the following consortia:

Science Data Processor
Visit The Website

iVEC

Part of the following consortia:

Science Data Processor
Visit The Website

International Centre for Radio Astronomy Research (ICRAR)

Part of the following consortia:

Central Signal Processor
Low-Frequency Aperture Array
Science Data Processor
Visit The Website

Commonwealth Scientific and Industrial Research Organisation (CSIRO)

Part of the following consortia:

Assembly, Integration & Verification
Central Signal Processor
Dish
Infrastructure Australia
Science Data Processor
Signal and Data Transport
Telescope Manager
Visit The Website
Top