The EISCAT_3D Data Solutions project has ended
The NeIC EISCAT_3D Data Solutions (E3DDS) project has come to its end. The E3DDS project started in 2018, and during the lifespan of the project a lot has been achieved. This article allows us to look back at what has been done and outline the significance of the E3DDS project in the Nordic area. The text is written in collaboration with E3DDS’s project leader, John White.
What is E3DDS?
The aim of the EISCAT_3D Data Solutions (E3DDS) project is to support the scientific EISCAT community to build the online data processing for the EISCAT_3D project. The E3DDS project follows on from the previous E3DS project which looked at the computing from a higher level.
In this project, the online data production of the EISCAT_3D project is to be evaluated and scale-tested. The recommendations for the on-site EISCAT_3D on-site computing software and hardware architecture were to be made. The simulations of the EISCAT_3D online data productions were to be deployed to national provider e-infrastructures and the produced simulated data stored into a national provider storage system. Finally the synergies between the EISCAT_3D operations and the NeIC NT1 programme are explored.
Achievements
Since the project started in April 2018, a lot has been achieved.
The project has worked to measure the performance requirements of the various online computing services. The EISCAT scientists have provided the prototypes of the online computing and measured their requirements. In addition to this, national e-infrastructure providers have made computing and storage resources available for testing the EISCAT_3D online data production software prototypes. The requirements, software and hardware choices for the EISCAT_3D on-site computing have been determined but still evolve with technology. A data transfer test of analyzed EISCAT data files has been deployed in collaboration with the national e-infrastructure providers. The E3DDS project has also provided support for technical problems and received offers to host a test cluster.
E3DDS has also looked at the synergies between the EISCAT_3D and NeIC NT1 operations at each stage in the data chain. Also the wide-area networking configurations between the future EISCAT_3D sites have been investigated and designed in collaboration with Nordic Network providers. Collaboration between the project members and EISCAT scientists has made it possible to understand the balance between doing the science and the costs involved.
There are also project documents, deliverables and even [some conference talks that was given on behalf of E3DDS. Documents from the E3DDS project can be found here and from the E3DS project here.
Outcome
It will still take time to see whether an end-to-end data production test cluster will be in place in the future. The E3DDS project ran until April 30th, and in March the EISCAT_3D prototype hardware was still in the acceptance phase, so units were not available for a scale deployment. The project has been successful for testing online data production software, the wide-area network design and the online computing design, although these still need to be tested at scale.
A good understanding of the issues regarding the design and deployment of required computing clusters has been found. Also, the beneficial aspects of some of the EISCAT_3D data production services being operated by NeIC NT1 or other partners has been investigated. It is clear that parts of the EISCAT_3D data production can be deployed and/or operated by partners such as the NeIC NT1.