TEN SUCCESS STORIES FROM TEN YEARS OF SUCCESS: Cloud computing and services

Year 2022 marks the 10-year anniversary for NeIC. To celebrate this milestone, we will publish ten stories that showcase how NeIC has contributed to developing best-in-class e-infrastructure services beyond national capabilities and enhanced the productivity of research in the Nordic Region.

The Nordic e-Infrastructure Collaboration, also known as NeIC, was established in 2012. NeIC facilitates collaboration on digital infrastructure within the Nordic countries and Estonia by providing experts coming from different countries, organisations and fields opportunities to work together. This Nordic collaboration on digital infrastructure had started already before NeIC was established. Since 2003, the Nordic countries have been collaborating on the Worldwide Large Hadron Collider Computing Grid (WLCG) at CERN, providing research computing and storage for high-energy physicists worldwide. The successful collaboration that started with the services offered by the Nordic Data Grid Facility (NDGF) was after some years expanded into NeIC, which was tasked to run the Nordic WLCG Tier-1 facility, also known as NT1. The initiation of NeIC made it possible to facilitate collaborations to benefit other science areas.

This article focuses on NeIC’s efforts in the field of cloud computing and cloud services. To get the best possible insight on the subject, we reached out to Dan Still, Partnerships Manager at CSC - IT Center for Science and the manager for a former NeIC project that enhanced Nordic collaboration on cloud services.

Freedom through virtualisation of services

Are you using cloud services? If you have an account on AWS, Azure, Dropbox, Google Drive, Spotify, Slack, Zoom, or Netflix, the answer is yes. Cloud services are services offered by a third-party company or institution and made available over the internet. You can use the services on for example your phone or laptop as long as you are online without having to install them on your device. The word ”cloud” refers to the digital environment. Running workloads within these online environments, instead of actually installing the needed software or connecting the hardware to your device, is called cloud computing.

Since as early as 2014, NeIC has facilitated Nordic collaboration on cloud computing and services through the Glenna project which ended in 2020. Dan Still, who worked as Glenna’s project manager, says cloud computing is even today a fairly fresh concept and an area of rapid development.

Virtualisation of services has a history dating back to the 1960’s, but the advent of Amazon Web Services in 2006 revolutionised the field and provided new commercial solutions for cloud computing. According to Still, his was quite significant, and in the US, also in academia researchers started to run smaller jobs on commercial clouds, simply because it was that much easier than going through the bureaucracy of applying for computing time at a HPC centre.

In the Nordics, CSC, which is owned by the Finnish Ministry of Education and Culture and Finnish universities, had been successfully setting up community cloud services for research. The development of the OpenStack platform was especially influential. The benefits of cloud computing started to be acknowledged elsewhere in the region: suddenly researchers could get full access with administrative rights to the resources and could install and run basically whatever they wanted with full control. Still emphasises that the freedom that cloud computing provided was an important component.

Nordic collaboration started with Glenna

The Nordic countries started collaborating on improving cloud services in 2014, when CSC and KTH, the Royal Institute of Technology in Sweden, joined forces and took the idea of collaborative cloud computing to NordForsk in the form of a project proposal. Management at NeIC saw the possibilities and agreed that working together in the area would be beneficial for all the countries. Still had been involved in NeIC with an early project called B2Share Nordic, which was an effort to promote EUDAT services to the Nordics.

– After B2Share Nordic, I continued as the project manager for Glenna. Things at NeIC were starting up back then, and a lot of new, positive things were happening as the organisation was taking shape, Still says.

Glenna - an opening in the clouds

The initial idea was to build Nordic cloud environments but divide the work related to providing the services to users, so that each country would offer different services to Nordic researchers. In Glenna, the idea of resource sharing didn’t take off, but efforts continued in a later NeIC project called Dellingr. However, Still says that both Glenna and Dellingr faced the same, mostly political issues related to the sharing of resources: even if the technical solutions would be there, sharing the costs of nationally funded infrastructure between countries remains a challenge. However, the European Open Science Cloud (EOSC) has dedicated efforts to address the issue and the work is still in progress.

– I’m personally optimistic about EuroHPC, where the EuroHPC Joint Undertaking (JU), consisting of several governments, owns the pre-exascale machines. In the Glenna project, it was never on the table to build a new data centre or share ownership of infrastructure: the idea was to use existing resources, i.e. the clouds, at the Nordic centres, he says.

Still believes the idea of joint resource sharing and attempting cost recovery in the cloud environment had not been introduced on the European level before Glenna. From an European perspective, it was also unusual that all the providing centres in the Nordics were using the same platform, OpenStack. Still says that in the Nordic Region we tend to have the same preferences, whereas in continental Europe, the needs and also the organisations differ more from one another. Outside of Nordics, cloud services have also traditionally been offered by network organisations (NRENs) nationally due to their close interaction with the end-users.

There were some initiatives, like the EU-funded Contrail project and the EGI Federated Cloud, that Glenna used as early references. Still says these two projects, especially EGI influenced Glenna early on, and there was some initial collaboration between the two, but later organisational and technical differences pulled the Nordic development in a different direction.

Benefits for the Nordics and beyond

The first phase of Glenna sought first and foremost to explore the possibilities for cross-border resource exchange and billing services. Even though the former turned out to be problematic, Glenna produced several use cases. Still adds that the iOBS project - the collaboration between the MET offices (FMI, MET Norway, SMHI) in the Nordics - was in a sense a follow-up to one of the early use cases. iOBS ran as a NeIC project between 2019 and 2021 and as an extended NeIC affiliate project for a year after its end.

Another goal of Glenna was to build access mechanisms to the resources. Building on top of the national authentication mechanisms, such as Haka in Finland, Feide in Norway and SWAMID in Sweden, Glenna wanted to use a general mechanism for users from everywhere in the Nordics and decided to use the now defunct Kalmar2 service. Here, they were very successful. For example, Glenna managed to open the University of Oslo’s Lifeportal, a service to ease the access to high-performance computing resources for researchers in Bioinformatics, for Nordic users through the Kalmar2 Nordic authentication mechanism. Kalmar2 was a “super federation” of all the national Nordic federations. Still says this, in addition to the use cases, was one of the more significant outcomes of the Glenna project.

The cloud activities that Glenna initiated were continued under GÉANT (Collaboration of European National Research and Education Networks), in the current GN4-3 Cloud Services effort, which also has the responsibility to support procurement of commercial cloud services for use by European NRENs. Already during the project period, Glenna had opportunities to influence the European cloud developments, which in turn brought investments and opportunities to the Nordic centres. According to Still, this is an outcome and a long-term impact that would not have happened if NeIC didn’t host Glenna.

Bureaucracy may hinder collaboration

Still tells that in Glenna, they sometimes found it problematic that the Nordic countries had already built services and mechanisms with overlapping function, use and support. Significant investments had been placed in developing these national services, and finding a new, pan-Nordic solution that would maybe replace some of them was not easy. This problem, Still adds, was not technical but political, and he has noticed the same issue also at the European level; sharing nationally tax-funded resources is challenging.