Remote Experiment from the Opposite Side of the Earth
─ Demonstration of High-Speed Data Replication Relay via the Round-the-Globe Loop Path ─

Technical verification has been progressing for high efficiency data replication between ITER and the Remote Experimentation Centre (REC) in Japan. Transferring a huge amount of data simultaneously to multiple destinations may cause excessive loads and network bandwidth on the sender so that daisy-chained relay transfer would be a considerable solution. This study demonstrates how efficiently the replication relay could be realized for the next-generation fusion experiments, such as ITER and JT-60SA.

Technical verification has been progressing for high efficiency data replication between ITER and the Remote Experimentation Centre (REC) in Japan. Transferring a huge amount of data simultaneously to multiple destinations may cause excessive loads on network bandwidth for the sender, so a daisy-chained relay transfer would be a considerable solution. This study demonstrates how efficiently the replication relay could be realized for the next-generation fusion experiments, such as ITER and the JT-60SA.

All the LHD data were consecutively sent to the REC through the round-the-globe loop path (Toki - Gifu - Tokyo - Amsterdam - New York - Los Angeles - Tokyo - Aomori - Rokkasho) on SINET5 L2VPN, whose round-trip time is almost 400 ms. MMCFTP was used for the data transferring application. In both the Japan domestic path and the round-the-globe cases, every transfer sustained very stable speed at preset 8 Gbps. However, longer gap times were needed in initial negotiation to establish numerous parallel sessions. Optimally configured NVMe and iSCSI striped storages have shown higher throughputs than the ITER initial data rate of 2 GB/s. That knowledge enables the design optimization of not only the sender/receiver servers with their storages but also the intermediate relay server system.

The International Thermonuclear Experimental Reactor (ITER) is now under construction in Provence, France, with the aim of starting experiments in 2025, which will produce a large amount of experimental data, several petabytes (1015 bytes) per day, under an international collaboration framework of seven parties including Japan. Since all the data cannot be processed at the local site, each country cooperates to analyze the ITER data, and for this purpose, high-speed data replication technology and also a fast and large-capacity data storage system will be essential to transfer the data around the world.

The research team has been collaborating with the National Institutes for Quantum and Radiological Science and Technology (QST), i.e. the Japan domestic agency for ITER, and also the National Institute of Informatics (NII) to verify the essential technology by using the massive multi-connection file transfer protocol “MMCFTP”, developed by NII with utilizing the large amount of fusion experimental data obtained in the LHD. As a result, it has been mostly confirmed that the initial amount of the ITER data, 1 terabyte per every plasma discharge (2 gigabyte/second × 500 seconds), can be transferred from ITER in France to Japan.

However, if a large amount of data is sent directly from a source site to the multiple destinations, the network bandwidth and data reading load would be excessively concentrated at the sender, so that daisy-chaining the multiple destinations and relaying the data from one to another could be an effective solution. In this study, we established an L2VPN route (Toki-Gifu-Tokyo-Amsterdam-New York-Los Angeles-Tokyo-Aomori-Rokkasho) on SINET5 round-the-globe international links and conducted data transfer experiments, to verify whether this relay transfer can be efficiently realized on a global scale long-distance communication system. The communication response time for this round-the-globe route is approximately 400 ms.

All the LHD data existing at NIFS (Toki, Gifu pref.) is duplicated, once to the ITER Remote Experimentation Centre (REC) in Rokkasho, Aomori pref., and then immediately re-transferred to NIFS (Gifu) or NII (Chiba). Since the physical bandwidth of the domestic line at both ends is 10 gigabits per second, the experimental speed was set to 8 gigabits per second, 80% of the physical bandwidth. The pre-set speed was maintained almost stably for both round-the-globe and domestic routes.

However, through the round-the-globe test, it has been discovered that the MMCFTP takes a longer time to establish a huge number of parallel connections before data transfer begins, depending on the communication response time, i.e. distance. In the case of repetitive transfer tests of 18 gigabyte data, the efficiency of the domestic transfer (response time ≅ 16 milliseconds) was about 91%, while it dropped to about 71% for the round-the-globe transfer (response time ≅ 400 milliseconds). This is an important finding that will lead to future improvements in the data transfer algorithm, since the efficiency would decrease even further in transferring smaller files.

We also conducted some performance evaluation of the data storage systems simultaneously, and it has been also confirmed that NVMe SSD and iSCSI HDD striping storage have several times higher performance than the ITER initial data rate of 2 gigabyte pers second. On the other hand, SSD/HDD storage in a redundant configuration for data preservation did not provide sufficient data read/write throughputs. This information will be used to improve the technology and optimize the design of storage systems and relay servers to achieve sufficient performance to handle the 50 gigabyte per second data rate at the highest stage of the ITER experiment.

This research was conducted by Dr. Nakanishi, Hideya and his research group at the National Institute for Fusion Science (NIFS), in cooperation with Drs. Tokunaga, Shinsuke, and Ishii, Yasutomo, et al. at Rokkasho Fusion Institute of the National Institutes for Quantum and Radiological Science and Technology (QST), and Dr. Yamanaka, Kenjiro, et al. at the National Institute of Informatics (NII).

This research result was published in Plasma and Fusion Research, an electronic journal published by The Japan Society of Plasma Science and Nuclear Fusion Research on 10 February 2021, and the above figure also graced the cover of the March 2021 issue of the Japanese journal of the Japan Society of Plasma Science and Nuclear Fusion Research (Vol. 97, No. 3).

Publication

• H. Nakanishi et al., "Demonstration of High-Speed Data Replication Relay Across Multiple Repository Sites Using a Global Loop Path", Plasma and Fusion Research 16 (2021) 2405017.