Wide area network (WAN) connections between clouds are moving from 1 Gbps lines to 10 Gbps lines, but with the recent advancement of digital technology, including IoT and AI, there is an even greater demand for faster, high-speed data transfers because large volumes of data are being collected in the cloud. Until now, the effective transfer speed of WAN connections has been increased using techniques to reduce data volume, such as compression and deduplication. However, with 10 Gbps WAN lines, there are enormous volumes of data to process, and existing WAN acceleration technologies usable on cloud servers have not been able to sufficiently increase the effective transfer rate.
Fujitsu Laboratories has developed WAN acceleration technology capable of real-time operation at speeds of 10 Gbps or higher. This technology utilizes a dedicated computing unit for various processes, such as feature value calculations and compression processing, mounted on an FPGA within a server. This allows for the parallel operation of these computing units by supplying data at the appropriate times, based on the expected completion of each calculation.
In a test environment where this technology was implemented on servers using FPGAs and connected via 10 Gbps lines, Fujitsu Laboratories confirmed that it achieved effective transfer rates of up to 40 Gbps, the highest performance in the industry. This technology has enabled high-speed data transfer between clouds, including data sharing and backups, facilitating the creation of next-generation cloud services that share and utilize large volumes of data across a variety of businesses and locations.
Fujitsu Laboratories intends to implement this technology, which can be used in cloud environments, as an application loaded onto a server equipped with an FPGA. It is continuing evaluations in practical environments with the goal of commercializing this technology in 2018.
Fujitsu Laboratories announced the details of this technology at the 2017 International Conference on Field Programmable Technology (FPT 2017), an international conference held in Melbourne, Australia, from December 11 to 13.
Development Background:
As the cloud has grown in recent years, there has been a movement to increase the efficiency of data and server management and maintenance by migrating data (i.e., internal documents, design data, and email) that was previously managed on internal servers to the cloud. Furthermore, as evidenced by the widespread adoption of digital technologies such as IoT and AI, there are high expectations for how work and businesses will be transformed through the analysis and use of large volumes of data, including images from cameras in factories and other on-site locations, and device log data. In response, there has been explosive growth in the volume of data passing over WAN lines between clouds, creating a need for next-generation WAN acceleration technology capable of high-speed, large-scale data transfers between clouds.
Issues:
WAN acceleration technologies improve effective transfer speeds by reducing data volume through compression or deduplication of the data being transferred. When transferring data at even higher speeds using 10 Gbps network lines, the volume of data that needs to be processed is so large that compression and deduplication processing speed create server bottlenecks. Therefore, to improve real-time performance, there is a need for CPUs that can operate at higher speeds or for WAN acceleration technology with faster processing speeds.
Regarding the newly developed technology,
Fujitsu Laboratories has developed WAN acceleration technology that can achieve real-time operation usable in the cloud at speeds of 10 Gbps or higher, using server-mounted FPGAs as accelerators. Efficient operation with WAN acceleration technology is achieved by using an FPGA to handle computationally intensive processing tasks that are difficult to improve on the CPU through compression or deduplication. This is done by efficiently connecting the CPU to the FPGA accelerator.
The details of the technology are as follows:
1. FPGA Parallelization Technology Using Dedicated, Highly Parallel Compute Units:
Fujitsu Laboratories has developed FPGA parallelization technology that can significantly reduce the processing time required for data deduplication and compression by implementing specialized compute units for data partitioning, feature value calculation, and lossless compression processing on an FPGA in a highly parallel configuration. This enables highly parallel operation of the compute units, delivering data at the appropriate times based on predictions of the completion of each calculation.
2. Technology to Optimize Processing Flow Between CPU and FPGA:
Previously, when determining whether to apply lossless compression to data based on identifying duplication, it was necessary to read the data twice—both before and after the duplication identification was performed on the FPGA. This increased overhead and prevented the system from delivering sufficient performance. By consolidating the processing transfer to the FPGA, managing both the duplication identification preprocessing and the compression processing on the FPGA, and using a processing sequence that controls how the compression processing results are reflected in the CPU based on the duplication identification results, this technology reduces the overhead between the CPU and FPGA of reloading input data and control exchanges. This reduces waiting time due to data and control transfer between the CPU and FPGA, providing efficient, coordinated operation of the CPU and FPGA accelerator.
via
10 Gbps connections. In a test simulating regular data backups, including documents and video, the technology achieved transfer speeds of up to 40 Gbps, an industry record. This technology has improved data transfer over WAN connections, enabling high-speed data transfers between clouds, such as data sharing and backups, and facilitating the creation of next-generation cloud services that share and utilize large volumes of data across a variety of businesses and locations.
Future plans:
Fujitsu Laboratories will continue to evaluate this technology in practical environments, implementing it in virtual appliances that can be used in cloud environments. Fujitsu Laboratories intends to make this technology available as a Fujitsu Limited product during fiscal year 2018.
