학술논문

Big data transfer optimization based on offline knowledge discovery and adaptive sampling
Document Type
Conference
Source
2017 IEEE International Conference on Big Data (Big Data) Big Data (Big Data), 2017 IEEE International Conference on. :465-472 Dec, 2017
Subject
Aerospace
Bioengineering
Computing and Processing
General Topics for Engineers
Geoscience
Signal Processing and Analysis
Transportation
Throughput
Data transfer
Protocols
Data models
Real-time systems
Adaptation models
Mathematical model
Language
Abstract
The amount of data moved over dedicated and non-dedicated network links increases much faster than the increase in the network capacity, but the current solutions fail to guarantee even the promised achievable transfer throughputs. In this paper, we propose a novel dynamic throughput optimization model based on mathematical modeling with offline knowledge discovery/analysis and adaptive online decision making. In offline analysis, we mine historical transfer logs to perform knowledge discovery about the transfer characteristics. Online phase uses the discovered knowledge from the offline analysis along with real-time investigation of the network condition to optimize the protocol parameters. As real-time investigation is expensive and provides partial knowledge about the current network status, our model uses historical knowledge about the network and data to reduce the real-time investigation overhead while ensuring near optimal throughput for each transfer. Our novel approach is tested over different networks with different datasets and outperformed its closest competitor by 1.7x and the default case by 5x. It also achieved up to 93% accuracy compared with the optimal achievable throughput possible on those networks.