［ 2017年02月28日 ］
RIKEN International Symposium on Data Assimilation 2017
"A Flexible I/O Arbitration Framework for netCDF based Big Data Processing Workflows on High-End Supercomputers"
We present an initial study of a direct communication framework designed for complex workflows that eliminates unnecessary file I/O among components. Specifically, we propose an I/O arbitrator layer that provides direct parallel data transfer among job components that rely on the netCDF interface for performing I/O operations, with only minimal modifications to application code. We present the design and an early evaluation of the framework on the K Computer using up to 4800 nodes running RIKEN's experimental weather forecasting workflow as a case study.
|名前:Jianwei Liao||所属:Southwest University of China|