Weather data integration and assimilation system

Abstract

Numerical Weather Models (NWMs) utilize data collected via diverse sources such as automated weather stations, radars, air balloons, and satellite images. Before using such multimodal data in a NWM, it is necessary to transcode data into a format ingested by the NWM. Moreover, the data integration system’s response time needs to be relatively low to forecast and monitor timesensitive weather events like hurricanes, storms, and flash floods that require rapid and frequent execution of NWMs. The resulting weather data also need to be accessed by many researchers and third-party applications such as logistic and agricultural insurance firms. Existing weather data integration systems are based on monolithic or client-server architectures; hence, unable to benefit from novel computational models such as cloud computing and containerized applications. Moreover, most of these softwares are proprietary or closed-source, making it difficult to customize them for an island like Sri Lanka with different weather seasons. Therefore, in this research, we propose Weather Data Integration and Assimilation System (WDIAS) that utilizes microservices to achieve scalability, high availability, and low-cost operation based on cloud computing. The use of stateless microservices also enables WDIAS to add new features on the fly with rollover capabilities. Moreover, WDIAS provides a modular framework to integrate data from different sources, export into different formats, and add new functionality by adding extension modules. We demonstrate the utility of WDIAS using a cloud-based experimental setup and weather-related synthetic workloads.