Abstract:
Numerical Weather Models (NWMs) utilize data
from diverse sources such as automated weather stations, radars,
and satellite images. Such multimodal data need to be transcoded
into a NWM compatible format before use. Moreover, the data
integration system’s response time needs to be relatively low to
reduce the time to forecast weather events like hurricanes and
flash floods. Furthermore, the resulting data need to be accessed
by many researchers and third-party applications. Existing
weather data integration systems are based on monolithic or
client-server architectures, and are proprietary or closed source.
Hence, they are not only expensive to operate in an era of
cloud computing, but also challenging to customize for regions
with different weather patterns. In this paper, we present a
Weather Data Integration and Assimilation System (WDIAS)
that uses microservices architecture and container orchestration
to achieve high scalability, availability, and low-cost operation.
WDIAS provides a modular architecture to integrate data from
different sources, enforce data quality controls, export data into
different formats, and extend the functionality by adding new
modules. Using a synthetic workload and an experimental setup
on a public cloud, we demonstrate that WDIAS can handle 300
RPS with relatively low latency.
Citation:
H. M. Gihan Chanuka Karunarathne, H. M. N. Dilum Bandara and S. Herath, "WDIAS: A Microservices-Based Weather Data Integration and Assimilation System," 2020 Moratuwa Engineering Research Conference (MERCon), 2020, pp. 289-294, doi: 10.1109/MERCon50084.2020.9185270.