لخّصلي

1.9 Case Study: Weather Data Analysis
Let us look at a case study of using the big data stack for analysis of weather data. To come
up with a selection of the tools and frameworks from the big data stack that can be used for
weather data analysis, let us first come up with the analytics flow for the application as shown
in Figure 1.10.
Data Collection
Let us assume, we have multiple weather monitoring stations or end-nodes equipped with
temperature, humidity, wind, and pressure sensors. To collect and ingest streaming sensor
data generated by the weather monitoring stations, we can use a publish-subscribe messaging
framework to ingest data for real-time analysis within the Big Data stack and a Source-Sink
connector to ingest data into a distributed filesystem for batch analysis.
Data Preparation
Since the weather data received from different monitoring stations can have missing values,
use different units and have different formats, we may need to prepare data for analysis by
cleaning, wrangling, normalizing and filtering the data.
Analysis Types
The choice of the analysis types is driven by the requirements of the application. Let us say,
we want our weather analysis application to aggregate data on various timescales (minute,
hourly, daily or monthly) to determine the mean, maximum and minimum readings for
temperature, humidity, wind and pressure. We also want the application to support interactive
querying for exploring the data, for example, queries such as: finding the day with the
lowest temperature in each month of a year, finding the top-10 most wet days in the year,
for instance. These type of analysis come under the basic statistics category. Next, we also
want the application to make predictions of certain weather events, for example, predict the
occurrence of fog or haze. For such an analysis, we would require a classification model.
Additionally, if we want to predict values (such as the amount of rainfall), we would require
a regression model.
Analysis Modes
Based on the analysis types determined the previous step, we know that the analysis modes
required for the application will be batch, real-time and interactive.
Visualizations
The front end application for visualizing the analysis results would be dynamic and interactive.
Mapping Analysis Flow to Big Data Stack
Now that we have the analytics flow for the application, let us map the selections at each step
of the flow to the big data stack. Figure 1.11 shows a subset of the components of the big
data stack based on the analytics flow. To collect and ingest streaming sensor data generated
by the weather monitoring stations, we can use a publish-subscribe messaging framework