Lakhasly

1.9 Case Study: Weather Data Analysis Let us look at a case study of using the big data stack for analysis of weather data.To collect and ingest streaming sensor data generated by the weather monitoring stations, we can use a publish-subscribe messaging framework to ingest data for real-time analysis within the Big Data stack and a Source-Sink connector to ingest data into a distributed filesystem for batch analysis.Let us say, we want our weather analysis application to aggregate data on various timescales (minute, hourly, daily or monthly) to determine the mean, maximum and minimum readings for temperature, humidity, wind and pressure.To collect and ingest streaming sensor data generated by the weather monitoring stations, we can use a publish-subscribe messaging frameworkData Preparation Since the weather data received from different monitoring stations can have missing values, use different units and have different formats, we may need to prepare data for analysis by cleaning, wrangling, normalizing and filtering the data.Data Collection Let us assume, we have multiple weather monitoring stations or end-nodes equipped with temperature, humidity, wind, and pressure sensors.We also want the application to support interactive querying for exploring the data, for example, queries such as: finding the day with the lowest temperature in each month of a year, finding the top-10 most wet days in the year, for instance.Next, we also want the application to make predictions of certain weather events, for example, predict the occurrence of fog or haze.Analysis Modes Based on the analysis types determined the previous step, we know that the analysis modes required for the application will be batch, real-time and interactive.Visualizations The front end application for visualizing the analysis results would be dynamic and interactive.Mapping Analysis Flow to Big Data Stack Now that we have the analytics flow for the application, let us map the selections at each step of the flow to the big data stack.To come up with a selection of the tools and frameworks from the big data stack that can be used for weather data analysis, let us first come up with the analytics flow for the application as shown in Figure 1.10.These type of analysis come under the basic statistics category.

1.9 Case Study: Weather Data Analysis
Let us look at a case study of using the big data stack for analysis of weather data. To come
up with a selection of the tools and frameworks from the big data stack that can be used for
weather data analysis, let us first come up with the analytics flow for the application as shown
in Figure 1.10.
Data Collection
Let us assume, we have multiple weather monitoring stations or end-nodes equipped with
temperature, humidity, wind, and pressure sensors. To collect and ingest streaming sensor
data generated by the weather monitoring stations, we can use a publish-subscribe messaging
framework to ingest data for real-time analysis within the Big Data stack and a Source-Sink
connector to ingest data into a distributed filesystem for batch analysis.
Data Preparation
Since the weather data received from different monitoring stations can have missing values,
use different units and have different formats, we may need to prepare data for analysis by
cleaning, wrangling, normalizing and filtering the data.
Analysis Types
The choice of the analysis types is driven by the requirements of the application. Let us say,
we want our weather analysis application to aggregate data on various timescales (minute,
hourly, daily or monthly) to determine the mean, maximum and minimum readings for
temperature, humidity, wind and pressure. We also want the application to support interactive
querying for exploring the data, for example, queries such as: finding the day with the
lowest temperature in each month of a year, finding the top-10 most wet days in the year,
for instance. These type of analysis come under the basic statistics category. Next, we also
want the application to make predictions of certain weather events, for example, predict the
occurrence of fog or haze. For such an analysis, we would require a classification model.
Additionally, if we want to predict values (such as the amount of rainfall), we would require
a regression model.
Analysis Modes
Based on the analysis types determined the previous step, we know that the analysis modes
required for the application will be batch, real-time and interactive.
Visualizations
The front end application for visualizing the analysis results would be dynamic and interactive.
Mapping Analysis Flow to Big Data Stack
Now that we have the analytics flow for the application, let us map the selections at each step
of the flow to the big data stack. Figure 1.11 shows a subset of the components of the big
data stack based on the analytics flow. To collect and ingest streaming sensor data generated
by the weather monitoring stations, we can use a publish-subscribe messaging framework