-
In many data centers, different type of servers generate large amount of data(events, Event in this case is status of the server in the data center) in real-time.
-
There is always a need to process these data in real-time and generate insights which will be used by the server/data center monitoring people and they have to track these server’s status regularly and find the resolution in case of issues occurring, for better server stability.
-
Since the data is huge and coming in real-time, we need to choose the right architecture with scalable storage and computation frameworks/technologies.
-
Hence we want to build the Real Time Data Pipeline Using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexmonster on Docker to generate insights out of this data.
-
The Spark Project/Data Pipeline is built using Apache Spark with Scala and PySpark on Apache Hadoop Cluster which is on top of Docker.
-
Data Visualization is built using Django Web Framework and Flexmonster.
Environment Setup
Development | Project Code Walk-through
Complete Project Demo
-
9Event Simulator using Python(Server Status Detail)
-
10Building Streaming Data Pipeline using Scala | Spark Structured Streaming
-
11Building Streaming Data Pipeline using PySpark | Spark Structured Streaming
-
12Setting up PostgreSQL Database(Events Database)
-
13Building Dashboard using Django Web Framework and Flexmonster | Visualization