Course Overview
Streaming Data and Stream Processing

Introduction To Stream Processing

Lesson #2

In this lesson we will:

  • Introduce the concept of stream processing;
  • Introduce stream processing frameworks such as Apache Flink and Kafka Streams.

What Is Stream Processing?

Stream processing is the method by which we analyze, manipulate and respond to streaming data as it is received. Common examples of things businesses need to achieve with stream processing include:

  • Calculating real time analytics e.g. the total value of orders in the last hour;
  • Identifying situations e.g. a potentially fraudulent transaction;
  • Manipulating data - e.g. cleaning data to meet organisational standards;
  • Providing a real time customer experience - e.g. informing the user of wait times in a queue;
  • Filtering data - e.g. to focus only on the subset of data of interest.

By processing data in real time and building experiences such as the above, in this way, organisations are empowered to react quickly to changing conditions and respond to events as they occur.

Stream Processing Frameworks

Processing real time streams of data is complex. Not only do we have data volume and latency challenges, we also need to ensure correctness and reliability even in the case of failing hardware or crashing processes.

Rather than coding this logic from scratch, a number of Stream Processing frameworks have been developed which allow developers to work at a higher level of abstraction and leave lower level concerns to the framework. Popular choices include Apache Flink and Kafka Streams, though more are discussed in a subsequent lesson.

Next Lesson:

Challenges Associated With Streaming Data

In this lesson we will explain why working with real time and streaming data is challenging.

0h 15m

Work With The Experts In Real-Time Analytics & AI

we help enterprise organisations deploy powerful real-time Data, Analytics and AI solutions based on ClickHouse, the worlds fastest open-source database.

Join our mailing list for regular insights:

We help enterprise organisations deploy advanced data, analytics and AI enabled systems based on modern cloud-native technology.

© 2024 Ensemble. All Rights Reserved.