In this lesson we will:
- Learn about batch processing;
- Learn about stream processing;
- Cover the difficulties in moving from a batch to a streaming model.
Batch Data Exchange
There are many situations where businesses need to move data from one location to another. For example:
- Copying data from one application into another e.g. eCommerce into a fullfilment system
- Copying data into a repository e.g. a Data Warehouse or a Data lake for later analysis
- Copying data for a partner organisation e.g. Sending orders to a third party warehouse who will fulfil the order
- Copying data into a location for backup, archive or audit purposes
Traditionally, businesses have done this using a batch model, whereby every hour, day or week, data.
From Batch To Streaming
Streaming technology involves moving data from source to destination as a continuous stream, typically immediately after it has been created.
By doing this, business can get instant visibility and response to situations much earlier.