Streaming
1 Sept 2023
As your business operates day to day, a number of events are taking place. Examples include orders, dispatches, customer enquiries or complaints, data from connected devices and of course those events more specific to your industry.
To build a truly compelling customer experience and an efficient business, we need to be able to monitor, analyse and react to these events in real time, and make automated changes in response to them.
Unfortunately, many businesses do not have this real time data processing capability. Instead, their data is trapped in siloed systems and perhaps integrated into a data warehouse with batch “extract, transform and load” where it is then analysed by humans days or weeks after the events have happened.
Real Time analytics is about moving beyond this, using modern approaches to understand what is happening “right now” across your business, and respond to situations immediately, intelligently and automatically in order to improve business performance.
By processing data in real time, businesses can dramatically improve their performance and bottom line.
In addition to using data for strategic long-term decision making, you'll be able to leverage it for operational purposes to identify immediate the steps you can take right now to grow your business and improve efficiency.
This of course improves the customer experience, which becomes more proactive and more personalised through data insights.
Likewise, employee experience can be improved by arming your people with an “up to the minute” view of what is taking place right now, advising on their “next best action”.
Using streaming real time data, we can also detect anomalies and situations of interest as they happen, and ideally intervene quickly or automatically before they ever impact a KPI.
This all ultimately feeds through to increased market share and revenue, and decreased costs by operating a more efficient business.
Real Time Data processing is a very valuable capability. However, technically it is a hard problem to solve.
First, it is a data intensive task, potentially requiring thousands of events to be processed in parallel and with low latency. These can vary in load, such as a sudden spike in user activity or machine data which needs to be ingested without delay.
Many scenarios also need to be very accurate, in some instances providing exactly once processing such that we never lose or never double process a message. To achieve this, every part of the technology stack needs to be reliable to failure.
The analytics we need to perform over real time data could be complex in nature. For instance, we might need to aggregate data and ask questions across data streams and across time windows and spanning both historical and real time data. We also need to deal with edge scenarios such as errors, anomalies, duplicate or late arriving data.
Individually all of these are solveable, but to deploy real time event stream processing with good performance, reliability and which provides the type of complex analytics that we need to can be a large undertaking.
The immediate opportunity is to expose real time analytics to your users through dashboards and reports, so that they can see operationally what is taking place in the business.
This turns business intelligence from a backwards looking activity into something which can guide your employees “next best actions” in order to improve business outcomes
However, the real value starts to come when we automate responses to situations before they even impact a KPI. For instance, having identified that a certain warehouse is shipping orders late, we might wish to re-route orders to another destination until the original warehouse has caught up.
This is the definition of an intelligent business. where we are observing the state of the world, processing data intelligently, and automating responses intelligently and automatically. Streaming Analytics are the basis for doing this.
