Data is most valuable in the moment when it’s hot. Machine, vehicle or asset transparency is key to delivering better user and customer insights.
But how do you enable high value use cases? Fraud detection in finance, real-time overviews and predictive analytics of fleets or packages in logistics are just some examples. Others are large-scale IoT use cases like home / building automation, connected devices in manufacturing or autonomous fleets in automotive.
One of the biggest technological foundations in building these kinds of services is processing and delivering real-time (also called streaming) data. While the “old” way of integrating and processing data is Extract, Transform, Load (ETL, known as batch processing) is robust and proven, valuable time is lost in this process.
That’s where Event-Driven Architecture (EDA) comes into play.
While the concept is anything but new (here’s a great read and overview about EDAs from 2006), it’s now getting increasingly relevant due to the impact and value of the use cases mentioned above.
Amazon connects in-store Go infrastructure with sensor, vision, and analytics capabilities via API calls. That means your API infrastructure needs to process API calls in 30 ms or less. For some use cases, you need as little as 6 ms. That might not sound difficult, but let’s consider that API infrastructure has to:
Ensure the API consumer – like your ride-hailing app – is correctly directed to the right backend resource, like the mapping service.
Is this API consumer a valid user allowed to access this backend resource? You have to authenticate the user to ensure it is.
APIs are arguably the gateway to your most critical application capital, so they better be secure.
Not all API calls are equal. You need a way to shape the traffic to avoid resource contention, provide proper bandwidth, and prioritise certain API calls.
Many companies process billions of API calls per day. How do you handle that volume? Caching API responses is one way to boost performance
That’s definitely a lot to consider and it can become overwhelming for developers.
Since you are reading this article, we guess that you are already aware that leveraging real-time data will help you. But you might ask yourself: where do I start? Most companies have neither the resources to undergo a full technical pivot nor they have to wave goodbye to their existing infrastructure investments. Several vendors in the streaming industry have established maturity stages that explain a companies’ journey towards being a real-time enterprise, step-by-step.
This is how Confluent, the company behind Apache Kafka sees it:
We love that idea of having a clear path towards being a global streaming company where all business units, devices, legacy systems are inter-connected and communicate in real-time with each other. Based on this, we have derived our own EDA Maturity Stages:
We want to help companies that are in the early stages of the EDA maturity model. This would be between pre-streaming to early adoption in the model by Confluent. Your first steps would be to compile more information around the topic, build demo pipelines on our platforms and use our tutorials on how to get started and finally, to evolve.
For those at higher maturity levels the complexity of bridging the streaming and request-response-based systems world quickly escalates. We have numerous examples for that as well to help you. Let’s connect for a follow on conversation for a deep dive.
As a starting point, we created this self-assessment which should take you only 5 minutes to complete. Find out how mature your EDA capabilities are and receive tips and assistance on how to best move forward!
Orchestration with existing APIs, web services, and established systems worlds is key to truly connect real-time streaming efforts as a reliable backend component into the enterprise application world. Xapix is that connecting layer to remove the complexity of orchestration, which allows you to focus on the real-time and scaling benefits resulting from the new technology.
With that being said - have fun taking the assessment. Please let us know if you have any questions!