Event-Driven Platform, Integration
Hub, Apache Kafka Event Streaming
New markets are opened each year, and the company interacts with its existing and potential customers offering an assisted customer journey in a number of ways, from physical store presence to online sales through the web as well as omni channel operations (e.g., order online, pick up in-store).
Our partner is going through digital transformation.
Not only tech is changing, but the whole company is as well. The journey from being a showroom retail company to an online tech-empowered one requires changes in focus, organizational structure, processes, mindset and culture. We help our partner with challenges specifically in tech, so that they can stay focused on the business impact, while we are delivering new solutions.
Our goal was to improve the efficiency and correctness of data exchange between numerous third-party vendor systems supported by external contractors and using different technologies.
Historically, and because of numerous markets’ non-standard local functionality and regulatory needs, the company has contractually implemented numerous third-party vendor systems supported by external contractors and using different technologies.
All of these systems need to exchange data with others and functionality differs considerably for various markets. With the fragmented distribution of knowledge about systems and their business functionality in each market, integrations were often developed as batch-based data exchanges with full-loads of non-reusable data, per market. Correspondingly, each point-to-point integration became a long and resource-consuming project in itself, which complicated the integration landscape further with each new market introduced.
Latency, extra memory, and processing resources, as well as complicated maintenance, were symptoms afflicting the company as a result of previous decisions made. The efficiency and correctness of data exchange were hardly trackable. Operational and analytical data discrepancies proliferated.
Having analyzed the main technical challenges and keeping in mind a new vision, we developed a custom cloud-based infrastructure solution as a new-generation integration hub, connecting producers and consumers of real-time data and allowing reusage of business event notifications streamed by their owners.
The conceptual approach is based on business events, which includes both data objects and real-time notifications about something important that has happened. Any interested subscriber can get these notifications for operational processing and analytical purposes. With a first-stream data ownership approach, discrepancies in data usage across domains and consumption type are eliminated.
The product utilizes Apache Kafka open-source distributed event streaming technology with extended infrastructure around it, allowing seamless connectivity to stream and to consume data, internal storage for history tracking, UI to search and debug single events, and deduplication service. Tokenization service allows hiding of sensitive data and remaining compliant with regulations across markets. The messages streamed through Kafka are defined in Central Schema Registry which allows transparent data definition, standardization, reusage and data quality control.
The new Event-Driven Architecture (EDA) builds on a number of principles:
- Distribution of the ownership of integrations by making agile product teams responsible for ingestion micro-services and consumption;
- Clear engineering standards in place for all integrations to ensure reusability, reduce costs and drive operational excellence;
- Building all real-time and near real-time integrations on the platform to limit the number of point-to-point and batch integrations;
- Exposing reusable data objects on the integration platform and business logic moved to the consumption layer;
- Delivery of data for both operational and analytics use cases.
Azure (AKS & ABS queues,CosmosDB, Application Insights)
The main functional and technological advantages include:
- Real-time data streaming;
- Technical solutions enabled within a high-quality ecosystem along with scalable architecture, achieved by using microservices;
- Consistency and reusability of data objects;
- Accessibility and reusability of integration modules;
- Logging, failed message storage, and retry mechanisms embedded on all integration points to avoid data loss;
- Event notification management for production maintenance and support;
- Accessibility of the platform functionality for facades both within and outside the used network zones;
- Shared-code client library diversity corresponding to the client’s engineering standards and personnel hiring priorities.
The resulting technical product has decreased integration latency from 36 hours to milliseconds, allowing real-time tracking and affecting the customer journey. The platform is characterized by high scalability, low cost of maintenance due to open-source, and cloud-based storage with an on-demand pricing model for cost efficiency as well as overall durability and persistence.
In addition, the new custom technological product developed around the open-source data streaming tool fully belongs to the partner and its maintenance is more resource-use efficient compared to other licensed third-party vendor solutions.