European manufacturer and retailer selling its products in over 60 countries on 6 continents with more than 7,000 points of sale. The jewelry giant constantly expands its global presence.
Seamless integration between multiple diverse systems
When expanding to new markets, which had specific regulatory needs and called for different functionality, European manufacturer and retailer implemented numerous third-party vendor systems supported by external contractors and using different technologies.
All of these systems needed to exchange data with each other. Integrations were often developed as batch-based data exchanges with a lot of non-reusable data, per market. As a consequence, each point-to-point integration became a long and resource-consuming project, which complicated the integration landscape further with each new market introduced.
As a result of such legacy architecture and ineffective data integration, the company suffered from high latency, data inaccuracy, and high maintenance costs.
So European manufacturer and retailer needed to find a solution that would help to:
- Reduce latency and increase data accuracy
- Increase scalability
- Eliminate data duplication and usage of extra memory and processing resources
- Reduce maintenance costs and complexity
- Make the efficiency and correctness of data exchange more trackable
- Eliminate operational and analytical data discrepancies
To solve these problems and build a solid foundation for digital transformation, European manufacturer and retailer decided to develop a scalable integration hub based on Event-Driven Architecture and Kafka.
The ultimate business goals of the solution were to:
- Ensure seamless integration between many different systems, making data exchange between them fast, efficient, and accurate
- Integrate new and old markets, unify functionality, and standardize data objects
- Shift from many-to-many to many-to-one
- Provide customers with personalized, omnichannel user experience (e.g., order online — pick up in store)
- Enable omnichannel inventory management and more efficient logistics and transportation management
- Provide real-time visibility into all business processes (via dashboards and BI reports)
- Ensure future-proof scaling
- Reduce development and maintenance costs
- Optimize performance
From in-depth analysis to implementation
Product Discovery Phase
We analyzed the legacy system and its limitations as well as investigated the current and upcoming needs of European manufacturer and retailer. With a precise focus on European manufacturer and retailer’s business needs, we helped our client choose the most suitable tech stack and architecture for the platform to reduce maintenance costs, withstand high load, and mitigate risks.
PoC, MVP, and Further Development
We provided European manufacturer and retailer with a strong engineering team. The team built a PoC and MVP and proceeded with further product development. Also, the Zoolatech team prepared a set of clear engineering standards to drive operational excellence and optimize costs.
Best DevOps Practices
We helped our client to streamline, unify, and standardize all DevOps processes as well as apply best DevOps practices.
- Security: Secretless approach and workload identity, secret key rotation, private endpoints.
- Automation: CI/CD pipeline (running a pipeline after each merge request), Infrastructure as a Code; building templates for deploying new services.
Site Reliability Engineering
Our site reliability engineers analyze European manufacturer and retailer’s IT ecosystem, monitor their software reliability in the production environment and help make European manufacturer and retailer’s software systems more reliable and scalable.
Omnichannel Inventory Management
We help our partner integrate their ERP, Transportation management, and Warehouse management systems and make the data exchange between these diverse systems fast, efficient, and consistent. It helps to provide real-time or near real-time visibility into all inventory levels and optimize logistics and transportation management.
Also, we help accelerate adoption of the platform among other teams by building templates, writing documentation, and transferring knowledge. What’s more, it helps to considerably reduce development time and costs.
Custom Kafka integration landscape
Having analyzed the main technological challenges and keeping a new vision in mind, the Zoolatech team developed a platform called Nexus. Nexus is a custom cloud-based integration hub based on EDA and Apache Kafka. It connects producers and consumers of real time data and facilitates the data exchange between them (e.g producer — ERP system; consumers — Warehouse management system, Accounting system, Reporting system).
The conceptual approach of the data exchange between different systems is based on business events. A business event comprises a data object and a real-time notification about something important that has happened (e.g., B2B order created: the notification about the order creation and the details of the order). Any interested consumer can get these notifications for operational processing and analytical purposes. Nexus allows the reuse of business event notifications streamed by their owners, helping to standardize and unify them.
Nexus utilizes Apache Kafka open-source distributed event streaming technology with extended infrastructure around it, allowing seamless connectivity to stream and to consume data.
The key challenge was to integrate Nexus with European manufacturer and retailer’s third-party vendor systems, which are large, have different architecture, functionality and data structures.
To solve the challenge and facilitate integration between Nexus and different business systems, our team has been implementing facades. A facade is a middleware layer that helps to address compatibility issues, modify and unify data, and do error-handling.
To make the platform more efficient, scalable, and secure, our team has extended the Apache Kafka integration hub with additional custom services and systems, such as:
Nexus Management System
It comprises an internal storage for history tracking and UI to search and debug single events. The system helps to improve the user experience for all end-users of the platform.
Tokenization Service
It allows replacing sensitive data with tokens, improving data security, and remaining compliant with regulatory requirements across markets.
What’s more, the messages streamed through Kafka are defined in the Central Schema Registry, which allows transparent data definition, standardization, reusage and data quality control.
What we have achieved together
Real-time data streaming platform
Together with the client, we redesigned European manufacturer and retailer’s analytics and integration platform from a batch point to point solution to near real time events that can be reused. We’ve built a scalable integration hub based on Event-Driven Architecture, with extended infrastructure around it to meet European manufacturer and retailer’s specific needs.
The platform is characterized by high scalability, low cost of maintenance, and high fault-tolerance. The new Event-Driven Architecture builds on a number of principles:
- Clear engineering standards in place for all integrations to ensure reusability, reduce costs, and drive operational excellence
- Making agile product teams responsible for data ingestion and consumption
- Exposing reusable data objects on the integration platform and business logic moved to the consumption and production layers
- Consistency and reusability of data objects
- Accessibility and reusability of integration modules
- Logging, failed message storage, and retry mechanisms embedded on all integration points to avoid data loss
- Event notification management for production maintenance and support
- Accessibility of the platform functionality for facades both within and outside the used network zones
- First-stream data ownership approach, which allows to eliminate discrepancies in data usage
- Shared-code client library diversity, which corresponds to the client’s engineering standards and personnel hiring priorities
In addition, the new custom platform, which is based on the open-source data streaming tool, fully belongs to European manufacturer and retailer and its maintenance is more cost-efficient compared to other third-party vendor solutions.
increased scalability, reliability, and efficiency
The product created a digital foundation for integrating all business processes and operating in a digital manner end-to end. It helps our partner to:
- Ensure seamless integration between multiple European manufacturer and retailer’s systems, making data exchange between them fast, efficient, and accurate
- Decrease latency from 36 hours to milliseconds and improve data accuracy
- Unlock the value of real-time data analytics and provide visibility into all business process
- Build a solid foundation for Advanced Data Analytics and AI
- Provide customers with omnichannel user experience
- Increase fault-tolerance
- Enable more efficient logistics and transportation management
- Ensure future-proof scalability
- Reduce maintenance costs and complexity
