Building a mission-critical microservices cluster in Vodafone 170 microservices
At Vodafone Greece, we take pride in our ability to develop robust and enduring platforms that stand the test of time. Our extensive portfolio encompasses various platforms designed to handle every aspect of our business, ranging from customer billing and management to data retrieval and analysis.
One such platform that plays a crucial role in our operations is the Digital eXperience Layer (DXL). Even if you're a Vodafone customer who may not be aware of it, chances are you are already benefiting from the functionalities provided by DXL.
What is the DXL?
The DXL was designed to cope with the needs of the modern digital world. Our backend infrastructure was built with Enterprise Architecture in mind, many years ago, and it has served this purpose well over the years. The explosion of the Digital World made us face challenges in performance, deliverability, and scalability that were impossible to anticipate when those systems were originally designed.
The rise of REST APIs and the need for faster data serving prompted us to update our systems. We sought to overcome slow response times and ensure zero-downtime mechanics, while keeping critical infrastructure operational. To bridge the gap between REST clients and our backend, we started developing the DXL in 2017.
Acting as a robust middleware layer, it improves data serving speeds and safeguards uninterrupted operations, delivering enhanced performance, scalability, and deliverability. However, the DXL goes beyond mere translation; it also serves as an efficient caching layer. Leveraging real-time data streaming via Kafka, selected microservices capture updates from the backend databases of enterprise infrastructure. These updates are then stored in their entirety in MongoDB, ensuring swift and accurate data retrieval.
We are now a dedicated team of 30 skilled professionals, distributed across 16 development teams. Following the Spotify model, we have tailored our approach to meet the specific needs of Vodafone.
To create the DXL, we established foundational principles that have guided us from its inception and continue to shape our work today.
This article will explore these **principles** and how we remain dedicated to upholding them.
Uncompromising Performance
Currently, the DXL is responsible for the sheer amount of performance of Vodafone's Digital Services.
According to our monitoring tools, the average latency of our calls across all microservices is 200ms for all API calls. This number is even more impressive if we take into account the fact that the corporate backend is not tuned to deliver such API performance.
Looking back, there are some pivoting decisions we took to deliver this kind of throughput and efficiency.
Implementation of our Digital Datastore (CQRS Pattern)
A critical component of the DXL is the Digital Datastore, which incorporates the Caching Layer. To optimize data handling, we have adopted the Command Query Responsibility Segregation (CQRS) pattern. This approach separates data fetching from data storage, allowing for efficient and scalable operations.
By leveraging clever hooks, we stream changes from our corporate backend databases to the DXL's Kafka. A dedicated batch of microservices then processes these changelogs, transforming them into meaningful information that is stored in our MongoDB database. This seamless integration enables real-time data updates and ensures the availability of accurate and up-to-date information.
The implementation of the CQRS pattern within our Digital Datastore has delivered significant performance improvements, reducing API calls by an order of magnitude. In critical scenarios, we have witnessed a remarkable decrease in API response time, from 15 seconds to just 150 milliseconds. This enhanced performance empowers us to seamlessly execute complex data fetching and construct intricate user journeys within the MyVodafone application, all without compromising performance considerations.
Adoption of cutting-edge technologies
Our development stack is built upon an impressive lineup of technologies including Quarkus, MongoDB, Kafka, Apigee API gateway, and GitHub for community collaboration.
During a pivotal moment, we made the decision to migrate from Spring to Quarkus, seeking to enhance microservice boot performance. Despite the inherent risks of adopting a relatively new technology, our upper management demonstrated a refreshing openness to explore uncharted territory. This forward-thinking approach aligned with the innovative nature of our DXL project.
Through the adoption of Quarkus, we have witnessed a boost in developer happiness and productivity. Leveraging the standard JAX-RS framework and enjoying the convenience of hot-reload functionality, our team has experienced increased efficiency. Moreover, the simplicity of developing with MicroProfile interfaces has empowered us to swiftly create the necessary components, removing concerns regarding performance and resource usage. This newfound freedom has allowed us to concentrate our efforts on delivering a greater number of microservices and fueling our drive for innovation.
Efficient handling of HTTP requests has been a notable achievement for us. Recognizing the importance of reduced response times and seamless user experiences, we embraced a parallel-request-first approach for the entirety of the cluster. When fetching data from multiple sources, we would do it in parallel and not sequentially, unless there was a specific reason to. Using parallel I/O in all our projects, we significantly enhanced performance, particularly in scenarios where multiple microservices need to gather information. In the scope of many DXL journeys, a microservice needing to perform 10 HTTP requests to other data holders is not uncommon. Had we not embraced parallel I/O, such response times would not have been achieved.
Non-sequential HTTP requests force the developer to change their code workflow. Our adoption of reactive frameworks, such as Reactor for Spring, RXJava for Quarkus, and Mutiny as the default in Quarkus, has enabled us to effectively handle parallel requests and process responses and data seamlessly. This expertise in reactive programming contributes to building scalable and robust systems.
Attention to developer experience
As software engineers at Vodafone, we understand the importance of a robust software stack for an optimal developer experience. In our microservices journey, we encountered challenges like code duplication and maintenance. To address this, we took a forward-thinking approach and developed Common Libraries that became the foundation of our microservices architecture.
These Common Libraries encompassed essential functionalities such as authentication, authorization, logging and monitoring, and security utilities. By centralizing these features, we empowered developers to focus on their microservice development tasks without reinventing the wheel.
Embracing collaboration, we treated the development of Common Libraries as an internal open-source project on GitHub. This allowed our agile teams to contribute their expertise and insights, driving continuous improvement. Even after 5 years, library contributions are still treated as they are in an open-source world.
The adoption of Common Libraries brought significant benefits. They ensured consistency across microservices, adherence to best practices, and reduced redundancy. By addressing cross-cutting concerns at the library level, we focused on delivering innovative features and business value, increasing productivity, and accelerating time-to-market.
Conclusion
The Digital eXperience Layer (DXL) stands as a testament to our team's expertise in creating enduring platforms at Vodafone Greece. Our commitment to developing solutions that withstand the test of time has resulted in a diverse portfolio of platforms, seamlessly handling every aspect of our business operations.
DXL, acting as a silent force behind the scenes, ensures a seamless digital experience for our customers. Its role as a middleware and caching layer streamlines data access and simplifies interactions, allowing our customers to effortlessly access Vodafone services.
As we look to the future, DXL remains a vital component of our technology ecosystem, continually adapting to meet the evolving needs of our customers. We take pride in our ability to innovate and build platforms like DXL, empowering us to provide exceptional digital experiences to our valued customers at Vodafone Greece. With our dedicated team of software engineers and a commitment to excellence, we will continue to shape the landscape of digital services, offering cutting-edge solutions that stand the test of time.