Valitive

B2B consumer data handling application designed over microservice architecture for processing, cleansing, filtering and transferring data related to the entire population of Sweden.

Size of the project

Technologies / Tools

CI/CD Solution for Microservices

All the application repositories are managed on GitLab along with a clean Git Flow pattern. TeamCity & AWS CodePipeline are used for CI/CD automation. The CI/CD pipeline is designed to build, check, and run automated tests of Spring Boot apps. Once the build is successful, it's packaged as a Docker image and registered in ECR for releasing a new version. All the new containers are deployed to ECS seamlessly with zero downtime of the live system.

Automated Data Cleansing Process

This automation aims to get updated data from different sources and make those available in a structured B2B exchangeable format that involves various data merging & verification processes.

Beyond the DevOps Role

Our DevOps team worked beyond their duties and designed a fully automated data cleansing process that connects around 5+ data sources with a volume of around 120M records, using Apache Kafka. It uses different Kafka brokers for different data types and produces data to MySQL, an Elastic Index & static data files. This Kafka setup runs on a Kubernetes cluster and processes data on a per-scheduler basis.