USA
High-Load Near Real-Time Analytics Platform Development
Sigma Software delivered data platform engineering and development services building an analytics platform that processes 2.5 million events per second and is used by over 30K companies globally
Customer:
One of the world’s biggest advertising media conglomerates
Business Needs:
In 2014 Aol. acquired a video ad platform Sigma Software had been working on. The new owner needed help with adjusting the solution to the Aol. ecosystem and ensuring that the platform can process highly increased data volumes.
Result:
Our Big Data engineers elaborated an efficient high-load data system with AWS Cloud-based infrastructure, microservice architecture, and custom reporting tool. The product is used worldwide providing a 24/7 accurate & scalable ad serving environment and processing 26TB of data daily.
Mastering such massive data volumes is an art rather than just an engineering task. The project was a great challenge and a fantastic experience for our data engineers & development team.
Collaboration Overview
Key Facts
The innovative advertising management platform we developed for a start-up called Vidible was acquired by Aol. in 2014. As a Vidible core technology partner, we continued working with Aol. and focused on adapting the platform to the increased loads gradually turning it into the enterprise-scale ad platform.
We created a new Big Data analytics platform and an advanced ad hoc reporting system based on the microservice architecture and AWS Cloud infrastructure. Our team continued working on the project through all the mergers, acquisitions & rebranding (Vidible-Aol.-Oath-Verizon Media).
- Over 2.5M events processed per second
- Smart real-time data analytics and fast custom reporting solution based on over 400 metrics (it takes only 1,5 minutes to generate a report)
- Custom service for automatic best-fit ad selection based on various tags and 100 additional metrics (i.e. theme relevance, popularity)
- Flexible campaigns configuration allowing to easily modify ad units & include new features on the fly
- Rich ad-in-ad technology enabling pre-rolls, interactive mid-rolls, synchronized banners, etc. to be included into campaign itself
Data Platform & Reporting
The video analytic platform was expected to serve millions of users & billions of video plays, thus the Vidible Data Platform and the reporting layer could no longer cope with the task.
Our team focused on redesigning the AWS Big Data platform’s architecture to reduce the data latency and be able to process bigger data volumes in a near real-time manner. As a result, our experts safeguarded that the data management platform could efficiently handle up to 120TB daily.
- Data warehouse: we redesigned the real-time Data Platform decreasing the data latency from 2h. to 5 min., enabling the ad management platform to process 2.5M events per second, ensuring data governance, smart monitoring & alerting over the infra and platform components
- Ad-hoc reporting solution: our team developed an advanced campaign analytics solution that ensured near real-time reports delivery to the end-users allowing them to keep up with the recent forecasts & trends; the newly developed custom reporting software safeguarded 24/7 reporting services and reduced report design time from days to minutes
Cloud Infrastructure
The custom analytics platform infra has initially been designed according to the best AWS & DevOps practices and relies on IaC (Infrastructure-as-Code) approach making the Big Data real-time analytics solution flexible and very easy to scale.
After the acquisition by Aol. the BI system infrastructure easily adapted to the drastically increased loads and we continued hardening and fine-tuning it along the way for better availability, security, and robustness.
- Infrastructure built on top of the Amazon Elastic Compute Cloud (EC2) service to ensure fault-tolerance and availability
- AWS Elastic Load Balancer used to automatically spread and drive traffic to the different EC2 instances thus ensuring horizontal scalability for all services
- Amazon Virtual Private Cloud and Availability Zones are implemented and configured to safeguard high security of the whole system and support diversity of services & environments
- Different databases used to handle different types of data (Redshift, RDS, DynamoDB) with Athena DB used as a master backup
Gallery
Testimonials
Sigma Software’s prior focus is to ensure qualitative services and coherent selection of engineering practices during the product development whenever we are talking about a start-up or a more mature business. This allows to lay a solid foundation for the product so that it could be easily scaled and adapted to further Clients’ growth and loads increase.
Reporting is definitely one of our biggest challenges. It is the case when the Big Data is really BIG and knowing we can master it to produce reports nearly in real time makes me feel proud.
Let us discuss how our team can contribute to your success