Security was one of the main requirements as the system handles sensitive information. Yet, the solution had to be flexible enough to meet changing business needs. We designed the architecture that elegantly fits both requirements. We also added an ingestion framework to easily integrate multiple data sources and databases into one system as we knew that the Client would need data from other systems in the future.
Our Big Data experts analyzed the requirements, mapped those to data solutions available on the market, and selected the most effective set of components, transforming those into a data management & reporting system including:
- MS SQL Server & Tableau – ensure a robust Data Warehouse with all the data analyzed and represented in tables & charts (Tableau ensures high data security since it can be hosted on a local machine)
- SharePoint & JS technologies – allow to gather charts from reports in one place and simplify data representation based on a user’s needs (e.g. reports can be represented through a web page)
- Python – serves as a basic programming language as it’s simple, efficient, and supports lots of libraries, which safeguards required scalability
- Apache Spark framework – powers ETL operations, ensuring big data volumes processing
- Azure DevOps – safeguards code retention, information lifecycle management, and CI/CD management automation
- Prometheus & Grafana – safeguard servers monitoring and errors alerting with all the metrics visualized in the dashboard
- Jenkins – orchestrates data ingestion process