Data Management Consulting Services for ESMIA Consultants

ESMIA Сonsultants unites experts in energy system modeling who help firms and government institutions design sustainable energy strategies. The Client wanted to optimize the time and effort spent on data operations, so they sought a partner to address the challenge with a custom solution.

We analyzed the Client’s setup, clarified needs, and suggested a more suitable and budget-friendly data management plan – integration of a ready-built Datuum service that could cover all requirements with a few additional customizations. The Client opted for the idea, so we stuck to it and built a new system with extended capabilities, including:

• Streamlined data processing (minutes instead of hours) for large datasets containing dozens of millions of records
• Advanced reporting that allows modelers to swiftly create rich and accessible online reports for their end-clients
• New Change Management process enabling ESMIA Сonsultants to control every modification in the system and make sure it fits their requirements
• Modern UI allowing the Client’s experts to intuitively interact with the system and more efficiently manage complex datasets

Initially, the Client used a combination of the legacy modeling system and Excel for their data operations. So, we started with mapping out their existing data flow to identify key bottlenecks. Our team discovered that Datuum default functionality could resolve most of the processing issues. As for the rest, we suggested adopting a set of modern services – PostgreSQL, Power BI & Azure Cloud to replace Excel.

Together with the Datuum team, our experts tailored the new four-component system to meet all the Client’s needs. Also, we helped move the company’s data management system to Azure Cloud and assisted in other directions, including:

• Developed custom data upload/download and single sign-on features to streamline routine tasks
• Extended Datuum with bespoke UI modules enabling it to process data from GAMS /Excel in their default formats
• Implemented integrational and load testing to detect and correct possible issues
• Took over ongoing support and maintenance to ensure stable system performance

ESMIA Consultants regularly provide clients with reports on their modeling results. The existing data management operations took hours of manual work in Excel and the legacy software system to create reports with diverse charts. Another goal for our team was to build an efficient process for reporting & data visualization.

We leveraged Datuum to automate the data processing and consolidation into a unified format for reporting. It allowed us to tangibly reduce manual efforts and speed up the report preparation to 5-10 minutes per document. Further, we helped integrate Power BI into Azure infrastructure to enable interactive charts and configured:

• Azure Cloud, Power BI, and Datuum consolidation into one cluster so the Client can easily adjust it according to their needs
• Data filtering that allows ESMIA Consultants to split large datasets into smaller parts and flexible aggregations for more efficient analysis using data visualization charts
• Multiuser access for the Client’s team to speed up the reports turnover