Innovation Pilots Data & Analytics Services

In an innovative collaboration with a large government department, CIMSOLUTIONS launched three pilots to harness the power of data and analytics. The pilots focused on creating advanced dashboards that would provide crucial insights into energy regulations, price caps, the electricity market, Media Analytics, Satellite AI and AI SDG (Sustainable Development Goals). The goal was to develop a robust and scalable MVP on the Azure Cloud platform, with regular updates from various data sources.

Contribution

CIMSOLUTIONS provided expertise in various roles, including Data Engineer, Data Platform Engineer, AI Engineer, Cloud Engineer and Solution Architect AI. To keep the dashboards up to date, there is a need for multiple data pipelines on the (Azure) Cloud platform that regularly retrieve and process new data from both public data sources (CBS, RVO, ACM) and paid APIs (TenneT, KNMI, NetAnders, LexisNexis).

CIMSOLUTIONS has analyzed multiple business requirements of end users in these pilots as a Data (Platform) Engineer and applied them to multiple data products (data pipelines for the customer dashboards). In addition, CIMSOLUTIONS has drawn up and realized the data (platform) solution architecture (medal lakehouse architecture) for the respective pilots, set up and configured the required ETL (Extract, Transform and Load) components of the data pipelines (including for data annotation) based on various pipeline triggers, set up monitoring of Cloud resources consumption and performance and fully documented all deliverables.

Methods and Techniques

The team used advanced technologies such as MS Azure Cloud, MS Azure DevOps, MS Entra ID, ADLS Gen 2, Azure KeyVault, Azure Virtual Network, Private Endpoints, Hub-Spoke network topology, Azure Firewall, Network Security Groups, Azure Data Factory (V2), Azure Batch Account, Azure Functions (V2), App Service Environment (V3), Azure Virtual Desktop, MS Power BI Desktop 2, ArcGIS Pro 3.0, SAS Viya 2022.09+, Python 3.8+, Agile / Kanban.

Results

Dashboard Energy Regulations: subsidy schemes in the energy domain;
Dashboard Price Cap: developments in the consumer price of gas and electricity;
Dashboard Electricity Market: factors that influence the functioning of the electricity market (electricity consumption / generation);
Dashboard Media Analysis: semantic analysis of media reports;
Dashboard Satellite AI: object recognition on satellite photos;
Dashboard AI SDG: allocation of the SDG (Sustainable Development Goals) contribution(s) based on project descriptions.