BACKGROUND
Most utility companies embark on immense projects to become more information-driven and customer-centric organizations but without efficient data analytics that potential often goes unrealized. Big Data is of paramount importance for energy firms embracing regulatory changes, technology shifts and market pressure for lower service cost and new services and products.
An Italian utility company guaranteeing the supply of electricity to the customers of the protected market, has trusted KDM Force and our partners to make its data become a valuable business asset.
With the liberalization of retail electricity sales, our client serves domestic customers and small businesses that decide not to switch to the free market and are supplied under the protection regime established by law. As an aggregator of demand, our client allows small consumers to benefit from the opportunities offered by wholesale markets in the same way as larger consumers with higher individual negotiating power. Though it does not generate energy, send it directly to customers, or sell it, our customer ensures that electricity is transported safely and reliably from where it is produced to the distributors. This includes balancing the system second by second, nonstop, to make sure supply always meets demand.
CHALLENGES
In order to manage the colossal information flows collected in the electricity and gas markets our client set up and runs the Integrated Information System. Through its centralized structure that interacts with the operators, such System guarantees safety and timeliness in the management of information flows and favors the competitiveness of the entire market. At the other end, every year more than 250 distributors and over 500 vendors exchange around 100 million data flows, involving just under 60 million consumers. In order to compete in such a dynamic environment successfully and provide reliable, safe and affordable energy, transforming fundamentally the present business processes for utilities today is a business imperative.
THE SOLUTION
KDM Force is inspired by the challenge to design and implement Big Data Infrastructure with the goal to guarantee the fluidity of data exchange and their quality while maintaining the crucial for our client highest standards of data security and privacy. Moreover, data available within the Integrated Information System can be leveraged to improve both the commercial policies of operators and the decision-making capacity of consumers, thus making the market more dynamic and efficient.
The solution involves the installation and implementation of the technological platform dedicated to the management of consumption measures, within the Integrated Information System of the client. This project is based on the Hadoop platform; in particular, Cloudera’s suite of programs was chosen for Big Data processing. Hadoop works by distributing data across computing clusters, either in the cloud or on-premises, sidestepping the failure rate limitations of even the most powerful single machines. The data collections can be scaled up to larger collections of clusters, known as data lakes. For each processing environment (Production and Pre-production) a data lake, accessible from the Cloud, is configured on shared Dell Isilon storage. Moreover, the disaster recovery process is thoroughly planned and tested, and automated data flows to a secondary site are well-orchestrated to keep all data available for eventual restoring operations. The diagram below represents the architecture of the solution.
To bring this architecture to life KDM Force has implemented advanced technologies step-by-step starting with the computing component and Dell Isilon storage set-up, through Cloudera solution deployment to Storage Unity solution implementation for the VMware Farm server.
Cloudera is a fusion of open-source Hadoop and proprietary software and the industry leader in building platforms for easing access to Hadoop’s depth of data storage, data governance and analysis. The increased functionality and simplicity of data analysis helps prevent data lakes from becoming swamps, limiting the effectiveness of Hadoop.
As an additional service, KDM Force designed and constructed a dashboard for monitoring the operation and performance of the systems and networks that make up the client’s technological platform (e.g. servers, storage, active network equipment, firewalls and devices of geographic network) using the Zabbix enterprise-class open source monitoring solution. Undeniably, a flexible state-of-the-art monitoring platform is the key to an efficient IT infrastructure composed by well-functioning machines, systems and networks.
With the staggering amount of data being collected on both the supply and demand side of the coin big data analytics is the answer that enables cost and time reductions, new product development and optimized offerings, and smart decision making. At KDM Force, we believe that with the power of technology utilities can match energy supply and demand more accurately and, therefore, provide better service at lower cost. However, this is only the beginning. By combining big data with high-powered AI, utilities can adopt efficient and easy-to-schedule predictive maintenance strategy, detect fraudulent behaviour before it affects the organization, optimize power quality and discover many other benefits hidden in their data.