The application of BigData technology is expected to impose a significant impact in near future in numerous sectors of the Energy domain, namely in Electricity production, transmission and distribution, in renewable energy production, in distributed production and smart grids and in energy saving.
Key players in the application field include Utilities and Operators for asset/system monitoring and forecasting; TSOs, DSOs and aggregators for system monitoring, smart grid metering, control, forecasting and demand side management; system manufacturers for fleet monitoring; renewable plant operators for system monitoring and resource forecasting; industrial and building sector in energy saving, etc.
The origin of data produced and processed in Energy domain activities present a high variety including the monitoring of complex electro-mechanical systems (O&M, condition and health monitoring CM/SHM, preventive maintenance, optimization based on historic data etc), the monitoring of energy flow on transmission and distribution grids (RTUs, smart metering etc), the forecasting of energy demand and renewable energy production (localized weather, access historic reanalysis data, model optimization etc), market data, the monitoring/optimizing/control of Internet connected distributed systems or components (BMS systems, inverters etc) and various socioeconomic, geospatial, resource and legislation data related to Energy policy making.
BigDataEurope platform will address selected topics, chosen through the project activities and interaction with its stakeholders. The initiation of the process was the organisation of the first of a scheduled series BDE Workshop in Energy domain, held in Brussels at the 16th of June 2015.
The aim of the workshop was the identification of current and future challenges for data management and analysis in the energy domain; challenges to be tackled with the evolving Big Data technology. In the workshop real examples of the potential, challenges and complexities of using big data in the energy domain were discussed. The workshop addressed an audience comprising data users from a variety of fields in the energy domain.
Distinguished key note speakers presented the views and data management related challenges in their field. The views of the Electricity Industry on Data value was presented by Mr Hans ten Berge Secretary General of EurElectric; The data management challenges in energy resource forecasting were presented by Mr Martin Qvist (Head of Super-Computing and BigData applications of VESTAS, the leading company in wind energy sector) and Prof. George Kallos from the Forecasting Unit of University of Athens. The data management challenges in system monitoring and a candidate use case on the topic were presented by Mr. A.Papoutsakis of TERNA S.A., a RES developer and operator. Finally the data challenges in Smart Grids field were presented by Dr. S. Tselepis from SmartGrid.eu platform and Mr Tierry Pollet (ETP on Smart Grids, Landis+Gyr).
The outcome of the workshops will support the design and realisation of the necessary ICT infrastructure on which the deployment and use of the BigDataEurope platform will be based. BigDataEurope platform (comprising the architecture, components, guidelines and best practices) will offer to the interested participating third parties the opportunities of the latest European RTD developments, including real time streaming, multilingual data harvesting, data analytics and data visualisation.
Specifically on the technical background, the BDE platform will be built upon existing Big Data industry best-practices making use of the Lambda-Architecture that constitutes generic, scalable and fault-tolerant data processing architecture. The envisaged implementation integrates mature, existing, open-source components into a comprehensive software stack suitable for serving and consuming interoperable data. The platform will be available as an open source implementation maximizing software re-usability and community involvement, while paving the new comer path to data products and services.
The architecture of the BDE platform is tailored to consume high-volume streams of real-time data (e.g. sensor measurements, social network activity, mobility data) and process them in two parallel pipelines, namely the:
- batch pipeline: that handles data at preset time intervals (e.g. hourly/daily) using Map-Reduce algorithms to provide aggregated views and
- real-time pipeline: that interactively manipulates incoming data and provides data views up to a certain timeframe
In the workshop, the system monitoring and forecasting group discussed the general attitudes of people involved in the energy sector towards big data usage and potential. The key points that were raised and discussed were the following:
- In the industrial sector the data are generated internally and as such they are proprietary
- Large companies develop in-house BigData applications or the rely on available commercial tools provided by the major ICT companies
- The majority of the energy industry related companies they do not exploit the full value of their data, as they do not invest in BigData solutions
- The convergence of Information Technology (IT) with Operation Technology (OT) is of primary importance; this is a field for BigData applications
- Available standards in data exchange in energy domain (i.e. IEC 61400-25).
The group considered two options for candidate pilot applications on BigDataEurope platform, namely the development of a platform capable to provide a complete asset fleet operational and condition monitoring and/or the development of a platform capable to provide the data management of localised (point) weather prediction in country level.
The technology group discussed the issues of data acquisition, storage and curation.
During the data acquisition phase, energy domain experts identified data heterogeneity as something that BDE could help with. Experts typically work with streams of data originating at sensors located on distributed asset devices. Other data of interest include more traditional, yet still streaming, multimedia data, such as video. These data are analysed both on the fly as well as in-situ, i.e. after having been stored in data centres. The various information models were discussed. For accessing data on a large scale some federated querying and aggregation solution would be required. This solution also needs to be able to convert the incoming streams into the desirable format, by making use of existing, standard mappings.
Energy experts indicated that they typically make use of in-house analysis tools, with R being the de facto standard. Other, commercial, software packages are also in use. Regarding processing and analysis, it often needs to take as soon as the data arrives, in a streaming fashion, as delays may incur costs, for instance when such analysis is used for the purpose of maintaining remote devices.
Regarding storage and curation, a number of items were raised, such as the need for mapping between standards used. It may be the case that these transformations take place on the fly in order to support streaming analysis, before results and byproducts are optionally transferred onto disk for longer-term storage.
BDE will need to cover the needs above by making use of the HortonWorks solution, which encapsulates technologies overlapping these required by the energy community, such as Hive+ORC. It will further need to provide relevant data-transformations in order to support the chosen pilot use-cases.