The Big Data Europe Integrator Platform (BDI) is specifically designed to capture, manage and process the enormous amounts of data being created on a daily basis, with the objective to derive meaningful results and make a difference to people’s lives, solving thus societal challenges by putting cutting edge technology in the hands of experts in fields other than IT.
The updated BDI platform now includes a number of additional components and innovative features. Within the three years of continuous efforts of our brilliant technical team, BDI offers a production-ready toolchain for big data crunching, using open source tools like Docker and Docker Swarm in order to leverage a modular platform that can be very easily installed and used.
As Prof. Dr. Sören Auer, Director of the Technical Information Library (TIB) and Professor of “Data Science and Digital Libraries” at the Leibniz University of Hannover, who serves as Project Coordinator, stated “BDI synergistically integrates technology from the Big Data analytics and semantics technology realms. Especially for the societal challenges not only very large data analysis but also the semantic integration of heterogeneous data assets is of paramount importance”.
Indeed BDI has been successfully tested by the seven pilot use-cases, addressing thus the respective 7 Societal Challenges identified by the European Commission. More specifically, work in the Societal Challenge in the domain of Food and Agriculture has focused on the needs of the agri-food community in terms of big data, validating thus BDI in this sector. “BDE creates a very low entry barrier compared to other big data platforms” affirmed Valeria Pesce (FAO).
Martin Kaltenbröck (Semantic Web Company) and Ivana Versic (cessda eric) concur: “The joint work on the Big Data Europe project in the area of social science helped us not only to learn and apply big data principles and technologies in our organisations and projects, but also supported us in establishing a stable partnership beyond the project’s duration for our future endeavours”.
Dr. Georgia Aifandopoulou (CERTH) announced that “the work in the Societal Challenge of the Transport domain allowed the mobility lab of Thessaloniki to increase its data processing capabilities, mostly the amount of historical data that can be analyzed in short time, having thus a direct impact in the reliability of the traffic status predictions”.
In the Societal Challenge of Security and more specifically in the “Secure Societies” pilot, implemented by SatCen (Domain Leader), the University of Athens (Technical Leader) and NCSR “Demokritos” (Technical Support), a web tool for information extraction from heterogeneous data sources has been developed, which addresses the need expressed by the users to exploit the increasing amount of open data coming from space-based systems and other sources of information (e.g. online news). Therefore it can be deduced that BDI constitutes the ideal solution for all organisations and institutions aiming at advancing society with the innovative use of data, including open data.
From a technical standpoint, the platform provides monitoring and visualization tools for novice users while supporting full access and control for expert users at the same time. A rich documentation is provided by Webinars, wiki pages, instructions for installation, and accompanying videos on the BDE channel. BDI provides a number of ready-to-use components in docker images, while remaining extensible to new components that can be dockerized and integrated into the workflow.
In this release, BDI has been enriched with the provision of an Integrated Development Environment (IDE). The IDE bundles together various features:
- the stack builder to select from the ready-to-use components
- a SwarmUI to monitor the status of the cluster
- a logger service that provides logging of all the http traffic generated by the containers and pushes it into an elasticsearch instance
- a visualization with kibana within that elasticsearch instance
- the Workflow Builder to define a specific set of steps that have to be executed in sequence, as a “workflow”.
While BDE is approaching its completion, other projects have already started not only using and maintaining BDI but also further developing the BDE toolchain. For instance, BDI components are already used by many software projects such as Proteus, onebox, project-ember. Moreover BDI will be maintained and further developed by the SPECIAL project that will use and extend the semantification of the BDE’s big data stack with the objective to create a GDPR-compliant big data system. This will secure further contributions to the Ontario and SANSA stack within the BDE platform. In addition, the Holistic Benchmarking of Big Linked Data (HOBBIT) project utilizes BDI in order to deploy and run components such as Halyard, which is based on Apache HBase, Apache Hadoop and YARN technologies available from BDE repositories. HOBBIT evaluates various Linked Data technologies (e.g. triplestores) on a Docker-based HOBBIT platform, which can deploy all the BDE components natively. Last but not least please stay tuned as the upcoming projects that will use BDI will be announced soon on our website.