The BigDataEurope consortium would like to thank all stakeholders for following the project, attending our activities and contributing to the results. Your endorsement and participation contributed to a successful outcome! 

Summary of Major Results

One can summarise the results of the project in three categories:

Strengthening the Societal Communities: the large number of project activities provided a platform for stakeholders interest in the seven societal challenges could come together, network, identify common issues and identify common solutions – the latter, at least the ones of a technical nature, largely supported by the project. We augur that a large part of the communities strengthened by the project will continue to work closely together.

BigDataIntegrator Platform: perhaps the most concrete project result, the BDI is a flexible and open-source platform that can be more easily deployed and customised to build Big Data pipelines that address open-ended challenges. Based on the Docker virtualization, the base platform is enriched with a layer of services that support the workflows’ setup, creation and maintenance. Supported by a simple graphical UI, it offers basic building blocks (e.g. Apache Spark, Hadoop HDFS, Apache Flink, etc.) to get started with common Big Data technologies. The BDI continues to be maintained (on Github) beyond the project, and is being used in various external projects and initiatives. Given it’s impact in the big data technical area, it is also being proposed as an Apache Incubator.

Pilot Demonstrations: the value of the BDI has been demonstrated in 7 separate pipelines that target a selected big data problem from each of the 7 societal challenges. These pipelines extend a predefined BDI pipelines designed for each of the societal challenges, based on the requirements identified for each domain. The pilots have been useful to demonstrate the flexibility of the BDI platform and its ability to target big data challenges of a varying from any domain. Some of the final pilots have taken a life of their own, and are being extended in follow-up projects and initiatives.

In addition to the above, the large amount of valuable material generated by the project is available for posterity on the project profiles:

Youtube Channel (50 videos): The recorded technical webinars (e.g. final BDI launch, relevance in the context of activities like the Big Data Value Association) and societal hangouts; demonstrations of BDI and its pilots, and interviews with the developers. 
SlideShare (237 slides): All project presentations, from a wide variety of physical and online activities (including all workshops, webinars, hangouts and external events).
Flickr: Pictures from the physical events (workshops, conferences and other events).

Follow-up Projects

Below is a list of H2020 and other projects that build on the BigDataEurope results, most notably by adapting a BDI pipeline for their needs. The list is valid until early 2018 and might not be updated at a later date.

Big-data Earth observation Technology and Tools Enhancing Research and development is an EU-H2020 research and innovation project, started in November 2017 to the end of October 2020.

The project’s main objective is to implement Big Data solutions (denominated as Data Pipelines) based on the usage of large volumes and heterogeneous Earth Observation datasets. This should help addressing key Societal Challenges, so the users can focus on the analysis of the extraction of the potential knowledge within the data and not on the processing of the data itself.

To achieve that, BETTER is improving the way Big Data service developers interact with end-users. After defining the challenges, the promoters validate the pipelines requirements and co-design the solution with a dedicated development team in a workshop. During the implementation, promoters can continuously test and validate the pipelines. Later, the implemented pipelines will be used by the public in the scope of Hackathons, enabling the use of specific solutions in other areas and the collection of additional user feedback. www.ec-better.eu  

We are happy to announce SANSA 0.4 – the fourth release of the Scalable Semantic Analytics Stack. SANSA employs distributed computing via Apache Spark and Flink in order to allow scalable machine learning, inference and querying capabilities for large knowledge graphs.

• Website: http://sansa-stack.net
• GitHub: https://github.com/SANSA-Stack
• Download: http://sansa-stack.net/downloads-usage/
• ChangeLog: https://github.com/SANSA-Stack/SANSA-Stack/releases

You can find the FAQ and usage examples at http://sansa-stack.net/faq/.

The following features are currently supported by SANSA:

• Reading and writing RDF files in N-Triples, Turtle, RDF/XML, N-Quad format
• Reading OWL files in various standard formats
• Support for multiple data partitioning techniques
• SPARQL querying via Sparqlify
• Graph-parallel querying of RDF using SPARQL (1.0) via GraphX traversals (experimental)
• RDFS, RDFS Simple, OWL-Horst, EL (experimental) forward chaining inference
• Automatic inference plan creation (experimental)
• RDF graph clustering with different algorithms
• Terminological decision trees (experimental)
• Anomaly detection (beta)
• Knowledge graph embedding approaches: TransE (beta), DistMult (beta)

Noteworthy changes or updates since the previous release are:

• Parser performance has been improved significantly e.g. DBpedia 2016-10 can be loaded in
• Support for a wider range of data partitioning strategies
• A better unified API across data representations (RDD, DataFrame, DataSet, Graph) for triple operations
• Improved unit test coverage
• Improved distributed statistics calculation (see ISWC paper)
• Initial scalability tests on 6 billion triple Ethereum blockchain data on a 100 node cluster
• New SPARQL-to-GraphX rewriter aiming at providing better performance for queries exploiting graph locality
• Numeric outlier detection tested on DBpedia (en)
• Improved clustering tested on 20 GB RDF data sets

Deployment and getting started:

• There are template projects for SBT and Maven for Apache Spark as well as for Apache Flink available to get started.
• The SANSA jar files are in Maven Central i.e. in most IDEs you can just search for “sansa” to include the dependencies in Maven projects.
• Example code is available for various tasks.
• We provide interactive notebooks for running and testing code via Docker.

We want to thank everyone who helped to create this release, in particular the projects Big Data Europe, HOBBIT, SAKE, Big Data Ocean, SLIPO, QROWD, BETTER, BOOST and SPECIAL.

Spread the word by retweeting our release announcement on Twitter. For more updates, please view our Twitter feed and consider following us.

Greetings from the SANSA Development Team