The technical workshop of May 2016 focused on sharing information on key technologies as required by the Societal Challenges. Both developments on making the base platform easier to use, as discussions on database technologies colored the technical discussions. University of Athens hosted the workshop.
We started the first day with a recap of the work done so far to make sure all technical partners have the same view on the needs and status of the base platform. Currently there is a base platform available which can be installed through a manual installation guide or using Docker Machine. Multiple technical partners agreed that more extensive installation instructions would help disseminating the platform. The guides will be enriched with an installation of the BDE platform on blank in-house machines. We also aim to have installation screencasts available in the near future.
Next to the base platform, Docker images for (Big Data) components such as Spark, Kafka and Solr, are being developed. These images will serve as templates to build custom components for specific Big Data pipelines. We shared best practices and made further progress in selecting conventions for building the images. This will make it easier for the end-users to use these templates. It will also facilitate to try out new, related, technologies.
After the recap session, a lot of new work has been demonstrated. We started with the platform UIs that are being developed. This consists of UIs to build and monitor Big Data pipelines, SwarmUI – a UI on top of Docker Swarm – and an integrator UI that shows all existing component dashboards in one interface. Next, both the semantification of the data lake and the SANSA-Stack have been presented. More blog posts and webinars on these topics will follow.
On the second day the focus shifted more towards the technical challenges for the pilots. We shared knowledge and experience in the area of triple stores, mainly regarding performance, and geospatial and temporal storage systems. Several technologies have been demonstrated, including Virtuoso, 4store, SemaGrow and Ontop Spatial. All this was done in the light of the pilots that will be developed per societal challenge. We concluded the workshop with a pilot support session where we tried to solve pilot-specific questions and challenges.
The workshop was very fruitful. We are again a step closer to have a working Big Data platform on which we can deploy Big Data pipelines in an easy and user-friendly way. We all went home with a list of new challenges and topics that require further investigation. We are looking forward to the coming months where we will start running pilot implementations on the BDE platform.