The Big Data Europe Platform makes a lot of use of software developed within Apache projects: Spark, Flink, Kafka etc. have all been wrapped in Docker containers and, through the platform’s simple user interface, can easily be instantiated in a customised pipeline. The project is proud of its work and so was delighted recently to have the chance to show off in a special session at the Apache Software Foundation’s event in Seville, also called Big Data Europe. After a brief introduction from Simon Scerri (project deputy coordinator) that followed on from his earlier conference session, one of BDE’s lead engineers, University of Bonn’s Hajira Jabeen, explained the project’s many technical advances. These include as its use of Docker Swarm and the development of various init daemons to make sure that in any pipeline, components are started in the correct sequence so that one doesn’t fail just because the previous one is not yet up and running. Finally NCSR Demokritos’s Angelos Charalambidis described some of the pilots that form the showcase and proof of concept for the BDE project.
In the question and answer session that followed, moderated by W3C’s Phil Archer, it was clear that the project’s ideas were well received. One recurring request was to contribute the Big Data Europe code back to Apache – was there a reason this hadn’t been done already? Nope – we’ll be glad to do it! BigTop is an Apache project that offers powerful packaging, smoke testing and virtualisation capabilities and in that context, BDE’s SANSA stack looks to be of particular interest. BigTop doesn’t use Docker, preferring Kubernetes, but the concepts are very similar and in follow up discussions at the project’s booth, it was agreed there was a lot each could learn from the other. Both sides agreed to pursue further contact in future.
All in all, the Big Data Europe project’s participation Apache’s Big Data Europe event was a success for all concerned and helped achieve the goal of putting big data technologies in the hands of specialists in other domains.