Hajira Jabeen reports on the latest gathering of BDE’s technical team.
The fourth BDE Technical workshop was hosted by NCSR-D in the beautiful and historical city of Athens. The representatives from technical partners enthusiastically attended the workshop.
The purpose of these workshops is to review the emerging technologies, select the ones that best suit the project objectives and determine the upcoming project goals and milestones. For a recap, in the first workshop, we determined the desired features of the BDE platform. In the second workshop we selected the particular technologies from a wide range of Big Data zoo, that would help in achieving the desired features. We reviewed our progress, addressed the challenges and prepared a road map for the future in the third workshop.
The workshop at NCSR-D was particularly focused on the use case pipelines for the societal challenges partners, and the flexibility of the architecture. The first day started by a warm welcoming introduction by NCSR-D followed by an architecture overview and experimental platform accessibility by InfAI. Later, Tenforce highlighted the latest advancements in fast-paced technologies, particularly Docker Swarm. The preceding fruitful discussion concluded that BDE platform would adopt the Docker Swarm based clustering of Docker containers, which is not only flexible, but also scalable and offers portability with negligible performance cost. This would also enable realization of use case pipelines with flexible connections among multiple containers in addition to resilience within individual containers of the pipeline.
The pipeline can be thought of as realization of a particular data value chain use case, with different tools performing various steps in the chain. These chain links (Big Data tools) must be able to communicate to each other without conflicts, in addition to many other desired features like resilience, scalability, flexibility etc. Docker Swarm enables creation of such a pipeline with most of the desired features of the BDE platform. Given that BDE focuses on reusing the existing open source tools, this selection neatly aligns with the project goals. One of the core feature of BDE is platform independence, which is the primary reason we opted dockers as a packaging tool for numerous competing, sometimes conflicting, Big Data tools. All the partners presented the dockers for different tools that address one or the other application domain of the Big Data.
One of the core focus of BDE is ease of use and low barrier to entry. The BDE aim is to enable user to access and use Big Data tools from a higher level of granularity than is usually required. To address this, InfAI presented their vision for the User Interface which was followed by discussion on what should be the part of user interface to keep it generic and easy to use and maintain.
The beautiful weather and serenity of Athens, in addition to the excitement of accomplishment was well worth having a dinner together at a cozy restaurant near Syntagma square.
The second day was dedicated to the presentations and discussion of interesting use cases of the seven societal challenge partners. It was enlightening to learn about exciting applications of Big Data at such a wide scope. The presentations ranged from variety of drug formulations, to crop estimation, geographical changes, traffic indirection and wind farm maintenance. We discussed the tools that best address the presented requirements to create the use case pipeline.
In the evening, the workshop ended with an excitement in the air, with a lot more to discuss, a lot more to explore, but the time limits are hard. We had to leave, to get started with the exciting new ideas and face the challenges ahead to reach another milestone. A milestone that makes the BDE platform looking better and mature than we see it now!