The BDE technical team is very excited to announce the third public release of the BDI platform. We are thankful to all the partners, team members and everyone who facilitated the development by providing their wonderful feedback and comments.
Please register now to receive connection details before the Webinar.
We have successfully integrated several new components in Big Data Platform. The most prominent of them being the Big Data Integrator.
Big Data Integrator (BDI) Integrated Development Environment (IDE)
The Big Data Integrator is an application that can be thought of as a “starter kit” for big data pipelines. It is the minimal standalone system that helps you to create Big Data projects with multiple docker containers through an easy-to-use GUI.
Big Data Integrator acts as a “skeleton” application where you can plug & play different big data services offered by the Big Data Europe platform.
At its core it is a Web application that renders different services’ front-ends in a single view, thus allowing the users to navigate between each service with a sense of workflow continuity.
BDI has made generous use of the Docker ecosystem: all individual applications are packaged as Docker images. Docker Compose is used to start multiple containers at once and Docker Swarm is being used as the orchestrator. The prerequisites of getting started with the BDE platform is installation of Docker, Docker Compose and Docker Swarm.
The initial startup of BDI-IDE provides several components to the users which are briefly covered below:
Stack Builder: allows users to create a personalized docker-compose.yml definition describing the services to be deployed in the working environment. It is equipped with suggestions & search features to ease discovery and selection of components.
Swarm UI: after the docker-compose.yml has been created in the Stack Builder, it can be pushed into a git repository. From the SwarmUI, users can clone the repository and launch (start, stop, restart, scale etc.) the containers using a graphical user interface.
BDE Logger: provides logging of all the HTTP traffic generated by the containers and pushes it into an Elasticsearch instance, where it can be visualized with Kibana. To enable HTTP log collection for particular containers the user simply needs to set up the logging=true container label.
When visualizing data in Kibana, please make sure to specify the hars* pattern for the index so that the data can be discovered.
Workflow Builder: helps to define a specific set of steps that have to be executed in sequence, as a “workflow”. This adds functionality like Docker Healthchecks but more fine-grained. To allow the Workflow Builder to enforce a workflow for a given stack (docker-compose.yml), the mu-init-daemon-service needs to be added as part of the stack. It will be the “referee” that imposes the steps defined in the workflow builder. “Init_daemon”, given an application-specific workflow, orchestrates the initialization process of the components. It provides requests through which the components can report their initialization progress. The workflow builder reports the startup flow to init daemon that can validate whether a specific component can start based on the initialization status reported by the other components. The workflow needs to be described per application stack as it specifies the dependencies between services and indicates where human interaction is required.
WorkFlow Monitor: allows a user to follow-up the initialization process. It displays the workflow as defined in the workflow builder application. For each step in the workflow, the corresponding status (not started, running or finished) is shown as retrieved from the init daemon service. The interface automatically updates when a status changes, due to an update through the init daemon service by one of the pipeline components. It also offers the option to the user to manually abort a step in the pipeline if necessary.