The Elemental Data Engine adopts the microservice architecture, has a fully asynchronous, highly available, elastic and scalable technology stack, integrates distributed computing engines of different styles such as batch, streaming, and deep learning, and carries out corresponding driver development. The system seamlessly integrates the management of computing engine and computing resources as service invocation into the overall product architecture, with multi-level separation and abstraction. The background engine optimizes the process and automatically generates code for the logical calculation process customized by users through page interaction, decomposes tasks into jobs of different sizes, intelligently distributes them to the computing resource nodes hosted by the cluster for calculation, and feeds back the execution results in real time.
The Scala language is used as the main development language. Compared with the Java language, it has excellent expressive ability. Its strong type checking and functional features make the service more stable
Self developed process management engine, which supports local operators and seamless access to a variety of open source computing engines, automatically optimizes the computing process, and realizes Reactive style complex topology computing management through full asynchronous non blocking mode
Self developed cluster resource management framework to dynamically adjust computing resources and distribute the operating environment; Match appropriate operation nodes according to algorithm and data characteristics, and support advanced functions such as manual adjustment
The notification service based on Websocket provides users with real-time messages and running result feedback, and supports collaborative communication between different clients
Support multiple underlying data storage (HDFS/S3/OSS), and use Alluxio to improve data read/write efficiency
Support a variety of underlying distributed computing engines Spark and Flink, and optimize a lot of algorithms and architectures on the basis of open source products
Based on MXNet and TensorFlow deep learning frameworks, it implements multiple algorithms and provides powerful modeling capabilities with GPU
The micro service architecture based on ServiceMesh, combined with Docker and Kubernetes, realizes the automatic discovery, registration and deployment of services
Elemental adopts microservice architecture, has a fully asynchronous, highly available, elastic and scalable technology stack, and integrates distributed computing engines of different styles such as batch, streaming, and deep learning.
High performance process management engine
Self developed cluster resource management framework
Multiple underlying data stores
Multiple underlying distributed computing engines
Deep learning modeling capability
Provide elastic scaling deployment based on K8s
The front end adopts the cutting-edge MVVM architecture, which separates data, interface and interactive behavior. React+Redux works with WebSocket to ensure the real-time reflection of data updates on the interface. The throughout data visualization technology makes the complex data processing process intuitive, and can be flexibly modified without a line of code or mouse drag, Big data processing has never been so simple.
It needs appearance, interaction, experience and performance. Based on Facebook's open source React, Ant.design is deeply customized to make the user interface fresh, simple and smooth
Not only knowing, but also seeing pictures. With D3.js, every step of big data processing can be seen and touched, and a large number of charts and visual results can be displayed in real time
Completely separate the front end from the rear end, making the deployment and upgrade of front-end code flexible and smooth
The front-end routing system based on HTML5 history provides strong support for cooperative operation and task processing
NodeJS+MongoDB's golden partner, preferred for lightweight data applications
Webpack+Babel enables users to enjoy ES6 from the future in advance
Realize seamless data sharing among subsystems with PostMessage
GraphQL, the next generation API query language, makes the cooperation between the front end and the rear end more flexible and free
The drag and drop interaction and data visualization technology throughout make the complex data processing process intuitive.
D3.js supports visual interaction
Front end and rear end separation, asynchronous notification service
Lightweight data application