EnsembleJS is our new low-code framework which we use to rapidly develop real-time data applications powered by ClickHouse.
It dramatically reduces the time, effort and cost for businesses who are looking to make use of real time data, advanced analytics and AI to grow their business.
Why Is This Needed?
At Ensemble, we focus on building real-time and operational analytics systems. These systems typically need to ingest data really quickly, apply advanced transformations and analytics, and get the results into the hands of users and employees for immediate decision making.
One of the biggest challenges with this comes at the "last mile", specifically how we visually present the end results of this work to end users. And the main cause of this pain is that none of the front-end technologies are really fit for purpose.
For instance, we could use business intelligence tools such as PowerBI, Tableau, Metabase or Superset to serve the relevant information to employees. Though these tools have their place, they do not perform well when we have fast moving data and interactive analytics over very large datasets.
On the other hand, we could use low code development tools such as Streamlit or Dash. Though these are a step in the right direction, they can sometimes be limiting and opinionated with regards to the user experience. They are also Python based which is both a pro and a con.
Finally, we could build a bespoke application to disseminate the analytics, most likely using a React based framework. Though this is the most complex route and involves the most work, it is also the most flexible as it gives full control of the user experience.
Introducing EnsembleJS
Developing real-time analytical applications involves a fair bit of re-inventing the wheel. For instance, every applicaiton we develop will need to efficiently query for data, display data grids, plot line charts and interact with machine learning algorithms.
Our belief therefore was that by developing a reusable toolkit or framework using React, we could get the best of both worlds - the complete flexibility of a web application together with the rapid time to market of a low code tool.
EnsembleJS is this framework. It allows us to quickly build user interfaces that expose real-time data to business users or customers, but then have total flexiblity to configure those applications so they are most useful to the users.
Data Apps Not Dashboards
By building out front-ends in EnsembleJS, our users are able to move beyond simple dashboard type interactions, towards richer applications which interact with APIs to trigger actions and present information in creative ways.
As an example of this is our integration with large language model such as ChatGPT. These become infinently more powerful when they can interact with your private data so we decided to include the ability to interact with LLMs which can interface directly with your data.
Architecture
EnsembleJS applications are based on next.js, the leading React based application development framework. As with all React applications, there are a series of pages with each page consisting of a series of components:
Each of these components are reusable. For instance, we have a component for data grids, for bar charts, for line charts or for interacting with large language models. When we design a page, the components are configured with the query that is required to render the component.
Periodically, Cube will ask the database for the latest data, and send any updates to the browser to re-render exactly the required component. This keeps the page updating without any manual page refresh. For situations where faster data is required, we can also subscribe to updates from a web socket to render ticking data such as that seen in trading applications.
By default, EnsembleJS applications retrieve their data from ClickHouse, which we believe is uniquely suited to power applications like this. We also deploy a technology called Cube between the EnsembleJS application and the database which provides features such as a semantic layer, caching, logging, additional security hooks and subscriptions.
EnsembleJS applications would typically be deployed into a cloud environment such as AWS or Azure. There, we can easily interact with services such as AWS SageMaker for machine learning.
How We Will Use This
As a professional and managed services firm, we will primarily use this to accelerate our customer projects, bringing real-time analytics to our customers in minimal time and with less reinventing of the wheel. This said, we may consider open sourcing the framework in future should it be considered useful by the community.