Tools
3 Sept 2023
In the bad old days, the first step when building a database, analytics or business intelligence solution would be to order or provision a number of servers for hosting.
Particularly in the data world, you would have to worry about capacity planning and over provisioning of this infrastructure to account for future volumes of storage and processing. Without this, your data platform could be out of capacity before it is even ready to go live!
Next, you would need to purchase potentially software which was heavyweight and expensive and required long and complex procurement cycles. This software would need teams of skilled but expensive people to set it up and then manage on an ongoing basis.
To unlock the budget for all of this, you would inevitably need a waterfall style project and a business case.
Nowadays, the go-to tools and platforms in the data world are becoming completely SaaS and managed solutions. We simply begin using these tools and pay for them based on how much we consume out of OpEx budgets. There is no big procurement exercise, no setup time, and a large part of the technology is managed for us.
When designing a new platform today, the first place most enterprises would usually look is your cloud provider such as AWS, GCP or Azure. There, you will find a range of relational and NoSQL datastores, some of which completely insulate you from the concept of a server. The suite of serverless tools within the cloud providers arsenal is growing, spanning storage, ETL, analytics, streaming, business intelligence etc.
Outside of this, the leading vendors in the higher level tools are also increasingly serverless and SaaS. For instance, you might turn to Confluent Cloud for your streaming engine, Databricks for your analytics, Tableau Online for your business intelligence, Elastic for your search, or ClickHouse for your real-time analytics needs.
Leaving the infrastructure and middleware behind is great for total cost of ownership, in that you don't have to stand up and then maintain all of this technology. When we don't have to make a large up-front capital investment in infrastructure, licenses and project costs, you'll be able to innovate and experiment with data much easier and democratise it across the business.
Your time to value is also massively improved, allowing you to move "up the stack" and straight to building the data products and analytics which move the needle for your business.
In short, serverless, SaaS and consumption based pricing will be transformative for how data and analytics are used within industry, completely changing the dynamic for how project are delivered and the business value that they create.
