StarTree Inc., a company that’s commercializing the open-source Apache Pinot real-time data analytics platform, is setting its sights on the market for observability tools, saying it’s a ripe target for disruption.
Last week the company demonstrated how its StarTree Cloud, a managed Pinot service, can now be used as a time-series database compatible with the open-source Prometheus monitoring and alerting toolkit to drive real-time observability dashboards using the open-source Grafana visualization engine. StarTree said it can address the query and storage components of the observability stack at a much lower cost by using open-source technology.
Observability is a discipline and set of tools that provide insights into a system’s internal state and behavior based on the data it generates. The process typically involves collecting, analyzing and understanding three key types of data — metrics, logs, and traces — to monitor the health, performance and behavior of complex systems. Observability has become an essential feature of cloud-native architectures because of their distributed nature and large number of moving parts.
Like many early-stage markets, the observability landscape has been dominated by a few large companies with proprietary technology stacks and data formats. Many observability vendors also have complex pricing models based on factors such as data ingestion, number of agents, users, features, data storage and retention periods. costs can run as high as 30% of total infrastructure spending, and users often complain that the platforms are too expensive.
Chad Meley, StarTree’s senior vice president of marketing, said those are the characteristics of a market that is primed for disruption from below by open-source alternatives and low-cost competitors.
Pressure from below
“Super-profitable categories get introduced as monolithic, but as they are adopted, they force the environment to decompose,” he said.
Meley cited the example of his former employer, Teradata Corp., which achieved a market capitalization of nearly $13 billion in 2012 by selling a tightly integrated hardware and software package. As competitive platforms emerged that decoupled computing, storage and software, the price premium for an integrated technology stack became harder to justify. Teradata is still a successful software company but at a much lower market capitalization.
The same dynamics apply to observability, Meley said, “but it’s a jump ball because no one has figured out how to assemble the full stack.” StarTree isn’t proposing to do that, but the company thinks it has a compelling argument to establish its Pinot-based engine as a less expensive and more functional query and storage engine than those sold by the observability leaders.
At last week’s Current 2024 streaming data conference in Austin, Texas, it showed how its Pinto-powered StarTree Cloud can be used as a Prometheus-compatible time series database to drive real-time Grafana observability dashboards. Both Prometheus and Grafana are well-established open-source standards.
StarTree has recently added support for PromQL, a query language that lets user select and aggregate time-series data in real time with Prometheus. The company also plans to add support for LogQL and TraceQL, which are open-source query languages for logs and traces, respectively.
“Then you’ll have one open-source-based query and storage engine” for the three principal types of observability data, Meley said. “We think we’ll emerge as the standard.”
Pinot advantage
Six-year-old StarTree, which has raised $75 million in venture funding, thinks Apache Pinot already has many features observability teams will like at a far lower cost than what they currently use. Pinot allows for real-time ingestion and querying of data with sub-second response times, even for complex queries over large datasets. It can scale up to handle petabyte-sized datasets, supports a large library of plug-in extensions and integrates with popular analytics platforms such as Apache Kafka, Apache Spark and Apache Hadoop.
“We’re a columnar database, so we ingest in real time, index in real time and query in real time,” Meley said. “You get the benefits of the SQL dialect without the latency. You also get the scale that comes with the columnar structure.”
StarTree isn’t ready to announce its entry into the observability market just yet, but “we are going aggressively after it because we see some inherent advantages [of Pinot] and some pretty minor product gaps to fill,” Meley said. “We already have some existing customers that want to use us for this use case.”