Data quality – the practice of testing and making sure that the data and datasets you use are what you expect them to be – has become a staple in the world of data science. Data can be the “new oil”; but if it’s too coarse you may not be able to use it.
Today, a startup developing tools to measure and ensure the quality of the data you use is announcing some funding, a sign of how attention has shifted to this area.
Superconductive – a startup best known for creating and maintaining the open source data quality tool Great Expectations – has raised $40 million in a Series B funding round. It will use the capital to both build out its open source product and community as well as ready its first commercial product – a less technical and more accessible version of Great Expectations that can be used more than just engineers and data scientists – slated to to launch later this year.
Once the commercial offering is released, it will be named Great Expectations Cloud.
As Abe Gong, the CEO and co-founder of Superconductive puts it, data quality has long been a priority for engineering and data science teams. But as data use and access become increasingly democratized in more and more digitized organizations — thanks in part to low-code and no-code software — data quality becomes a consideration (not a “problem” or “challenge,” Gong is quick to point out) for more. people. The thinking is that having data quality tools that more people can use and understand gives people the ability to understand and resolve limitations or gaps.
“The broader question is, how does everyone in the organization get to a point where they rely on what the data is doing and what they’re trying to do,” he said. “The engineering team may trust it, but it may not be aligned with other teams. It doesn’t matter if it’s correct, it’s still a question of whether data is suitable for the purpose for which I want to use it.”
Even without a commercial product, Salt Lake City-based Superconductive is getting a lot of attention from high places. Tiger Global leads the round, with past donors Index, CRV and Root Ventures also taking part. The company is not disclosing its valuation, but we understand the dilution is less than 15%, bringing it to over $267 million.
The funding comes less than a year ago since Superconductive raised a $21 million Series A in May 2021. Part of the reason investors have come knocking so soon after the latest round is because of the strong traction of its open source tools.
Great Expectations is currently seeing more than 2.5 million monthly downloads (closer to 3 million, Gong told me), while members of its community, who maintain it on Slack, have now surpassed 6,000 (the downloads are based on machines running Great Expectations). , while the Slack users are engineers who actively work with the tools). Companies using it include Vimeo, Heineken, Calm and Komodo Health; and it’s also making its way into use through ecosystem partners Databricks, Astronomer, Prefect, and more.
Great Expectations began when Gong and his co-founder Ben Castleton
James Campbell – both computer scientists with decades of experience together – initially built tools to address the problem of data quality for organizations working in healthcare. In the end, they changed the company to address the greater opportunity: the problems healthcare organizations faced were the same as those faced by companies in other industries.
The crux of the matter is that when engineers build analytics or other tools to work with data, they may not consider whether the data ingested by those tools is in the right state to be used correctly (example: dates entered in the same, consistent format, or, if not, how best to reorganize them). Or they may not have thought about the different ways that users of the analytics might use them. For example, what happens if an analysis dashboard is suddenly looked at in the middle of the month at the end of the month? Do the insights remain consistent or do they completely reject people because of the way the formula and processes are set up?).
“By the end of the month the numbers would be correct. You could see a drop in sales in the middle of the month,” Gong said. “The engineering team might say it’s correct because the system is still doing the math, but from a business perspective, a lot can get confused even if the system is working correctly.”
Great Expectations tries to “fix” these situations with tools that help set parameters for data to ensure it remains consistent and at the same level of quality. The so-called “expectations” repository — some built by Superconductive and many built by the community — are declarative statements set up to make sense to both humans and computers, so they can do the work behind the commands.
Superconducting cites figures from Gartner that support the idea that data quality is a growing problem for organizations. The analysts estimate that organizations are currently costing $12.9 million annually because of poor data quality — both because the data hasn’t performed as it should, but also because of the decisions the bad data has led to. Gartner predicts that this year, 70% of organizations will turn to tracking data quality levels to address this.
That also means that Superconducting has competition. Companies such as Microsoft, SAS, Talend, and others have built data quality tools to complement other data services they provide. Gong also said that many companies are building homegrown solutions, although they may run into limitations, as internal tools often do. Superconductive believes it has many opportunities in space for several reasons.
First, the fact that it already has a large community using its open source tools, which becomes a funnel for users of the commercial product. Second, it is committed to the task of data quality.
“Others tend to cut it differently,” he said. “Sometimes you hear about data quality in the context of data observation and that’s why it’s aimed at engineers and not at the broader role. We see ourselves as different, a bottom-up open solution that sees the broader scope of this as our mission, not just a technical problem.”
Investors, especially those who have experienced the pain points of software debugging themselves, and knew the same issues existed with data, seem to agree.
“The vision was simple, yet ambitious: to create one place to observe, monitor and collaborate on the quality of your data at any level, on any system,” wrote Index Ventures’ Bryan Offutt at the time of their first investment in the company in 2021. “By providing data teams with an end-to-end way to monitor quality from pipeline to production, Abe wanted to bring the same ability to detect and solve problems that exist in traditional software to the world of bring dates. † Finally, data teams were able to detect problems before they made their way to end users. It was as if Abe had read the book on every problem I had encountered as an engineer working on data pipelines. It felt like the data world had its own DataDog.”
Updated with the correct co-founder name. James Campbell is the CTO who helped build Great Expectations.