Enterprise data engineering teams can now observe, fix, and optimize their Spark data applications, in-motion
definity announced the general availability of its pioneering Data Application Observability & Remediation platform for Spark data analytics environments, marking a significant advancement in data operations. The company is also announcing it has raised $4.5 million in a Seed funding round led by StageOne Ventures, with participation from Hyde Park Venture Partners and additional strategic angel investors.
Faulty data operation and poor data quality cost an enterprise $12.9 million each year according to Gartner, highlighting the critical challenge for data engineering teams to maintain the reliability of data applications (data pipelines), optimize their performance, and prevent data downtime.
This is particularly urgent for enterprises deploying new AI initiatives, relying on accurate feature engineering and model fine-tuning. The problem is amplified tenfold in the Spark ecosystem, which typically involves heavy and mission-critical workloads and a more obscure infrastructure, but lacks modern observability tooling.
definity bridges this gap with the industry’s first data application native solution, providing in-motion and contextualized insights into data pipeline execution, data quality, and data infrastructure performance. Using a unique agent-based architecture, definity runs inline with every data transformation on the platform, establishing ubiquitous observability with zero code-changes—in on-prem, hybrid, or cloud environments.
Also Read: HumanFirst & Infobip Partner to Boost Enterprise AI & Data
Designed specifically for Spark-heavy environments, definity helps data engineers to proactively prevent data incidents, find their root-cause, and fix them—faster than ever before. definity also enables engineers to automatically monitor data applications’ performance, identify concrete optimization and saving opportunities across the platform, and easily optimize performance. This empowers enterprises to minimize downtime, increase engineering velocity, and reduce infrastructure cost.
The company was founded by CEO Roy Daniel, former product executive at FIS; CTO Ohad Raviv, former big-data tech lead at Paypal and Apache Spark contributor; and VP R&D Tom Bar-Yacov, former data engineering manager at Paypal. After experiencing the challenges of managing mission-critical data applications at high-scale firsthand, they built the solution they sought for the enterprise segment.
“Enterprise data engineers demand a new standard of observability that doesn’t exist today” said Roy Daniel, co-founder & CEO, definity. “Traditional data monitoring focuses on the symptoms, assessing data quality at-rest in the data warehouse, which is too out-of-context, reactive, and simply not applicable for Spark. definity fills this void by taking a completely new approach focused on the data application itself, observing in-motion how data is processed and how the infrastructure operates, making Spark applications human-readable.”
“Today’s enterprise data leaders face a serious pressure to ensure the reliability of the data powering the business, while increasing scale, cutting costs, and adopting AI technologies”, said Nate Meir, General Partner, StageOne Ventures. “But without x-ray vision into every data application, data teams are left blind and reactionary. definity is addressing this need head-on with a paradigm shifting solution that is both powerful and seamless for data engineering and data platform teams.”
Source: PRNewswire