Prophecy, the data copilot company, announced Prophecy Data Transformation Copilot for Databricks, the industry’s first copilot that accelerates the preparation of raw data for analytics and AI applications. By leveraging generative AI, Prophecy Data Transformation Copilot expedites the development, deployment, and monitoring of enterprise-grade data pipelines native to the Databricks Data Intelligence Platform, ensuring the delivery of clean, trusted, and timely data for analytics.
“Generative AI is set to fundamentally change how we work with data, as technologies that leverage intelligence about data and language models provide novel approaches to transform, organize and get insights from data,” said Stewart Bond, Vice President of IDC’s Data Intelligence and Integration Software Service. “Copilots improve the accessibility and availability of data for technical and non-technical users throughout the enterprise, democratizing data and analytics.”
Cloud data platforms have become increasingly powerful to support a wide array of use cases, including streaming data pipelines, batch pipelines, and ad hoc SQL analyses. These platforms effectively handle diverse data types, from database tables to APIs to documents, and apply transformations such as cleaning, combining, aggregating, and structuring to prepare the data for utilization in AI and business intelligence.
Enabling a large number of data users to be productive has proven challenging for organizations, because traditional visual ETL products fail to seamlessly integrate with code-based cloud data platforms. Consequently, many enterprises resort to developing code directly for these cloud data platforms which, while powerful, remains inaccessible to most data users. Copilots address this issue by democratizing access to cloud data platforms, thereby increasing productivity across the board.
Also Read: New Square 9 Integration Brings Information Together for a More Informed, Connected Organization
“The industry’s current approach to data transformation is broken. Simplistic tools cater to many users, but lack power. Cloud data platforms are powerful, but remain accessible only to expert data engineers. Both of these approaches fail to adequately address the problem,” said Raj Bains, CEO, and Founder of Prophecy. “It’s a false dichotomy, as copilots make the full capabilities of cloud data platforms accessible to all users; however, to truly evolve, they must move beyond conversational-only interfaces.”
Prophecy Data Transformation Copilot uniquely integrates visual interface, generative AI, and compiler technology to provide the most efficient approach to data transformation. Prophecy enables organizations to accelerate every stage of the data pipeline lifecycle in the following ways:
- Develop: By suggesting next transforms, completing expressions, and generating equivalent data pipelines based on user input.
- Test & Deploy: By automating tasks, from suggesting tests to reduce errors, to generating documentation and commit messages to streamline deployment.
- Observe: By monitoring deployed pipelines, and suggesting fixes for errors, the platform enables any data user to fix and redeploy pipelines for improved uptime.
“As a growing Databricks partner, Prophecy continues to help our largest enterprise customers accelerate productivity and generate substantial value from their data transformation efforts,” said Adam Conway, SVP Products, Databricks. “We are excited for the Data Transformation Copilot for Databricks and look forward to continuing to enable data engineers and business professionals to become more productive with data.”
Prophecy integrates deeply with Databricks, supporting the full lifecycle of Apache Spark- or SQL-based data pipelines. Pipelines developed with Prophecy’s AI-powered visual interface turn into Spark code, or into SQL code with dbt core, making the full power of code available to all data users. Integrations with Databricks Unity Catalog and Delta Lake ensure a unified and well governed experience. Additionally, Databricks workflows can be developed visually to orchestrate and schedule multiple pipelines. Finally, Prophecy supports preparing unstructured data for AI, integrating seamlessly with features and models developed in Databricks.
Source: PRNewswire
