Vectara, the trusted Generative AI product platform, announced the inclusion of a Factual Consistency Score (FCS) for all generative responses based on an evolved version of the groundbreaking Hughes Hallucination Evaluation Model (HHEM)—the #1 hallucination detection model on Hugging Face with 100,000+ downloads since its launch last November. The associated Hallucination Leaderboard is now the industry standard for how LLMs benchmark their average factual consistency. Vectara’s end-to-end Retrieval Augmented Generation-as-a-service (RAGaaS) platform is setting a new standard as an industry-first feature for GenAI response transparency by providing real-time end-to-end RAG observability. This innovative metric provides unprecedented visibility into the factual consistency of summarized responses within Vectara’s RAGaaS platform, empowering users to set personalized thresholds for response acceptance based on a detailed accuracy score.
With average hallucination rates of LLMs on the market ranging from 3% to 16.2%, the risk of unknown inaccuracies in their response remains a major concern, preventing widespread business adoption of this powerful technology. Vectara mitigates this ambiguity for enterprises by providing a Factual Consistency Score grading the likelihood that the generated response is a hallucination or not. Only with a standardized, scientifically calculated method for grading responses can businesses responsibly introduce GenAI into business critical applications. Users have the ability to set thresholds for response acceptance based on a detailed accuracy score, giving product teams the flexibility to act on this information according to their preferences.
Vectara’s Factual Consistency Score is a groundbreaking tool in GenAI, setting a new benchmark for real-time hallucination detection and offering superior performance, affordability, and speed, thus marking a significant leap forward in trust. Its efficiency and effectiveness enable businesses to deploy GenAI into critical product use cases without being worried about exposure to liabilities that might arise from hallucinated responses.
Vectara’s Factual Consistency Score equips developers with the capability to refine and enhance a wide range of applications, from internal Q&A systems to the quality of interactions with end consumers. The strength of this score lies in its calibration, making it interpretable as a direct probability—for instance, a score of 0.98 indicates a 98% probability of factual consistency. This contrasts sharply with many contemporary ML classifiers that disregard calibration, thus sacrificing clarity and direct interpretability.
“Integrating Vectara’s Factual Consistency Score into the Yobi app will revolutionize how we handle AI transparency and accuracy for business use cases. By providing visibility and accountability into answers provided by our platform, we can stay true to our commitment to responsible AI that enterprises can depend on,” said Ahmed Reza, Founder and CEO of the Yobi app. “As a Co-Innovate Partner with Vectara, we’re thrilled to see such advanced technology directly incorporated into the Vectara platform.”
The advanced HHEM that powers the Factual Consistency Score gives greater visibility than previously released open-sourced versions, offering enhanced accuracy and extended language support. This initiative is part of Vectara’s commitment to transparency and control, empowering businesses with the autonomy to manage AI responses effectively.
“Just as we were early in pioneering RAG to enhance the relevance and quality of generated content, we are once again at the forefront of responsible AI by being completely open about our efforts to mitigate hallucinations in Generative AI,” said Amr Awadallah, co-founder and CEO of Vectara. “By providing our customers with real-time access to factual consistency scores, we’re not just engineering trust; we’re handing over the control, enabling them to make informed decisions on how to utilize the responses generated by our RAGaaS platform.”
SOURCE: BusinessWire
