Most AI projects fail to reach production due to challenges with accuracy, data quality, and robustness in real-world scenarios.
Transformer models understand the full context of a sentence, enabling semantic search by converting text into vectors for comparison.
Transformer-only models often struggle in production due to inefficiency, reliance on domain-specific training data, and a lack of robustness.
Classical NLP methods like BM25 keyword search are computationally efficient, require no training data, and are highly robust across different domains.
Hybrid AI combines the high accuracy of modern NLP with the efficiency and robustness of classical methods to create superior production models.
Vespa is an open-source tool that simplifies building hybrid systems by allowing you to define parallel search pipelines for Transformers and BM25.
The hybrid AI approach was four times faster than a pure Transformer model while maintaining high accuracy and robustness.
Hybrid AI can be used for expert identification by building correctable knowledge graphs and for safety-critical systems like train controls.
A summary of how hybrid AI balances deep learning's accuracy with rule-based systems' robustness, plus recommended libraries to get started.
msg
Ismaning, Germany
35:16Ankit Patel
29:26Alexander Birke & Silke Eggert
24:07Mike Butcher, Jürgen Müller, Katrin Lehmann & Tobias Regenfuss
59:39Julian Joseph
57:52Rainer Stropek
25:17Scott Hanselman
26:05Christian Weyer
32:26Krzysztof Cieślak
Jobs that call for the skills explored in this talk.
Collaboration Betters The World GmbH
NLP PEOPLE
Cornelius, United States of America
BAE Systems
San Diego, United States of America
Smartly
Berlin, Germany
Eiden Systems Corporation
Chantilly, United States of America
