Delphi at 100% - dbt semantic layer
Delphi also achieves 100% accuracy on the dbt semantic layer
Jason Ganz wrote a great post on the The Analytics Engineering Roundup - covering a benchmark created by Juan Sequeda of data.world, on the use of LLMs to answer ad hoc data questions.
In short, it was shown that the more context and constraint given to the LLM when interfacing with the data, the better the results.
Text-to-SQL (16.7% accuracy) was inferior to using a knowledge graph with the data (54.2% accuracy), which was inferior to interfacing with a semantic layer (83% accuracy).
We replicated the results the dbt Labs team produced, using the same questions. Delphi was able to achieve 100% accuracy on all questions including the ones which required multi-hop joins - we used the Cube semantic layer for this.
Now we have replicated our results on the dbt semantic layer! As before we have achieved 100% accuracy, but this time excluding the questions which required multi-hop joins, as the dbt semantic layer does not support this yet.
Our repo which shows how we built the semantic layers to replicate this is here.
Here are the details of the results, including generated queries.
If you’d like a demo or to speak to us, sign up at delphihq.com