is there a good text2SQL benchmark? I've tried many models/services and all of them come up short (<90% reliability) and there are many more out there left to try.

Hey guys. I have been trying to build a text2sql agent. As of now its a PoC for few tables but it will expand to a larger schema in the future. Been trying out various approaches but wanted to know if there are any suggested approaches and any advice on building a production grade system.

The approaches i tried:

1. Built a general text2sql flow in crew ai. Used 2 agents. One was for extracting relevant schema and entities. The second agent was for the actual SQL with the output of the first agent. It also works in a loop and can keep retrying until the query is proper.

2. A similar approach to above, but built out example questions but parameterized sql ,that cover major user queries. Performed a keyword + vector search on user queries and sent it out to the LLM to then construct the SQL. But worried about the issue with larger schemas.

3. A almost reverse process of the first approach which i wanted to try out. There is the logic of building out the sql written down as code and the LLM’s job is just to supply the parameters. It doesnt generate the sql. Idea behind it was to minimise what the LLM can do to prevent wrong queries.

Another problem i want to tackle is wrong output rather than a failed sql. Approach 2 and 3 seem better suited for this but approach 3 feels very rigid in terms of the queries it can make, which I also feel helps in preventing wrong queries.

I wanted to take your opinions on the same to see if i am missing some steps or are there better ways to approach this since I am quite new to this.