Floworks AI Research: Advancing AI Function Calling

Large Language Models (LLMs) have undoubtedly changed our world in the last 3 years. Since ChatGPT was first released in November 2022, it now sees 100 million users every week. In just the last 2 years, massive amounts of funding have been poured into AI startups, and AI news has dominated the airwaves. And yet, in spite of all the hype and anticipation, we haven't seen the economic tidal wave of AI startups that was hoped for. So what gives?

Well, having a chatbot for answering questions is one thing, but it's another thing entirely to build an "AI assistant" that can perform sophisticated tasks, which require interfacing with many different software tools across complex APIs. We believe the technical challenges involved in using LLMs to interact with software - a process known as "function calling" - are what have kept LLM Assistants from becoming widespread.

Function calling has proven to be a challenge that even SOTA foundation models have struggled to meet. For example, GPT-4 obtained a disappointing average task accuracy of 55% on this benchmark. In fact, its inability to perform function calling is the primary reason why OpenAI's GPT-store, which generated a lot of hype in the beginning, has been a bust. It is also partly the reason why recently launched AI assistant-products like RabbitR1 and Humane AI Pin, have not been received well by the market.

Traditional AI systems have treated function calling as a monolith; where the model accepts a task and a bunch of relevant function schemas, and outputs the proper function call filled with all appropriate arguments. This traditional approach suffers from a few key disadvantages:

Traditional function calling architecture showing the flow from User Query through LLM to API Call

• •
  Retrieving the correct functions: Firstly, retrieving the appropriate functions for performing the function call is itself an error-prone task. After all, retrieval is usually performed by computing vector similarity between the task and a function key. Vector similarity is a heuristic approach which is known to suffer from a number of issues with accuracy, scalability, domain specificity, and a general lack of intelligence.
• •
  Huge token lengths: Function schemas are often long, so these models have a huge number of tokens in their prompt. This greatly increases the deployment cost and time consumption of traditional function calling relative to other LLM tasks. Moreover, large prompt lengths in LLMs are also associated with sharp declines in accuracy on reasoning tasks.

  

  Model accuracy significantly degrades as input token length increases across all major LLMs

• •
  High sensitivity of output: LLMs are trained on large bodies of text, most of which is free-flowing. They are trained to think creatively not deterministically. Unfortunately, the nature of function calling is very rigid - the variable names must be exact, the json structures precise, and the arguments must be correct to the last decimal. Two variables with similar but non-identical names will have completely different effects. These characteristics confuse most LLMs that are not trained to think in such inflexible terms.

In fact, the problem of function calling using LLMs has proven to be so challenging that even the best closed-source LLMs (GPT-4o, Claude-3 Opus) can't solve it. So what makes us at Floworks different?

At Floworks, we have been trying to solve the problem of tool use in a valuable domain like CRM for many months. We have finally achieved a respectable level of accuracy (viz. 90% on our benchmark dataset) with our ThorV2 architecture. So what makes us succeed when so many other attempts at solving this problem have been unsuccessful?

When we started out, we tried many of the same approaches that were popular in this space. We gathered a large number of schemas for the Hubspot API, and used Retrieval Augmented Generation (RAG) to fetch the right functions for a particular task. However, what we found was that schemas alone are woefully inadequate to perform function calling. The API documentation of most software is not self-contained or self-explanatory, and current LLMs struggle to interpret them.

These LLMs thus require a framework, or a Cognitive Enhancement Architecture (CEA) to simplify and guide the model along its task. ThorV2 is precisely such a Cognitive Architecture. Think of it as a high-tech power suit like the one worn by Iron Man – providing him better strength, flight capabilities, augmented reality vision etc. But there are two key differences – this power suit is worn by an LLM rather than a human, and it enhances their cognitive capabilities rather than physical attributes. You can also think of a CEA like Augmented Reality glasses for LLMs, increasing their effective intelligence.

Edge of Domain Modeling

The reason Instructions prompting and Example prompting fail to work is that the domain of all the various functions on Hubspot is simply too large. It cannot be effectively captured in terms of examples else the number of tokens will explode. Our initial attempts found that hubspot requires 400,000 tokens to fully capture all the variants of function calling!

So what's our secret sauce? It was to model the domain in a new way, which takes advantage of both the LLMs innate knowledge and enhances it only where required. This is done by giving feedback at the output stage after an initial API call is generated – fixing mistakes rather than giving a surplus of instructions at the start. This is similar to the concept of Agentic Workflows that has gained a lot of attention in AI over the last few months, which has led to agents such as Devin, an autonomous Software Engineer.

In our framework, we have a Naive Assistant which solves the user's query using its innate intelligence plus a short system prompt, which provides broad instructions about its task and a general domain overview of Hubspot. The Assistant generates an initial API call which is frequently wrong on the first try – this is forwarded to a Domain-specific Expert Validator (DEV) which inspects the API call for any errors. If there is no error, the API call is sent to the output. The initial API call can cycle between the Assistant and the Validator any number of times, though in practice it rarely exceeds three attempts.

A notable distinction of our architecture from commonly discussed agentic workflows is that our DEV is a static agent, written entirely in code. We are able to achieve this because we found the errors made by the LLM tend to be highly repetitive and predictable.

The key insight here is that by only fixing mistakes, rather than dumping a massive set of instructions right from the get-go, we avoid information overload on the LLM. This massively reduces the number of tokens boosting accuracy, and reducing cost and latency. We also greatly simplify the job of building the Validator, reducing the turn-around time of these agentic systems from months to weeks. We call it "edge of domain" modeling, as opposed to "whole of domain" modeling, which is the traditional approach to function calling.

Edge of Domain focuses on the intersection of Hubspot-specific function calling and general intelligence

Most queries are resolved in 0-1 validator cycles, with decreasing frequency for multiple corrections

Our ThorV2 model clocking in at 90.1% accuracy is ahead of Claude-3 Opus model (78%), the Gpt-4o model (51%), and the Gpt-4-turbo model (48.6%). Note that these models, released in March, May, and April respectively, are considered SOTA in most domains among current AI models. Our ThorV2 model delivers these results consistently – thus the 10% of errors it encounters are always on the same set of queries.

Our model also manages this feat while maintaining an impressive pace - being 28% faster than gpt-4o, 98% faster than gpt-4-turbo and a whopping 570% faster than Claude-3 Opus! In terms of cost, ThorV2 once again outshines all 3 models, costing only 40% as much as gpt-4o, 25% of gpt-4-turbo and only 1/28th as much as Claude-3 Opus!