Following are lightly edited questions and answers from Patricia’s presentation on Orchestrating Optimization with Precision Agents at the INFORMS Analytics+ conference and in a subsequent webinar (watch the recording here and see Patricia’s previous blog post for reference.)
Question: What is the difference between an agent and a tool?
Patricia: The agent is the decision maker, acting like a user. It’s a proxy for the user, making decisions, gathering data, acting on that data. Tools are the analytical capabilities that the agent is using. A tool might be an optimization model, a simulation, a machine learning model, a data product. Many tools are just access to data. Tools themselves don’t make decisions. They provide input into the decisions, and it is the agent who is in charge of making the decision.
Question: When you and I are making the decision, we know what’s good and what’s not. There is a cognitive aspect of that human feeling that role brings. When does an agent know which tool to go to, and when does an agent recognize that “you don’t know what you don’t know” and then determine, “No”?
Patricia: The agent must know which tool to use, and the orchestrator has to know which agent to call. The key is scoping. You give an agent only what it needs, not your entire toolbox—often through an interface such as an MCP that tells the agent what each tool does and how to use it. In my experience, LLMs can get tied up looking at everything available in the universe, which leads to confusion about which tool to use. In the railroad train-building case, the optimization agent had just two tools: one to obtain track inventory and one to optimize. There was another set of safety tools, but that agent wasn’t scoped to call them. The project scope and business rules define what each agent is responsible for.
Question: How do you run tests of your precision agents, and how do you make sure people are comfortable relying on these AI systems?
Patricia: We think of agents as augmenting people rather than replacing them, the same way we treat optimization models. At the end of the day, agents are tools themselves, to help humans do their jobs, just as optimization models have been for many years.
For training and post-training monitoring, we have developed “Agent Dojo,” a framework for training, evaluating, and monitoring agents and models in general. Once agents have been running for a while, a common concern is drift—agents gradually behaving differently than intended. We run the agents in a shadow mode to see how they respond and to watch the questions and answers. The Agent Dojo is essentially a test harness on steroids that keeps an eye on the agents, gathering metrics on the quality of their activities and answers, and flagging when retraining is required.
Question: As LLMs leapfrog each other what seems like every month, and as other related tools emerge, how do you as a practitioner view the different technologies and vendors when you’re building these kinds of applications and tools?
Patricia: We advise holding technology lightly in this space. We try to be agnostic. We don’t want to get chained to one of the leading models and then find out it’s no longer the latest and greatest. I will say that they are all getting so good. Small language models keep improving. We recommend modular thinking in your design. Be flexible and willing to swap out things when needed.
To discuss this topic with Patricia, contact us to set up a call.
