Publications

Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics

The 34th IEEE International Symposium on the Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2026), Acceptance rate 23%, Genova, Italy, 2026. (Accepted)

Wonmi Choi, Minuk Park, Zhixiong Niu, Yongqiang Xiong, Chuck Yoo, Gyeongsik Yang

Download PDF

Abstract

LLM-based AI agents process user requests through iterative reasoning and tool execution, often involving the invocation of remote LLM APIs with local tool containers. This execution model can make the optimization of agent serving difficult because latency, local resource demand, and container bottlenecks inter-mix across requests. However, the current agent ecosystem runs without much consideration of resource dynamics, which results in significant waste of the precious resources. This paper analyzes the resource inter-mix of AI agents for three representative tasks: retrieval-augmented question answering, web search, and software coding. To this end, we characterize the latency with respect to the resource dynamics of processing multiple requests and tasks concurrently. Our measurements show that agents have a wide range of behaviors depending on tasks, so that even the same tool can differ substantially in resource dynamics. We also find that running multiple requests concurrently exposes task-dependent bottlenecks in resource dynamics such as CPU, disk I/O, and memory. Furthermore, we uncover that faster LLM responses or more CPU cores do not always accelerate agents. Based on these observations, we demonstrate new optimization opportunities that exploit the resource dynamics of tasks: task-aware CPU allocation. Our results show that the latency of CPU-sensitive agent tasks improves ~5.4×, and the average latency across multiple tasks is reduced ~32% compared to native agents.

Paper summary

Optimize the tools as well as the model

Agent latency depends on local tool execution and remote model response time.

An agent alternates remote model calls with local tool execution. The paper measures how this loop behaves for retrieval-augmented question answering, web search and coding, showing that these tasks place different demands on CPU, memory and disk I/O.

Measure the execution loop before allocating resources

The study covers seven benchmarks on an Azure VM with 24 vCPUs and 216 GB of memory. It records Gemini 3 Flash outputs and per-step latency, then replays those outputs while executing the corresponding tool calls, keeping model-response variability from confounding resource comparisons.

The same agent framework can need different resource policies for different tasks.

Measurements separate model waiting from tool work and vary request concurrency and CPU allocation. Retrieval-heavy tasks show strong CPU sensitivity, web search is dominated by remote-service waiting, and coding combines local commands, I/O and request-specific state.

Retrieval, web access and coding use different combinations of local tools, remote APIs and per-request containers.View full-size figure
Retrieval, web access and coding use different combinations of local tools, remote APIs and per-request containers.

Test two measurement-guided policies

The authors demonstrate CPU-aware tool admission and task-aware CPU allocation. In a mixture of HotpotQA, BFCLv4 and SWE-bench arriving at 0.5 requests per second each, their six-core policy allocates 3, 1 and 2 cores respectively and applies admission control to HotpotQA.

Compared with an equal two-core allocation, average latency falls by 32%. HotpotQA is 5.4× faster, while BFCLv4 latency increases by 28%. A separate admission-only experiment reduces HotpotQA latency by 3.23× on average but increases SWE-bench latency by 2.76×—direct evidence for selecting the policy by task.

CPU allocation in the six-core mixed-workload experiment; task-aware allocation lowers overall average latency by 32%.
WorkloadEqualTask-aware
HotpotQA2 cores3 cores + admission control
BFCLv42 cores1 core
SWE-bench2 cores2 cores

Key takeaways

  1. An agent’s resource needs depend on its tool loop. Remote reasoning and local execution contribute differently across tasks.
  2. One resource policy can help one agent and hurt another. Admission control improves the tested retrieval workload but slows coding.
  3. Task-aware allocation can improve a fixed budget. The mixed-workload result reduces average latency using the same six CPU cores.