ArchAgent v2: A Case Study with the Data Prefetching Championship
Introduction
The Challenge of Hardware Prefetching
Every time your processor needs data that isn't in its cache, it stalls. The memory hierarchy—L1, L2, L3 caches, then DRAM—exists precisely to hide this latency, but no cache is perfect. When the CPU misses, it waits hundreds of cycles for data to arrive from main memory. That wait is wasted computation.
Hardware prefetching is the art of predicting what data the processor will need before it asks for it, fetching it into cache ahead of time. Done well, prefetching can hide most of that memory latency. Done poorly, it pollutes the cache with useless data and actually slows things down.
For decades, prefetcher design has been a craft. Experts study memory access patterns, hypothesize about program behavior, write heuristics, and then spend months tuning them against benchmarks. It's slow, expensive work, and the design space is enormous.
Introducing ArchAgent v2: AI-Driven Design
ArchAgent v2 is a framework that treats prefetcher design not as a craft, but as a search problem. It uses large language models (LLMs) to propose candidate prefetching policies, evaluates those policies in a cycle-accurate simulator, and then uses reinforcement learning to iteratively refine its proposals. The goal: automate the entire design loop that a human expert would perform manually.
This isn't a toy demonstration. ArchAgent v2 was entered into the Data Prefetching Championship (DPC), the premier competition for prefetching algorithms, and performed competitively against dozens of human-designed systems.
The Data Prefetching Championship as a Testbed
The DPC is the perfect proving ground for AI-driven hardware design. It provides a standardized simulator (ChampSim), a fixed set of benchmark traces, and a clear performance metric. Every submission runs under identical conditions. If your prefetcher is better, the numbers show it.
This article walks through how ArchAgent v2 works, what happened when it competed in DPC-3, and what the results tell us about the future of AI in computer architecture.
What This Article Covers
We'll start with the fundamentals of prefetching, then explain the DPC competition structure, dive deep into ArchAgent v2's architecture and workflow, present the case study results, and finally discuss the broader implications—and limitations—of using AI agents for hardware design.
Understanding Data Prefetching
Why Prefetching Matters: Hiding Memory Latency
Modern processors can execute multiple instructions per cycle, but they can't execute instructions that are waiting on data. When a load instruction misses all caches and has to go to DRAM, the penalty is typically 200-300 cycles. During that time, the processor may stall completely.
Prefetching attacks this problem by predicting future accesses and issuing memory requests early. If the prediction is correct, the data arrives in cache before it's actually needed, and the processor never stalls.
The math is compelling. A prefetcher that achieves a 20% reduction in average memory latency can improve overall performance by 10-15% on memory-intensive workloads. That's a bigger win than most other microarchitectural optimizations.
Traditional Approaches: Stride, Markov, and Other Heuristics
The classic prefetchers are based on observable patterns:
- Stride prefetchers detect regular patterns: if a program accesses addresses 100, 104, 108, 112, it predicts 116 next. These work well for array traversals and loops.
- Markov prefetchers track transitions between addresses: if address A is followed by address B, then seeing A again suggests B is coming. These handle irregular patterns like pointer-chasing.
- Signature-based prefetchers (like the Samsung "Signature Path Prefetcher" that won DPC-1) group addresses into "signatures" and learn common paths through them.
Each of these has strengths and weaknesses. Stride prefetchers are simple and cheap but useless on irregular code. Markov prefetchers are flexible but generate lots of useless prefetches. Real-world prefetchers are almost always hybrids, carefully tuned to specific workload characteristics.
The Limitations of Hand-Crafted Prefetchers
Designing a good prefetcher is hard for several reasons:
- The design space is vast. Prefetch distance, degree, confidence thresholds, table sizes, replacement policies—each parameter interacts with the others in non-obvious ways.
- Workloads are diverse. A prefetcher that excels on scientific computing might fail on database workloads. The SPEC CPU benchmarks alone contain wildly different access patterns.
- Evaluation is expensive. A single simulation run of one benchmark takes hours. Testing one design across 40 benchmarks takes days.
- Local optima are everywhere. Small tweaks to one parameter can produce surprising results, and human intuition doesn't always map to simulator behavior.
The result is that state-of-the-art prefetchers take months or years of expert effort to develop. The DPC was created to formalize this process and give researchers a common benchmark.
Key Metrics: IPC, Hit Rate, and Speedup
Three metrics dominate prefetcher evaluation:
- IPC (Instructions Per Cycle): The fundamental performance measure. Higher is better.
- Hit rate: The fraction of memory requests satisfied by the cache. A prefetcher can achieve high hit rates but still hurt performance if it pollutes the cache or consumes bandwidth.
- Speedup: The ratio of IPC with prefetching to IPC without. This is the metric that matters most.
The critical insight: hit rate is not the goal. A prefetcher that fetches aggressively will get high hit rates but may evict useful data or saturate memory bandwidth. The DPC judges prefetchers on geometric mean speedup, which directly measures end-to-end performance improvement.
Key Takeaway: Prefetching is about hiding memory latency, not maximizing hit rate. A good prefetcher improves IPC by fetching the right data at the right time, not by fetching everything.
The Data Prefetching Championship (DPC)
History and Evolution of DPC-1 to DPC-3
The Data Prefetching Championship began in 2019 as a way to standardize prefetcher evaluation. DPC-1 (2019) established the format: a common simulator, a common benchmark set, and a competitive leaderboard. DPC-2 followed in 2021 with more benchmarks and stricter rules. DPC-3 (2023) expanded further, adding cloud workloads and attracting over 100 submissions from academia and industry.
The competition has evolved alongside the field. Early editions were dominated by hand-crafted heuristics from veteran researchers. By DPC-3, a significant portion of submissions used machine learning in some form—a sign of the field's direction.
ChampSim: The Standard Simulator
Every DPC submission runs in ChampSim, a trace-based microarchitectural simulator. ChampSim models a 4-core out-of-order processor with a 256KB L2 cache and a 2MB L3 cache. It's fast enough to run dozens of benchmarks in reasonable time, but detailed enough to capture the interactions between prefetcher, cache hierarchy, and memory controller.
ChampSim is open source and widely used outside the DPC, making it a de facto standard for prefetching research.
Benchmark Suite: SPEC CPU 2017 and Cloud Workloads
DPC-3 uses 40 traces: a mix of SPEC CPU 2017 benchmarks (covering integer and floating-point workloads) and cloud traces from Google and Alibaba data centers. These traces were collected from real systems and represent diverse access patterns—from regular array traversals to chaotic hash table lookups.
The cloud traces are particularly important because they represent modern data-center workloads with irregular, multi-threaded access patterns that differ significantly from traditional HPC benchmarks.
How Winners Are Determined: Geometric Mean Speedup
Each prefetcher is run on all 40 traces, and the speedup over a no-prefetcher baseline is computed for each trace. The final score is the geometric mean of these speedups. Geometric mean is used instead of arithmetic mean because it treats all benchmarks equally—a prefetcher that does well on 39 benchmarks and terribly on one won't be unfairly penalized, nor will one that dominates a single benchmark.
The best human-designed prefetcher in DPC-3, called Berti, achieved a geometric mean speedup of approximately 1.12x. ArchAgent v2 achieved approximately 1.08x.
Key Takeaway: The DPC is a controlled experiment. Every prefetcher faces identical conditions, and the geometric mean speedup provides a fair, reproducible ranking.
ArchAgent v2: An Overview
Core Idea: Treating Prefetcher Design as a Search Problem
Human prefetcher designers explore the design space by intuition and trial. ArchAgent v2 formalizes this as a search: generate a candidate design, evaluate it, learn from the results, and generate a better candidate. The framework doesn't try to be clever about how to prefetch—it lets the search process discover good policies.
Role of Large Language Models (LLMs) in Proposing Policies
The LLM component of ArchAgent v2 generates candidate prefetcher implementations. Given a description of the problem (the benchmark characteristics, the current best performance, and a library of known prefetching techniques), the LLM proposes a new policy, often combining existing techniques in novel ways.
This is a critical design choice. The LLM isn't being asked to invent prefetching from scratch—it's being given a toolbox of known techniques and asked to assemble them into a working policy. This is analogous to how a human expert would work, but the LLM can explore combinations much faster.
Reinforcement Learning: The Iterative Refinement Loop
The LLM proposes, but it doesn't learn. The reinforcement learning component does. After each candidate is simulated, the framework computes a reward based on performance (geometric mean speedup across benchmarks). This reward is fed back into the system, which uses it to guide the next round of proposals.
The key insight is that the RL loop operates on the performance signal, not on the prefetcher's internal decisions. The framework doesn't try to explain why a prefetcher works—it just knows that certain proposals led to good results and biases future proposals accordingly.
Modular Architecture: Separating Agent Logic from Simulator
ArchAgent v2 is designed to be portable. The agent logic (LLM + RL loop) is completely separate from the simulator interface. To evaluate a candidate, the framework compiles the proposed prefetcher into ChampSim, runs the traces, and extracts performance metrics. This separation means ArchAgent v2 could be adapted to other hardware design problems—cache replacement, branch prediction, memory scheduling—by swapping the simulator and reward function.
How ArchAgent v2 Works: Step-by-Step
Step 1: Generating Candidate Prefetcher Policies with LLMs
The process begins with a prompt to the LLM. The prompt includes: - A description of the benchmark suite (e.g., "40 traces from SPEC CPU 2017 and cloud workloads") - The current best performance achieved so far - A library of known prefetching techniques (stride, Markov, signature-based, etc.) - Constraints (e.g., hardware budget, table sizes)
The LLM generates a complete prefetcher implementation in C++ that plugs into ChampSim's interface.
Step 2: Simulating Candidates in ChampSim
Each candidate is compiled and run against the full benchmark suite. This is the expensive step—each trace takes about an hour of simulation time, and there are 40 traces. ArchAgent v2 typically runs candidates in parallel across multiple machines.
Step 3: Evaluating Performance with a Reward Function
The reward function is the geometric mean speedup across all traces. This is the same metric used by the DPC, ensuring that the framework is optimizing for the competition's actual goal. The reward is a single number, which makes the RL problem tractable.
Step 4: Refining Policies via Reinforcement Learning
The reward feeds into the RL loop, which updates a policy model that influences the next LLM prompt. If a candidate performed well, the framework might prompt the LLM to explore similar designs. If it performed poorly, the framework steers away from that direction.
Step 5: Convergence and Final Policy Selection
The process repeats for hundreds of iterations. ArchAgent v2 required approximately 500 simulation runs to converge to a competitive policy. The final policy is the candidate with the highest geometric mean speedup, selected from all evaluated designs.
Key Takeaway: ArchAgent v2 is not a single algorithm—it's a design process that combines LLM creativity with RL-guided search. The LLM proposes, the simulator evaluates, and the RL loop learns.
Case Study: ArchAgent v2 in DPC-3
Setup and Configuration: Traces, Simulator, and Resources
ArchAgent v2 was entered into DPC-3 under standard competition rules: ChampSim simulator, 40 benchmark traces, and the geometric mean speedup metric. The framework ran on a cluster of machines, parallelizing simulation across traces. Each candidate required approximately 40 hours of simulation time (40 traces × 1 hour each), but parallelization reduced wall-clock time to a few hours per candidate.
Performance Results: Speedup Comparison with Human Designs
ArchAgent v2 achieved a geometric mean speedup of approximately 1.08x over the baseline. The best human-designed prefetcher, Berti, achieved 1.12x. This places ArchAgent v2 in the top tier of DPC-3 submissions—competitive with, though not exceeding, the best human effort.
The gap between 1.08x and 1.12x is meaningful but not enormous. ArchAgent v2 was competitive with the vast majority of human designs, many of which were the result of years of expertise.
Examples of Novel Policies Generated by ArchAgent v2
The framework didn't just replicate existing designs—it generated genuinely novel combinations:
- Example 1: A hybrid prefetcher combining a stride predictor (for regular loops) with a Markov predictor (for irregular pointer-chasing). While hybrid prefetchers exist, the specific combination and parameter tuning discovered by ArchAgent v2 were not previously explored.
- Example 2: On the SPEC CPU 2017 'mcf' benchmark (notorious for irregular memory access), ArchAgent v2's policy achieved a 1.08x speedup, demonstrating that the framework can handle pathological cases.
- Example 3: The LLM proposed a "prefetch distance adaptation" mechanism that dynamically adjusts how far ahead to prefetch based on cache pressure. This improved performance on cloud workloads, where cache contention is high.
Time Efficiency: Days vs. Months of Human Effort
The most striking result is time. ArchAgent v2 converged to its final policy in days. Human experts typically take months to develop and tune a competitive prefetcher. The framework didn't match the best human effort, but it came close in a fraction of the time—and it can be run again with different constraints or objectives.
Key Takeaway: ArchAgent v2 demonstrates that AI-driven design can produce competitive hardware optimizations without human expertise. It didn't win, but it proved the concept.
Analysis: Why ArchAgent v2 Succeeded (and Where It Fell Short)
Strengths: Automation, Exploration, and Combination of Techniques
ArchAgent v2's primary strength is automation. It explored hundreds of design combinations that a human would never have time to try. The LLM's ability to combine known techniques in novel ways proved valuable—many of the best candidates were hybrids of existing prefetchers with new parameter settings.
The framework also benefited from its reward function. By optimizing geometric mean speedup directly, it avoided the trap of maximizing hit rate at the expense of performance.
Weaknesses: Computational Cost and Dependence on Simulator Accuracy
The biggest limitation is computational cost. 500 simulation runs, each taking about an hour, requires substantial computing resources. Most academic labs couldn't afford this without dedicated clusters.
The framework also inherits ChampSim's limitations. If the simulator doesn't accurately model real hardware behavior, ArchAgent v2's optimizations may not translate to real systems. This is a general problem in computer architecture research, but it's particularly acute for AI-driven design, which has no intuition to compensate for simulator inaccuracies.
Comparison with the Best Human-Designed Prefetcher (Berti)
Berti, the DPC-3 winner, achieved 1.12x speedup. ArchAgent v2 achieved 1.08x. The gap likely comes down to Berti's sophisticated handling of memory bandwidth constraints—a subtle problem that ArchAgent v2 didn't fully explore. Human experts have deep intuitions about hardware constraints that are hard to encode in prompts or rewards.
Lessons Learned for AI-Driven Hardware Design
The DPC-3 experiment suggests several lessons:
- AI can generate competitive designs, but not superior ones—yet. ArchAgent v2 was close to the best human effort, but didn't beat it.
- The reward function is everything. Optimizing the right metric is more important than the search algorithm.
- Hybridization is a strength. The LLM's ability to combine existing techniques was more valuable than its ability to invent new ones.
- Computational cost is the bottleneck. Making this approach practical requires faster simulation or smarter sampling.
Implications for the Future of Computer Architecture
Beyond Prefetching: Cache Replacement, Branch Prediction, and More
The ArchAgent v2 framework is general. The same LLM + RL loop could be applied to cache replacement policies, branch prediction, memory scheduling, or even instruction scheduling. Each problem has a simulator, a benchmark set, and a performance metric—the framework's requirements are met.
The Rise of ML-Based Solutions in DPC-3
ArchAgent v2 wasn't alone. DPC-3 saw a significant increase in ML-based prefetchers, with many teams using neural networks or reinforcement learning in some capacity. The field is clearly moving toward ML-driven design.
Challenges and Opportunities for AI Agents in Hardware Design
The main challenges are computational cost, simulator accuracy, and the difficulty of encoding hardware constraints (area, power, timing) into the reward function. The opportunities are equally clear: AI agents can explore design spaces far faster than humans, and they don't have preconceptions about what "should" work.
Open-Source Availability and Reproducibility
ArchAgent v2 is open source, and the DPC provides a reproducible evaluation framework. This is crucial for the field—anyone can run the framework, verify the results, and build on them.
Key Takeaway: ArchAgent v2 is a proof of concept, not a final product. The framework's architecture is general, and its methodology can be applied to other hardware design problems.
Common Misconceptions About AI and Prefetching
AI Does Not Always Outperform Human Experts
ArchAgent v2 lost to Berti. AI is a tool, not a magic solution. The best results come from combining AI's exploration capability with human expertise in framing problems and interpreting results.
Prefetching Is Not Just About Hit Rate
A prefetcher with 95% hit rate can be worse than one with 80% if it pollutes the cache or wastes bandwidth. ArchAgent v2 optimized for speedup, not hit rate, and this was the right choice.
Hardware vs. Software Prefetching
This article discusses hardware prefetching, where the CPU predicts and fetches automatically. Software prefetching (explicit instructions in code) is a different technique with different tradeoffs. Both are useful, but they solve different problems.
LLMs Do Not 'Understand' Architecture
The LLM in ArchAgent v2 doesn't understand computer architecture. It generates code based on patterns in its training data and the prompt it receives. The RL loop is what makes the framework work—it filters out bad proposals and amplifies good ones.
Conclusion
Recap: ArchAgent v2's Journey in DPC-3
ArchAgent v2 took a fundamentally different approach to prefetcher design: instead of a human expert iterating manually, it used LLMs to propose designs and reinforcement learning to refine them. The result was a competitive prefetcher that achieved 1.08x speedup—not the best in DPC-3, but close, and achieved in days rather than months.
The Promise of AI-Driven Design in Hardware
The implications extend far beyond prefetching. If AI agents can design competitive prefetchers, they can probably design competitive cache replacement policies, branch predictors, and memory schedulers. The hardware design process is ripe for automation.
Final Thoughts: A New Era for Microarchitectural Optimization
ArchAgent v2 doesn't replace human experts. It augments them. The framework explores design spaces that humans don't have time to explore, and it does so in a reproducible, systematic way. As simulation tools get faster and LLMs get better, this approach will only become more powerful.
The next Data Prefetching Championship will be interesting. If AI agents continue to improve, they might not just be competitive—they might win.
Frequently Asked Questions
What is ArchAgent v2?
ArchAgent v2 is a framework that uses large language models and reinforcement learning to automatically design hardware prefetchers. It generates candidate prefetcher policies, simulates them, evaluates their performance, and iteratively refines them.
What is the Data Prefetching Championship?
The DPC is a competition series that evaluates prefetching algorithms under standardized conditions. Participants submit prefetchers that are run on a common simulator (ChampSim) with a fixed set of benchmark traces, and ranked by geometric mean speedup.
Why is prefetching important?
Prefetching hides memory latency by fetching data before it's needed. It can improve performance by 10-15% on memory-intensive workloads, making it one of the most impactful microarchitectural optimizations.
How does ArchAgent v2 differ from traditional prefetcher design?
Traditional prefetcher design relies on human experts who manually create heuristics and tune parameters. ArchAgent v2 automates this process: an LLM proposes designs, a simulator evaluates them, and reinforcement learning guides the search.
Did ArchAgent v2 win the DPC?
No. ArchAgent v2 achieved a geometric mean speedup of approximately 1.08x, placing it competitively but behind the winner, Berti, which achieved 1.12x. It was one of the top-performing submissions, but not the winner.
What simulator does the DPC use?
The DPC uses ChampSim, a trace-based microarchitectural simulator that models a 4-core out-of-order processor with L1, L2, and L3 caches.
Can ArchAgent v2 be used for other hardware design problems?
Yes. The framework's architecture is modular—the agent logic is separate from the simulator interface. It could be adapted for cache replacement, branch prediction, memory scheduling, or other optimization problems.
What are the limitations of ArchAgent v2?
The main limitations are computational cost (hundreds of simulation runs) and dependence on simulator accuracy. It also didn't outperform the best human-designed prefetcher, suggesting that human expertise still has value.
What is a 'trace' in this context?
A trace is a recording of memory access addresses from a real program execution. The DPC provides traces from SPEC CPU 2017 benchmarks and cloud workloads, which are used as inputs to the simulator.
Is ArchAgent v2 open source?
Yes. The framework is publicly available, and the DPC provides a reproducible evaluation environment. Anyone can download the code, run the experiments, and verify the results.
Explore the ArchAgent v2 paper and code to see how AI is reshaping hardware design, and consider participating in the next Data Prefetching Championship to test your own ideas!