Welcome to HackerNoon’s Meet the Writer Interview series, where we learn a bit more about the contributors that have written some of our favorite stories.
Let’s start! Tell us a bit about yourself. For example, name, profession, and personal interests.
Hey! My name is Ashton, and I’m a founding engineer at Theta where I work on RL infra, RL, and distributed systems. I specifically focus on computer-use and tool-use. In my past, I worked at Amazon AGI and tackled inference and tool-use infrastructure. In my free time, I love graphic design, side-projects, and bouldering.
Interesting! What was your latest Hackernoon Top Story about?
My latest story, “Can Your AI Actually Use a Computer? A 2025 Map of Computer‑Use Benchmarks,” touched on one of the hottest spaces in VC right now: RL environments and evals. I gave a comprehensive overview of the most-used computer-use benchmarks, plus practical advice on how to pick benchmarks for training and testing computer-use agents.
I kept running into the same gap: there aren’t many articles that review the benchmarks themselves. And as this field grows, it’s vital that we’re actually assessing quality instead of rewarding whatever happens to game the metric. We’ve been here before. In the early days of LLMs, benchmarks were random and disparate enough that they only weakly reflected the real winner.
Benchmarks became the de facto scoreboard for “best model,” and then people realized a lot of them weren’t measuring what they claimed.
One of the most revealing early-era failures was when “reading comprehension” quietly became “pattern matching on dataset structure.” Researchers ran intentionally provocative baselines (question-only, last-sentence-only), and the results were high enough to raise an uncomfortable possibility: the benchmark didn’t consistently force models to use the full passage. In a 2018 critique, the point wasn’t that reading never matters, but that some datasets accidentally made it optional by over-rewarding shortcuts like recency and stereotyped answer priors.
# Supposed task: answer the question given the passage and question
Passage (summary):
- Sentences 1–8: John’s day at school (mostly irrelevant detail)
- Sentence 9: "After school, John went to the kitchen."
- Sentence 10: "He ate a slice of pizza before starting his homework."
Question: "What did John eat?"
Answer: "pizza"
The benchmark accidentally rewards a shortcut where the model overweights the last sentence (because the answer is often near the end) and simply extracts the direct object of the most recent action (“ate ___”), which in this case yields “pizza.”
And then comes the even more damaging baseline: remove the passage entirely and see what happens. If a question-only model is competitive, it’s a sign the dataset is leaking signal through repetition and priors rather than testing passage-grounded comprehension.
Question: "What did John eat?"
This baseline is basically a sanity check: can the model still score well by leaning on high-frequency answer templates without grounding on the passage at all? In practice it just guesses a token the dataset disproportionately rewards (“pizza,” “sandwich”), and if that works more often than it should, you’re not measuring comprehension so much as you’re measuring the dataset’s priors.
Computer-use evals have already produced an even more literal shortcut: the agent has a browser, the benchmark is public, and the evaluation turns into an open-book exam with an answer key on the final page. In the Holistic Agent Leaderboard (HAL) paper, the authors report observing agents that searched for the benchmark on HuggingFace instead of solving the task, a behavior you only catch if you inspect logs.
# Supposed task: complete a workflow inside the web environment
Task: "Configure setting X in the app and verify it's enabled."
Failure mode:
1) Open a new tab
2) Search for: "benchmark X expected enabled state" / "HAL <benchmark> setting X"
3) Find: repo / leaderboard writeup / dataset card / issue thread
4) Reproduce the expected end state (answer)
At that point, the evaluation was measuring whether it can locate the answer key.
Task: "Find the correct page and extract Y."
Failure mode:
- Search: "<benchmark name> Y"
- Copy from a public artifact (docs, forum post, dataset card)
- Paste the value into the agent output as if it came from interaction
If an agent can pull the value from a dataset card or repo and still “pass,” the success check is grading plausibility, not interaction correctness. Public tasks plus shallow verification turn web search into an exploit.
These two examples are the warning shot: if we don’t hold computer-use benchmarks to higher standards early, we’ll repeat the LLM era just with better UIs and more elaborate ways to cheat.
Do you usually write on similar topics? If not, what do you usually write about?
Yes! Working on the RL environments and RL infra around computer-use, I’m constantly surrounded by the best computer-use models and the most realistic training environments. So I wrote another article, “The Screen Is the API,” which is the case for computer-use and why it’s the future of AI models.
This space is extremely underreported due to two reasons:
- Models aren’t as capable in computer-use as they are in other tasks (coding, math, etc.).
- Computer-use is fast-moving and extremely new.
I want to change that.
Great! What is your usual writing routine like (if you have one)
I usually read a bunch of research papers and speak to my peers in the industry about their thoughts on a topic. Other than that, I spend a lot of time reading articles by great bloggers like PG. So I usually take a lot of inspiration from other people in my writing.
Being a writer in tech can be a challenge. It’s not often our main role, but an addition to another one. What is the biggest challenge you have when it comes to writing?
Finding the time to sit down and put my lived experience into words.
What is the next thing you hope to achieve in your career?
To tackle harder problems with great people, to learn from those people, and share my experiences.
Wow, that’s admirable. Now, something more casual: What is your guilty pleasure of choice?
Watching movies! My favorite movie right now is Catch Me If You Can (2002).
Do you have a non-tech-related hobby? If yes, what is it?
I love bouldering because it makes me feel like I’m a human computer-use agent interacting with the climbing wall. I’m kidding. I think bouldering is a lot of fun because it allows me to take my mind off of work and consolidate my thinking.
What can the Hacker Noon community expect to read from you next?
I’m currently writing another piece on RL environment infrastructure!
What’s your opinion on HackerNoon as a platform for writers?
I think the review structure is awesome, and it was a great place for me to put my thoughts in front of technical readers.
Thanks for taking the time to join our “Meet the writer” series. It was a pleasure. Do you have any closing words?
I love writing. Thank you, HackerNoon!
