Welcome to dystopia: I helped ChatGPT pass a CAPTCHA and doomscroll my Facebook

Last week, OpenAI released what may be the most ambitious (and potentially controversial) update to ChatGPT since its launch: Agent mode. Unlike the standard conversational interface, Agent mode gives ChatGPT control over a virtual machine running Chrome, allowing it to interact with websites like a human would. It can identify elements on websites, scroll, click buttons, fill out forms, and, if granted credentials, even log into your online accounts.

For the first time since the AI’s launch, it can perform tasks instead of spitting out some text on how to do it yourself. On the surface, the potential seems endless. The chatbot could reply to your emails, shop for groceries, book a flight, or perform even more complex tasks spanning multiple websites. The best part is that you can watch ChatGPT “move” its mouse cursor around the virtual web browser and navigate the internet (as you can see in the video below). Admittedly, it’s a lot like a toddler struggling to walk at times, but it’s endlessly fascinating nonetheless.

So what can ChatGPT’s Agent mode actually do with all of these capabilities? To answer that question, I tested the feature with a couple of real-world tasks — the kind you might actually want to offload to an AI assistant. Here’s how it handled them, and what ChatGPT did when it encountered an obstacle.

Amazon’s Alexa can add toilet paper to your cart with a voice command, but ChatGPT’s Agent mode can be entrusted to do a whole lot more. Specifically, it can shop your entire grocery list on any platform of your choice. Case in point: I gave the agent a simple task: buy everything I would need for a homemade pizza from Walmart. I didn’t offer any specific ingredients, items, or even guidance on price just to see what it would pick.

The agent booted up a virtual computer and navigated to Walmart in no time. But it ran into a roadblock almost immediately — Walmart threw up an anti-bot verification screen requiring a human to press and hold a button. Shockingly, the agent recognized this screen and asked me to briefly take control of the browser and complete the task. I took control and about ten seconds later, we were in. I handed control back, and the agent immediately got to work. It looks like CAPTCHAs will need to evolve yet again if they are to keep bots out in the future.

Moving on, I watched the agent methodically search for “pizza dough,” “pizza sauce,” “mozzarella cheese,” and “pepperoni.” But to my surprise, the agent didn’t just grab the first result. Instead, it prioritized familiar and well-priced alternatives just like I personally would. In more than one instance, I watched it pick the third or fourth item in the results or call a competing product overpriced. The agent also correctly moved past inaccurate search results like a fully premade frozen pepperoni pizza when it was merely shopping for pepperoni, the ingredient.

Within four minutes, my virtual cart was filled with everything I needed to make a pizza. The agent navigated to the checkout page and then handed control back to me to complete another CAPTCHA, login, and enter my payment details securely. ChatGPT says it cannot see your inputs when you’re in control of its virtual machine, presumably meaning it can’t store your login or credit card info. Despite that, I didn’t elect to enter my login details and therefore, spent the night without any pizza.

Needless to say, this was a very impressive showing even if it was a rather straightforward task with a clear sequence of actions. But can the agent handle something more ambiguous? I decided to give it a harder challenge: find a list of used car candidates on Facebook Marketplace.

I love Facebook Marketplace because there’s always a great deal around the corner, but I dread scrolling through the hundreds of listings to find the right one. So I put ChatGPT’s Agent mode to the task, fully expecting it to stumble at some point. Specifically, I asked it to log into my Facebook account, navigate to the Marketplace tab, and compile a list of used Honda Fit specimens. As if that wasn’t difficult enough, I asked it to only look for 2015 to 2020 models within a 50-kilometer radius of Toronto.

Much to my surprise, the agent started off strong and logged into my Facebook account without a hiccup. I did have to step in to provide a two-factor authentication code, but other than that, it was an entirely painless process. ChatGPT navigated to the Marketplace section, changed the location to the city I provided, and even adjusted the maximum distance filter. It then started going through individual listings and recorded important details about each car it came across.

To be fair, ChatGPT’s Agent mode is not faster than a human — I could probably go through individual listings faster than the agent, at least in its current state. But like any computer, ChatGPT is stubbornly persistent. After watching it look through listings for a few minutes, I walked away from my computer to grab a coffee. When I returned about ten minutes later, the agent was still meticulously clicking on individual listings.

The final result was a comprehensive report of two to three dozen cars that met my criteria. As the image above shows, the agent had compiled a detailed brief for each vehicle it found. It organized the key details into columns: year and model, price and mileage, transmission, and location. It even created a notes section where it summarized the seller’s description, pulling out crucial information like “dealer listing,” “salvage title,” or “open to negotiation.” Finally, each line item had a screenshot of the Marketplace listing for me to peruse if I was interested in that specific vehicle.

If that seems impressive, it absolutely is. Putting together a list like this manually would otherwise have taken me at least a couple of hours. And the best part is that I could probably go one step further and ask the ChatGPT agent to contact the shortlisted sellers on my behalf. It’s not the responsible thing to do, but it’s certainly a possibility.

ChatGPT’s Agent mode is not perfect by any means; I watched it go in an endless loop countless times until I was forced to intervene. It can also follow instructions too closely, almost to a fault. For example, it refused to accept a search radius of 51km instead of the 50km I prescribed on a different used auto website. The result? The agent used its virtual mouse to “drag” a slider back and forth for several seconds until it landed precisely on the 50km it needed. Common sense doesn’t come naturally to AI, even in agent mode.

I also noticed a pretty big delay between ChatGPT executing an action like clicking on a link or pressing the browser’s back button. This causes the AI to retry the action, which is a big problem because it inevitably ends up on the wrong page. At times, the agent would click the back button twice and end up on the homepage. It would then restart the whole search all over again, wasting several minutes in the process.

But for all its clumsy moments, the true significance of Agent mode isn’t in its current speed or ability. The fact that I could walk away to make a coffee while an AI autonomously compiled a detailed report on used cars is the main value proposition here. The potential for increased productivity is massive. But at the same time, I’m not sure how I feel about an AI making decisions on my behalf. So the question now is: how much more autonomy are we willing to hand over to AI?

If you’d like to use Agent mode, keep in mind that you will need a ChatGPT Plus subscription. That will set you back $20 monthly, and the feature is currently capped at just 40 messages each month. But if you’re patient, I’m sure it will eventually trickle down to the free tier just like other new ChatGPT features in the past.