Microsoft Really Wants You To Talk To Your PC (and Copilot)

Don’t miss out on our latest stories. Add PCMag as a preferred source on Google.

Set aside your mouse and keyboard. Microsoft really wants you to talk to your Windows 11 PC.

Today, the company unveiled new features for its Copilot AI assistant, also highlighting how it believes voice interactions can sometimes provide a better experience than typing and clicking.

“You should be able to talk to a PC and have it understand you,” Microsoft Consumer Chief Marketing Officer Yusuf Mehdi told journalists. “In our minds, voice will become the third input mechanism to use on your PC.”

Talking to your computer or phone is nothing new (remember Cortana?). But Microsoft is signaling its ongoing enhancements to Copilot have made the assistant smart enough to excel at understanding natural conversation and carrying out the commands. The improvements also apply to all Windows 11 PCs, not just Copilot+ hardware.

(Credit: Microsoft)

According to Mehdi, this can make today’s Windows 11 machines truly “AI PCs,” a term the tech industry has been promoting but is still working to achieve. A key differentiator from other chatbot tech is how Copilot can be used across the OS, the browser, Microsoft programs, and even for a few third-party apps through a recently announced “Connector” function.

“We think we’re on the cusp of the next evolution, which is where AI happens not just in that chatbot, but gets naturally integrated into the hundreds of millions of experiences that people use every day,” Mehdi added.

As part of this push, Microsoft is making “Hey, Copilot” the default wake word for the assistant across Windows 11, following its initial rollout to Insiders in May. The voice command “goodbye” has also been added as a convenient way to end a conversation.

In addition, the company is displaying a new “Ask Copilot” text box in the Windows 11 taskbar, making the assistant even more visible.

Copilot interaction

(Credit: Microsoft)

Perhaps the most intriguing new feature is Copilot Actions, which can complete tasks on your behalf. The company already introduced the function for the web, enabling the assistant to book an online dinner reservation for you or buy a book. That said, it doesn’t offer full automation and requires the user to submit information periodically, as our hands-on testing found.

Get Our Best Stories!

Your Daily Dose of Our Top Tech News

What's New Now Newsletter Image

Sign up for our What’s New Now newsletter to receive the latest news, best new products, and expert advice from the editors of PCMag.

By clicking Sign Me Up, you confirm you are 16+ and agree to our Terms of Use and Privacy Policy.

Thanks for signing up!

Your subscription has been confirmed. Keep an eye on your inbox!

Microsoft now wants to test Copilot Actions for local files on your PC. The experimental mode will be available to beta users in Windows Insiders in Copilot Labs, enabling Copilot to sort through your photos, extract information from PDFs, or edit files in bulk. The same function can interact with both desktop and web applications, as long as you grant it permission.

copilot actions

(Credit: Microsoft)

“While this is happening, you can choose to focus on other tasks. At any time, you can take over the task or check in on the progress of the action, including reviewing what actions have been taken,” Microsoft says.

Despite the voice emphasis, Microsoft also acknowledges that the keyboard and mouse remain primary inputs. On Wednesday, the company demoed some of the Copilot functions to PCMag through voice commands. But interestingly, Copilot struggled to function normally because the demo was happening in a noisy restaurant. It’s why the company is adding text support to Copilot Vision, which lets the AI assistant view and understand what’s displayed on your PC’s desktop to answer your queries.

Recommended by Our Editors

“Previously, you could only use Vision using voice, and Copilot would coach you through it aloud,” Microsoft noted in a blog post. But in an upcoming release for Windows Insiders, Copilot Vision can work with your keyboard and display the corresponding answer with text.

Copilot text to talk

(Credit: Microsoft)

The company is touting these features when 40% of Windows desktop users remain on the older, now-unsupported Windows 10. Market analysts also say the PC market has stagnated, suggesting consumers don’t see a major need to upgrade.

The new Copilot push might convince some to finally buy in. Still, there are drawbacks with voice commands. They can expose your private details to anyone nearby while generating noise, especially if everyone else is talking to their computer. To operate, Microsoft’s Copilot also needs to collect some data from your PC, although the company says: “Once the Copilot Vision session ends, all images, audio, and context are deleted.”

The other issue is how generative AI can hallucinate. Microsoft itself notes that Copilot Actions could produce the wrong output or even be susceptible to “prompt injection attacks,” where text in a web page or document is misinterpreted as a command and executed.

copilot opt in

(Credit: Microsoft)

As a safeguard, Microsoft is asking users to opt in to the Copilot experiences first. The company is also imposing limits on the capabilities, explaining, “There is a well-defined boundary for the agent’s actions, and it has no ability to make changes to your device without your intervention. This access can be revoked at any time.”

In terms of privacy, Microsoft tells PCMag: “Your personal interactions with our services are kept private and are not disclosed without your permission. We remove information that may identify you, like names, phone numbers, device or account identifiers, sensitive personal data, physical addresses, and email addresses, before training AI models.”