First look: Gemini is learning to get smarter when you’re asking about your screen

How many different ways does Google give you to make sense out of what you’re seeing on your Android phone’s screen? You could open a screenshot in Lens if you were trying to be as clunky as possible, or just pull up Circle to Search for some quick analysis. Now we’ve also got Gemini ready to lend a hand, and its “Ask about screen” mode is already a deceptively powerful tool. Today we’re checking out one small way that Google could make this screen-searching solution even more streamlined.

Right now, the way you get Gemini to start answering questions about your screen’s contents takes a step or two. First you pull up the Gemini overlay, either with a hotword or button shortcut. Then you have to manually tap on “Ask about screen.” And only then can you ask Gemini your question — if you start directly without hitting “Ask about screen” first, Gemini won’t understand the context.

That works, but like we said, involves a few steps — could there be a better way of doing things? This still appears to be just in testing, so we can’t say with any confidence if Google will follow through with this approach, but in version 16.37.46.sa.arm64 of the Google app, we’ve uncovered evidence of an alternate workflow.

Google could add a new “Screen Context” setting to Gemini that would “Let gemini infer when it should access your screen for more context.” Basically, that would solve exactly the problem we just mentioned, where you ask Gemini a question about your screen prior to manually sharing it. With this feature engaged, Gemini would be able to recognize what you were trying to do and start looking at your screen, even without being explicitly told to.

The top option here would take you to the Digital Assistant app settings we already have, including the option to be able to access screenshots — so between those two settings, Google wants clear permission before it lets Gemini start looking at your screen. The improper lower-case “gemini” branding should also clue you in that we’re looking at an in-progress UI, and that none of this is ready for prime time just yet.

With the setting engaged, when Gemini realizes you’re asking about something on the screen, you’ll see a “Getting app content…” message pop up momentarily, before you get your Gemini report — there’s even a handy notice at the bottom spelling out exactly what just happened, in case you turned it on by accident and would rather toggle it back off. Here’s an early preview of it all in action:

Assuming Google’s reasonably successful about knowing when a question refers to something on your screen, this sounds like a decently executed compromise between giving Gemini full rein, and saving us from unneeded taps. We’ll keep an eye on future releases and will let you know if and when this becomes available for the public to finally try out.