You can now search the web by uploading a video through Google Lens

Google’s adding a nifty new feature to Lens, its visual search tool: you can now search by shooting a video. Previously, Lens only captured still images, but now you can use both video and audio to ask your question. Google imagines you might use it to ask about a problem you’re having with your car or to get more information about a product you see. It’s also yet another way AI is making its way into Google products — the company announced the new feature at its I/O developer conference, where the story of the year is very much “AI in everything.”

Video is an inherently multimodal input, and multimodal search is a big deal for Google right now. With a still image, Lens has to guess what you’re asking about the photo or prompt you again to refine your question after it processes your image. With video, though, you can just point your camera at your car, say “why is this thing hanging off the bottom,” and Google has everything it needs to both understand and answer your question. All that processing is complex and expensive, but it’s also right in line with the rest of Google’s AI work.

If you search with a video, you’ll still get relatively normal Google search results, says Liz Reid, Google’s head of search. The point is to get there faster and to make it easier to tell Google what you’re looking for. “With some of these, it’s the motion in the video, right? It’s not just that you’re talking with a still image; it’s that something is going on, and you’re trying to figure out how to articulate it.” Let’s say your dishwasher is broken, she says. There are lights blinking, “but which blinking lights, and how do I articulate which lights and the frequency of the blinking? It’s complicated!” With a quick video, though, you can show Google everything it needs; it can figure out your dishwasher model, which lights are doing what, and what the problem might be. There’s just no way to do that in keywords.

Lens is a key part of the future of Google Search, precisely because of its connection to Google’s AI initiatives. Google is obsessed with finding new ways for you to search, both in order to make search easier and also to give you new reasons to Google stuff. “If I were talking to you,” Reid says, “and I assumed you knew everything, how would I ask you something?” Sending a video does seem like a fairly normal, straightforward thing to do in that situation. And as Google continues to refine the way its core product works, making it feel less like a computer and more like an all-knowing best friend appears to be the goal.