How to Run Object Detection on a YouTube Video in the Browser (Without Downloading It)

Passerby counts the pedestrians, cyclists and vehicles in a video, live, in the browser: an object detection model looks at each frame, a small tracker follows each box, and anything that crosses a line is counted by direction. It already ran on Caltrans’ public traffic cameras and on a visitor’s own video file.

Then the obvious request came in: can I paste a YouTube link?

Try it on a YouTube video →

Two ways that don’t work

Reading the embed. Detection needs pixels: you draw a video frame onto a canvas and hand it to the model. A YouTube embed is an <iframe> from another origin, and the browser won’t let your page see inside it. There’s no video element you can reach, and the player’s JavaScript API gives you play, pause and the current time, never the frames. That’s the same-origin policy doing its job.

Downloading the video. Fetching the stream on a server (with yt-dlp or similar) and sending it back to the browser gets you pixels, but YouTube’s terms don’t allow downloading, YouTube blocks a lot of server traffic, and you’d be running video through your own server for every visitor. Not something to build a page on.

The way that works: let the visitor share the tab

The browser’s screen-sharing API, getDisplayMedia, hands your page a MediaStream of whatever the visitor chooses to share. If they share the tab with the YouTube player in it, you have the pixels: with the visitor’s permission, in their own browser, without downloading anything.

const stream = await navigator.mediaDevices.getDisplayMedia({
  video: { frameRate: { ideal: 30 } },
  audio: false,
  preferCurrentTab: true,      // Chrome: offer this tab first
  selfBrowserSurface: 'include',
  surfaceSwitching: 'exclude',
});

preferCurrentTab makes Chrome and Edge offer the current tab at the top of the picker, so it’s one click. The call must come from a click; a page can’t start sharing by itself.

Keep only the player

Sharing the whole tab gives you the whole tab: the page, the counter, and the canvas showing the capture of the canvas showing the capture. A hall of mirrors. You want the player alone.

Chromium browsers have two ways to do that:

• Element Capture (RestrictionTarget and track.restrictTo) captures one element and what’s inside it, even if something is drawn on top of it. The element must form its own stacking context; isolation: isolate on a wrapper around the iframe does it.
• Region Capture (CropTarget and track.cropTo) crops the capture to an element’s box on the screen. It’s older and still works where Element Capture isn’t available.

async function cutTo(track, el) {
  try {
    if (window.RestrictionTarget && track.restrictTo) {
      await track.restrictTo(await RestrictionTarget.fromElement(el));
      return true;
    }
    if (window.CropTarget && track.cropTo) {
      await track.cropTo(await CropTarget.fromElement(el));
      return true;
    }
  } catch {
    // The visitor shared another tab, a window or the screen: those can't be cut down.
  }
  return false;
}

Both only work on a capture of the current tab. If the visitor picks something else, the call rejects and you count the whole picture; Passerby tells them to draw the counting line over the road in that case. Other browsers don’t have either API, and phone browsers don’t offer screen sharing at all, so the page checks for getDisplayMedia after it loads and says so when it’s missing.

From there it’s just video

Put the stream into a <video> element and it behaves like any other video, mostly:

video.srcObject = stream;
await video.play();
// then, per frame: ctx.drawImage(video, 0, 0) and run detection on the canvas

The one real difference is time. For Caltrans video, Passerby plays two copies of the stream: detection runs on the first, and the second is held a second or so behind and drawn with its boxes, so every box is placed between two detections and lands exactly on its car. A screen capture is live in both copies. You can’t seek it, so you can’t hold one back.

So for a shared tab Passerby keeps one clock for both copies, seconds since sharing started, and draws each box where the latest detection found it, fading it if no new detection follows. On a fast laptop the lag is a fraction of a second. The counts don’t care: a car is counted when its tracked center crosses the line, whatever the drawing does.

The rest of the page

• The embed is YouTube’s own player in privacy-enhanced mode (youtube-nocookie.com), muted so it can autoplay. The page’s Content Security Policy allows that one origin in frame-src, on that one page.
• Links are parsed for every shape YouTube hands out: watch?v=, youtu.be/, /shorts/, /live/, /embed/, with t=90 or t=1m30s start times.
• Two videos on screen need explaining, so the page says plainly which is which: YouTube’s player is where you control the video, and Passerby’s copy is where the counting is drawn.
• Stopping is the browser’s job: the visitor ends sharing from the browser’s own bar, the track fires ended, and the counts freeze, ready to download.

Where this pattern fits

Anything that needs to look at a video you can’t fetch: a counter like this one, accessibility checks on a meeting’s shared slides, a study tool that reads code off a programming stream. The visitor stays in control (they pick what to share and can stop at any time), the frames never leave their device, and nobody downloads anything they shouldn’t.

Building something that needs to see what’s on the screen? Tell me about it.