How a Live Face Preview Runs in the Browser Without Uploading Anything

The plastic surgery example on this site lets you open your camera and move sliders for a slimmer nose, fuller lips and cheekbones, a face or neck lift, whiter teeth and smoother skin, and see the change live on your own face. Here’s how it works, and why nothing is uploaded.

The pipeline, per frame

camera → <video> → face landmarks (MediaPipe, WebAssembly)
                 → WebGL texture → one fragment shader → <canvas>

Every animation frame with a new video frame:

1. Find the face. MediaPipe’s face landmarker returns 478 points. The preview uses only a handful: the two iris centres, the inner lips (top, bottom, both corners), the nose tip and the bottom of the chin.
2. Steady them. Raw landmarks jitter a pixel or two between frames. A simple blend with the previous frame’s points keeps the effects still on a still face.
3. Draw. The frame is uploaded as a texture and one full-screen fragment shader computes every output pixel.

Everything is measured from the face itself

From those few points the code derives the face’s own coordinate frame:

• ux, along the eye line, and uy, down the face toward the mouth, so a tilted head just works
• d, the eye distance, as the unit of width
• s, eyes to mouth, as the unit of height

Each effect is then described in those units: “the nose is an ellipse 0.5 d wide around the tip”, “the jowls sit 0.85 d to each side of the mouth and 0.3 s below it”. That makes the same numbers work for a phone held close and a webcam across the room.

Reshaping is a backward warp

To make something smaller, you don’t move pixels; each output pixel samples the frame from a little further out. To make something bigger, it samples from a little further in. A smooth falloff, (1 − r²)² inside an ellipse, makes the change fade to zero at the edge so there’s no visible seam.

• Nose: pinch toward the centre line, strongest at the nostrils, fading up the bridge and stopping above the lip; plus a small lift of the tip
• Lips: magnify around the mouth, more in height than in width
• Cheekbones: magnify and raise the apex of each cheek, with a soft highlight
• Face lift: sample from lower down and further out at the jowls and cheeks, so skin moves up and the jaw moves in
• Neck lift: the same idea below the chin

All the displacements are added up in face coordinates and applied in one texture lookup.

Retouching runs on skin only

A skin mask decides where retouching is allowed: inside the face oval, skin-coloured in YCbCr (with generous ranges for every skin tone), and away from the eyes, brows, nostrils and lips, whose detail must stay.

• Wrinkles: a small bilateral blur. Neighbours of similar brightness (fine lines, texture) are averaged; strong edges like the outline of the nose are not.
• Freckles and spots: a pixel clearly darker than the ring of skin around it takes on that skin’s colour.
• Teeth: only inside the mouth opening, bounded by the inner lip line, which curves up toward the corners in a smile. Inside it, light and low-saturation pixels (teeth, unlike lips, gums and tongue) are evened out and whitened.

The first version of the teeth used a plain oval for the mouth opening. In a smile, the lips curve up at the corners, so the oval spilled onto the lower lip and whitened it. Following the actual curve fixed it: test with real smiles.

Privacy is the architecture, not a promise

• The model (face_landmarker.task) and the WebAssembly runtime are served from this site, not a CDN, and only downloaded when someone opens the camera or a photo.
• Frames go from the camera to the GPU and the canvas. There is no network request with image data, and the page has no third-party scripts that could send one.
• If someone books a consultation, only the procedures and slider numbers are sent.

A strict CSP, relaxed for one page only

The site’s Content Security Policy is script-src 'self', and its Permissions-Policy turns the camera off. WebAssembly needs 'wasm-unsafe-eval' (it allows compiling WebAssembly, not eval of JavaScript), and the preview needs the camera. Both are switched on for this one page with an nginx map on the URL:

map $uri $aiw_wasm {
    ~^/work/examples/plastic-surgery/  " 'wasm-unsafe-eval'";
    default                            "";
}
add_header Content-Security-Policy "... script-src 'self'$aiw_wasm; ..." always;

Every other page keeps the strict policy.

Things that bit me on real devices

• Safari on iPhone doesn’t reliably advance frames of a <video> that isn’t in the page. The video element now lives in the DOM (invisible), and the canvas shows the result.
• GPU start-up can hang on some phones instead of failing. Starting the landmarker on the GPU now has a time limit, then falls back to the CPU, and the status line says why if it can’t start at all.
• Phones held upright get an upright camera frame (ask for a portrait aspect ratio), so the face fills the screen.
• The canvas keeps the frame’s shape, set from the actual video size with a CSS custom property, so it can be big without being distorted.

Try it

Open the live preview → and turn your head while moving the sliders: the changes follow the landmarks.

Want something like this on your own site, for a clinic, a salon, glasses, make-up or anything else people want to see on their face before they buy? Tell me about it.