Building Curtain: searching 2.48 million movie frames

I wanted to build something which allowed people to match a screenshot they have with the specefic scenes from said movie.

for example, you would find a image on pinterest but you have no clue what movies its from. You open curtain paste and boom

That turned into Curtain. It now searches 2,481,255 frames from 150 movies. .

150movies
2,481,255searchable frames
128values per embedding

How it works

I sample each movie at two frames per second (probably overkill). I used 50,000 of those frames to train a ResNet-18 from scratch, then froze the model and ran all 2.48 million frames through it to build the actual search index.

Diagram of Curtain's movie extraction, training, indexing, and search pipeline
The expensive work happens once. A search only has to encode one upload and compare it with the saved frame embeddings.

The model turns every image into 128 numbers. Since those vectors are normalized, searching is basically one large dot product. I didn't need a vector database or anything especially clever.

Teaching it what “the same frame” means

I wasn't trying to find images with the same general subject. Two completely different movies can both have a close-up of a face or a car at night. I needed the model to recognize two altered versions of the exact same shot.

During training, I make two copies of a frame and mess with them independently—lighting, blur, resolution, JPEG quality, and sometimes a crop. The model learns to pull those two embeddings together and push unrelated frames away.

Diagram showing two altered views of one frame being pulled together while an unrelated frame is pushed away
Two altered copies of one frame are a positive pair. Everything else in the batch is a negative.

I also split training and validation by whole movies. Otherwise, nearly identical neighboring frames could leak into both sets and make the model look much better than it really was.

Getting through a few million images

I ran training and indexing on L4 GPUs through Modal. The indexer decodes JPEGs on the GPU, processes batches of 512, and stores compact float16 embeddings. It also records hashes and shapes so the site can't accidentally load an index made by the wrong model.

The live search runs on my own server,It embeds the uploaded image on the CPU, scans the saved matrix in chunks, and streams the progress to the browser.

One problem I actually understand

I'm calling Curtain somewhat finished, but it definitely isn't perfect. These three uploads are from the same shot. The first two miss; the third one works.

What disappears when the frame is cropped

Wide crop

The actual wide crop used in the failed search

MISS

The wallpaper, mirror, and lower frame are gone.

Portrait crop

The actual portrait crop used in the failed search

MISS

Most of the background is gone, and the aspect ratio changed.

Outside the training crops Training always kept at least 75% of both width and height.

Full frame

The actual full frame used in the successful search MATCH · full context stays
These are the actual three query images. The full frame gives the model the room layout and the original scale of the person; the two crops do not.

This makes sense once I look at how the model was trained. My crop augmentation always keeps at least 75 percent of the original width and height. These first two images go well beyond that, especially the portrait crop.

I see the same person and immediately know it's the same shot. The model also cares about the framing and the room around him. Once enough of that disappears, another frame can end up closer in embedding space.

I could train with more extreme crops or search several crops for every upload, but that would bring more compute and probably more false matches. I don't really feel like chasing that right now. The search works, the failure makes sense, and that's good enough for me to call the project done.

The diagrams were generated by GPT-5.6 Sol.