Trying to find smoking scenes in movies

This started with a simple idea. I wanted to take a screenshot from a movie every few seconds, embed all of them, and search for something like "a person smoking a cigarette."

I had no idea how well it would work. Cigarettes show up pretty often in older movies, so there were plenty of examples to test with. It also seemed like a fun way to learn how visual search, reranking, and fine-tuning fit together.

Starting with embeddings

I sampled one frame every two seconds from seven films, which produced 23,499 JPEGs. I used Qwen3-VL-Embedding-2B on Modal L4 GPUs to turn each image into a 2,048-dimensional vector, then searched those vectors with FAISS.

The first results were better than I expected. A text query about smoking brought back a lot of real smoking scenes. Once I looked more closely, though, I found plenty of strange matches mixed in. There were people touching their faces, drinking through straws, holding pipes, or just standing near smoke. The embedding model understood the general idea, but it wasn't actually checking whether a cigarette was visible.

So embeddings were useful for finding possible matches, but they were not enough to prove that a cigarette was really there.

Side note: Since it is a 2B-parameter model, you could probably run it locally. I just wanted to test out Modal.

Learning from the wrong results

Next, I added Qwen3-VL-Reranker-2B to take a closer look at each candidate. It helped, but I kept seeing the same mistakes. Hands near mouths were a big one. It also picked frames from just before or just after someone actually smoked.

I labeled those mistakes and trained a small LoRA adapter. A LoRA keeps most of the original model frozen and learns a small set of extra weights. In practice, I was showing the model which results I wanted and which lookalikes kept fooling it.

I trained on 443 frames: 174 positive and 269 negative. Neighboring frames stayed in the same split, so nearly identical images couldn't end up in both the training and evaluation.

At a 0.5 threshold, the LoRA fixed 19 mistakes from the default model and introduced 3 new ones. False positives dropped from 17 to 7, and false negatives dropped from 13 to 7. This is a small test set that I made for the project, so the numbers are not definitive. Still, it was a good sign that labeling my own failure cases was helping.

These examples helped me see what changed rather than relying only on one score.

Trying crops because the objects were tiny

Even after 'fine-tuning', I still had a basic problem. Sometimes the cigarette was only a few pixels wide. I thought the model might do better if I zoomed in, so I tried dividing promising frames into four large, overlapping crops and running the same reranker again. Each crop covered 60% of the width and 65% of the height.

I used overlapping crops because a hard four-quadrant split could cut a cigarette or a person in half. In this frame from The Good, the Bad and the Ugly, the full image scored only 0.063. The upper-right crop made the cigar easier to see and raised the score to 0.875.

Trying another crop size

The first crops helped, so I tried zooming in a little more. I added six smaller overlapping crops covering 50% of the width and 60% of the height. There wasn't a deep reason for those exact numbers. They made the object larger without cutting away too much of the person, which was enough reason to test them.

For each frame, I now keep the highest score from the full image and all ten crops. A crop score only puts the frame into the review pile; it does not prove that the result is correct.

The example below is from The Sting. The full frame scored 0.010 and the best original crop scored 0.154, both below my 0.4 candidate threshold. One of the finer crops scored 0.836, so the frame was no longer missed.

On my 83-frame test set, recall at a 0.5 threshold went from 79.4% with only the full image to 85.3% with the first four crops. With both crop sizes, it reached 94.1%. Across all sampled frames, the smaller crops found 108 extra candidates, but only 26 survived final review. Zooming in found real scenes that I was missing, but it also produced a lot more junk.

Using another model as a second pair of eyes

At first I reviewed uncertain images myself. That got repetitive pretty quickly, so I tried using DeepSeek V4 Flash Vision through OpenCode as another pair of eyes. It doesn't search the movies. It only looks at candidates the embedding and reranker have already found.

I use two passes:

1. The first pass sees the full frame and enlarged crops and checks for a cigarette, cigar, or cigarillo.
2. The second pass looks at the original frame and tries to find a reason the first answer was wrong.

I don't assume its answers are correct. It is another model, and it can be confidently wrong too. Still, it has helped me get through uncertain frames and catch mistakes that the Qwen reranker keeps making.

Making the results easier to look through

Sampling every two seconds creates many nearly identical images. I group accepted frames by movie and rolling three-minute window. The gallery shows one image for each group, and clicking or using the arrows cycles through the rest. This does not change the search quality; it just makes the results less repetitive.

What is still unfinished

The benchmark above was made for conventional paper cigarettes. The current LoRA was originally taught to reject cigars and cigarillos, but I later decided the gallery should count those too. That means I should not treat the cigarette benchmark as proof that the complete cigarette-and-cigar search works equally well.

I made this change because Clint Eastwood's character smokes a cigar in The Good, the Bad and the Ugly. The model correctly identifies it as a cigar, so it ignores it. In The Sting, however, there are very few paper cigarettes; most of the smoking scenes involve cigars. As a result, the current logic is a little roundabout.

The next thing I want to try is retraining with cigars labeled as positives, then comparing the old and new versions on scenes or movies they haven't seen. I also need a larger held-out set before I take small metric differences too seriously.

There wasn't one big fix that made the project work. I would look at the latest mistakes and try the simplest idea that might help. Embeddings found roughly related frames. My labels helped the reranker handle repeated mistakes. Crops made tiny objects easier to see, and a separate vision model gave me a second opinion. Most of the project came together by trying one small thing after another.

Next, I want to retrain the model with cigars and cigarillos labeled as positives. After that, I want to test it on more movies and a larger held-out set to see how well it works on scenes it has never seen before.

Open the image results

See the benchmark details