Machine Learning Pipeline for Egocentric Object Tracking
A machine learning pipeline that segments and tracks tools in egocentric video of a tailor at work. It chains two foundation models: Grounding DINO, an open-vocabulary detector, localizes objects from the plain-English names a Gemini captioning stage produces, and SAM2 propagates each mask through the video, re-detecting every few seconds so new objects get picked up.
Machine learningComputer visionGrounding DINOSAM2
01 · Raw footage (left) next to the pipeline output (right) — first 10 s
02 · Input — first-person footage, downscaled to 640 px
03 · Output — fabric, ruler, chalk and measuring tape tracked on every frame