Back to Projects
Project 16 - Sep 2026

Brook Vision — Reachy Mini learns my dog

Teaching a Reachy Mini robot to see, hear, and understand my dog Brook — with models I train myself.

This is the project I'm most excited about right now. I have a Reachy Mini on my desk and a dog named Brook who is deeply suspicious of it. I record short videos of Brook around the house — playing, eating, reacting to new rooms and new food — and train Hugging Face vision and audio models on them so Reachy can detect her, follow her, and predict what she'll do next. Still very much in progress, and the most fun I've had with machine learning.

🎬 In progress — training on new clips weekly

🤗 Models on Hugging Face Follow the build on GitHub

Tech Stack

RM
Reachy Mini

Camera, mic, expressive head

HF
Hugging Face

Vision + audio fine-tuning

CV
Video Detection

Brook detector + action model

WM
World-Model Ideas

JEPA-style next-move prediction

The Problem

  • Off-the-shelf detectors know 'dog' — they don't know this dog, her moods, or what her tail is saying
  • Reactions are temporal: a bark, a head-tilt, a sprint to the bowl — single frames miss the story
  • Robots that only see are boring; Reachy needs to hear the 'dinner' word and the food bag too
  • Tiny robot, tiny compute — the models have to be small enough to run near real time

The Idea

  • Personal Brook detector fine-tuned from an open vision backbone on my own labeled clips
  • Action & reaction model over short video windows: playing, resting, alert, begging, exploring a new environment
  • Food-reaction classifier — which foods get the happy dance vs. the polite sniff
  • Audio model for barks, whines and key words so Reachy turns its head toward what matters
  • World-model thinking (JEPA-style): predict Brook's next move from representations, not pixels, so Reachy reacts a beat early

How Reachy Learns Brook

  • Collect — phone videos of Brook in different rooms, lighting, and around food; clips labeled by hand
  • Train — fine-tune vision + audio models on Hugging Face; keep them small enough for the robot
  • Deploy — Reachy's camera + mic stream into the models; detections drive head tracking and gestures
  • Predict — a short-window model guesses what Brook does next (approach, ignore, zoomies)
  • React — Reachy nods, follows, or plays a sound; Brook's response becomes the next label
  • Repeat — every session adds data; the robot gets better at reading one very specific dog

By the Numbers

  • 1 dog (Brook), 1 Reachy Mini
  • Vision + audio models trained from my own clips
  • Short-window video models for actions and reactions
  • Hugging Face for training, hosting and sharing
  • JEPA-inspired next-move prediction
  • Ongoing — new clips every week

Meet Brook (and Reachy)

I share training clips, model cards and the occasional robot fail as they happen. If you're building on Reachy Mini too, say hi.

🤗 Models on Hugging Face Follow the build on GitHub