Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection

Hadrien Crassous1,*, Mohamed Yassine Kabouri2,*, Minahil Raza3,*, Joni Pajarinen3, Riad Akrour1
1Inria Scool, Univ. Lille, CRIStAL, Lille, France 2LAAS-CNRS, Univ. Toulouse, CNRS, Toulouse, France 3Aalto University, Finland
*Equal contribution
Accepted to IROS 2026
Paper arXiv Code

Abstract

This paper studies how to adapt a computer vision object detector to an unknown environment under both a robot navigation time and annotation budget constraint. Our approach selects informative robot trajectories and image samples to retrain the detector, explicitly targeting its failure cases. Formally, the approach is an embodied variant of batch active learning, where at each round an agent has a limited navigation budget to collect candidate samples and a limited annotation budget for the most relevant images. We leverage spatial consistency to identify images with inconsistent labels, which are likely to provide the greatest improvement to the vision model. We evaluate the approach using different active learning objectives on large scenes from the AI2-THOR simulator and on a real-world setup using a Boston Dynamics Spot robot with the real-time object detector YOLOv5. Through comparison against several baselines, our experimental results show that spatial inconsistency helps guide the agent and select relevant images without external supervision, achieving the highest detection accuracy at the end of the adaptation process under the same budget.

Method Overview

At each round, the agent explores the scene, collects images and detector predictions, and scores them by how inconsistent the predictions are between nearby views. The most inconsistent, diverse samples go to an oracle for annotation and are used to retrain the detector; the loop then repeats.

The collect → score → select → annotate loop:

Figure 1: Overview of our approach — collect, estimate consistency, select inconsistent samples, annotate.

Prediction Discrepancy

To score a frame without ground truth, we compare predictions in the region \(\mathcal{R}(a)\) shared by two consecutive views \(I_s\) and \(I_{s'}\) (reached via action \(a\)): the same objects should be detected in both. Treating predicted classes as multisets, we match same-class objects across the two views and count what's left unmatched:

\[ \Delta(\theta, I_s, I_{s'}, a) = \left| C^\theta_{s|\mathcal{R}(a)} \cup C^\theta_{s'|\mathcal{R}(a)} \right| - \left| C^\theta_{s|\mathcal{R}(a)} \cap C^\theta_{s'|\mathcal{R}(a)} \right| \]

\(\Delta = 0\) means the two views agree; \(\Delta > 0\) flags objects missed or mislabeled in one of them. A frame's final score averages \(\Delta\) over its neighboring views, so frames where the model is unstable score highest — without needing any external supervision.

Two example view pairs and their discrepancy:

Figure 2: Examples of Prediction Discrepancy for two pairs of successive frames (Rotate, LookDown).

Results

Simulation (AI2-THOR)

Prediction Discrepancy consistently outperforms the baseline scoring strategies (Entropy, Count, Random) across YOLOv5 model sizes, and guiding navigation with the Adaptive agent outperforms Frontier-Based Exploration, Sweep, and Random walk baselines under the same budget.

Scoring methods across model sizes:

Figure 3: Aggregate performance of different scoring methods across model sizes (simulation).

Samples picked by each method:

Figure 4: Examples of collected data across scoring methods.

Navigation strategies compared:

Figure 5 (left): IQM comparison of navigation methods.
Figure 5 (right): mAP50 curves for navigation methods.

Real-World (Spot Robot)

We validate our approach on a Boston Dynamics Spot robot equipped with YOLOv5, using OWLv2 as a semi-automated oracle. Prediction Discrepancy again consistently outperforms the baselines, though the performance gap is smaller than in simulation due to the simpler single-room setup and smaller budget.

The indoor test environment:

Figure 6: Indoor experimental environment with the Spot robot.

Objects used in the scene:

Figure 7: Objects used for real robot experiments.

Real-world scoring results:

Figure 8 (left): Real-world scoring methods, YOLOv5m.
Figure 8 (right): Real-world scoring methods, YOLOv5l.

Annotated Data Characteristics

Table I summarizes, for each scoring method, the average number of ground-truth objects in the annotated samples, the entropy of the annotated class distribution, and the model F1 score on the annotated samples, in both AI2-THOR and the real-world scene.

Scoring Method AI2THOR Real-World
#Object Annot. Cls. Entropy (↑) F1annot (↓) #Object Annot. Cls. Entropy (↑) F1annot (↓)
Random 14.33 ± 0.84 3.78 ± 0.08 77.78 ± 1.87 63.47 ± 8.47 2.33 ± 0.12 46.08 ± 4.60
Count 103.26 ± 4.50 3.38 ± 0.15 83.29 ± 2.01 184.07 ± 4.22 2.08 ± 0.05 47.11 ± 1.21
Entropy 85.57 ± 2.39 3.68 ± 0.10 74.41 ± 2.08 163.93 ± 7.76 2.14 ± 0.05 44.62 ± 0.83
Discrepancy (ours) 52.56 ± 1.51 3.96 ± 0.05 67.26 ± 1.66 75.56 ± 6.11 2.28 ± 0.07 39.63 ± 1.99
Oracle(*) 76.22 ± 2.75 3.97 ± 0.07 68.36 ± 1.90 — — —

(*) The Oracle scoring method uses ground-truth labels to select images containing classes with low average precision.

BibTeX

@inproceedings{crassous2026embodied,
  author    = {Crassous, Hadrien and Kabouri, Mohamed Yassine and Raza, Minahil and Pajarinen, Joni and Akrour, Riad},
  title     = {Embodied Active Learning under Limited Annotation and Navigation Budget for Object Detection},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)},
  year      = {2026},
}