Simone Rossi
← All publications

Class Balanced Dynamic Acquisition for Domain Adaptive Semantic Segmentation using Active Learning

  • Computer Vision

Class Balanced Dynamic Acquisition reweights pixel-acquisition scores using observed class counts and shares the annotation budget across images, improving minority-class segmentation on GTAV to Cityscapes.

Paper NeurIPS Workshop on Adaptive Experimental Design and Active Learning in the Real World 2023

Marc Schachtsiek1, Simone Rossi2, Thomas Hannagan2

  1. PSL University & Stellantis
  2. Stellantis
PublicationPDF

Acquisition scores based on uncertainty and diversity can increasingly favor common classes as the labeling budget grows. We introduce Class Balanced Dynamic Acquisition (CBDA) to allocate pixel labels when adapting a semantic segmentation model from synthetic to real driving scenes. CBDA uses previously acquired class counts to reweight these scores and selects pixels across the target dataset, allowing each image to receive a different budget. On GTAV to Cityscapes, the combination improves both mean and minimum class intersection-over-union (IoU) over the Region Acquisition (RA) baseline, with larger gains at higher budgets.

Why acquiring more labels can increase imbalance

We start with labeled source images S={(Xs,Ys)}\mathcal{S}=\{(X_s,Y_s)\} and unlabeled target images T={Xt}\mathcal{T}=\{X_t\}. Training combines supervised source learning with unsupervised target objectives. At each acquisition round, the model scores target pixels; queried labels then contribute a supervised target loss. Our experiments reveal selected pixels' existing ground-truth labels, simulating an annotator.

The baseline RA allocates the same pixel budget to every target image. In our GTAV experiments, larger budgets produce increasingly imbalanced selections and lower segmentation scores. Rare classes suffer most from this degradation, motivating control over the distribution of acquired labels.

CBDA training diagram showing class-weighted pixel scores and dataset-wide selection.
Source supervision, target objectives, and active labels train the model. CBDA weights acquisition scores using acquired class statistics, then stacks the score maps across images. These two operations control which classes and images receive labels.

Turning class counts into acquisition weights

At round ii, let Lc,iL_{c,i} count previously acquired labels of class cc. For ∣T∣|\mathcal{T}| target images of height HH and width WW, total labeling fraction BALB_{AL}, CC classes, and NiN_i rounds, we define a uniform cumulative class target and its weight:

Bc,i=(∣T∣×W×H)BALCiNi,Wc,i=max⁡(1−Lc,iBc,i,ϵ).\begin{aligned} B_{c,i} &= (|\mathcal{T}|\times W\times H)\frac{B_{AL}}{C}\frac{i}{N_i},\\ W_{c,i} &= \max\left(1-\frac{L_{c,i}}{B_{c,i}},\epsilon\right). \end{aligned}

A class approaching its target receives a smaller weight. The positive floor ϵ\epsilon, for example 10−610^{-6}, keeps every class eligible and preserves score differences within classes whose targets are filled. The uniform distribution serves as a soft target when the target-domain class frequencies are unknown.

We apply the weight according to the model's predicted class Y^t(u,v)\hat{Y}_t^{(u,v)} at pixel (u,v)(u,v):

A^(u,v)=A(u,v)Wc,i,c=Y^t(u,v).\hat{\mathcal{A}}^{(u,v)} =\mathcal{A}^{(u,v)}W_{c,i}, \qquad c=\hat{Y}_t^{(u,v)}.

Here A\mathcal{A} is the underlying acquisition score. Counts use acquired labels; prospective queries use predicted classes. Dynamic Acquisition (DA) stacks all weighted maps into a tensor of shape ∣T∣×H×W|\mathcal{T}|\times H\times W and selects its highest-scoring eligible pixels. This allows an image containing useful minority-class pixels to receive more queries.

Histograms comparing per-image acquired pixel fractions for RA and DA at five and ten percent budgets.
RA assigns nearly identical labeling fractions to every image, while DA spreads queries across images unevenly. The broader distributions show how dataset-wide selection reallocates queries across images.

Improvements concentrate on minority classes

We evaluate DeepLabv3+ with a ResNet-101 backbone. For GTAV to Cityscapes, CBDA reaches 72.3, 72.2, and 71.2 mean IoU (mIoU) at 5%, 10%, and 20% labeling budgets. The corresponding RA reruns achieve 71.7, 70.5, and 68.8. Minimum class IoU improves by 0.5, 2.9, and 4.6 points. The 5% CBDA result also exceeds the fully supervised comparison's 71.8 mIoU in this evaluation.

GTAV to Cityscapes results table with per-class IoU and mean IoU for all compared methods.
Classes run from most to least frequent. The paired RA/CBDA reruns between double rules show increasing gains at larger budgets. Other published methods appear above them; ILM-ASSL achieves a higher overall score than CBDA in this comparison.
Relative per-class IoU improvements of CBDA over RA at three labeling budgets, ordered by class frequency.
Relative gains generally increase toward rarer classes, with individual exceptions such as train at 20% and traffic light at 5%. The dotted lines describe qualitative trends across a categorical class axis. Each budget uses its own RA reference.

Why balancing and dynamic allocation work together

We compare RA, DA, class-balanced RA (CBRA), and CBDA at 5% and 10%. CBDA performs best in each displayed budget and acquisition schedule. DA alone underperforms RA at 5%, while spreading acquisition rounds further through training improves several configurations. These comparisons show typical runs, with greater run-to-run variability for DA alone.

Two-panel ablation comparing RA, DA, CBRA, and CBDA under original and spread acquisition schedules.
The left and right panels use 5% and 10% budgets. “Original” queries at training iterations 10k, 12k, 14k, 16k, and 18k; “Spread” uses 10k, 18k, 24k, 28k, and 30k. Both allocation and acquisition timing affect the result.

We quantify label imbalance through normalized KL divergence:

imbalance_score=KL(Qclass∥Qunif)KL(QOH∥Qunif).\mathrm{imbalance\_score} =\frac{\mathrm{KL}(Q_{class}\Vert Q_{unif})} {\mathrm{KL}(Q_{OH}\Vert Q_{unif})}.

QclassQ_{class} is the acquired class distribution, QunifQ_{unif} is uniform, and QOHQ_{OH} places all mass on one class. The score ranges from zero for uniform labels to one for single-class labels. CBDA's lower scores connect the acquisition rule to the intended balancing effect.

Normalized label imbalance for all acquisition variants at five and ten percent budgets.
Class balancing reduces selection imbalance, and CBDA produces the lowest values in each displayed setting. The corresponding mIoU comparisons show that acquisition timing and allocation also affect segmentation performance.

Dependence on source domain and labeling budget

With SYNTHIA as the source, CBDA scores 70.2 versus RA's 70.3 mIoU at 5%, then improves by 0.8 and 0.5 points at 10% and 20%. The benefit varies with source domain and budget. For GTAV, both RA and CBDA lose performance as the budget increases, while CBDA reduces the size of this decline.

SYNTHIA to Cityscapes table comparing per-class and mean IoU across labeling budgets.
The second source domain gives smaller gains than GTAV and a slight decrease at 5%. Its fully supervised comparison reaches 72.2 mIoU, above the displayed CBDA results.

CBDA operates on pixel-acquisition scores, with RA supplying the scores in these experiments. The labeling budget counts sparse pixel queries whose ground-truth labels are revealed during acquisition. It measures the amount of supervision supplied to the model; annotation time also depends on the interface and the human labeling workflow.