Michał Chromiak's blog - Self-Supervision

I-JEPA: Learning from images by predicting missing features

2023-06-17T14:43:00+02:00

Imagine covering the head of a dog in a photograph. The visible body and surroundings give you clues about the hidden region. You may infer its shape and position without knowing the exact color of every pixel.

I-JEPA turns that distinction into a learning task: predict useful features of a hidden region, rather than draw the missing pixels. It learns from images without requiring class labels during pretraining. Afterward, its features can serve as inputs to models for tasks such as classification.

The name stands for Image-based Joint-Embedding Predictive Architecture. Mahmoud Assran and colleagues introduced the method in a paper first submitted in January 2023, later published at CVPR 2023. Meta released the code and checkpoints in June. This article follows the paper's April 2023 revision.

What is a feature, if it is not a pixel?🔗

A pixel records color at a particular location. An encoder is a neural network that transforms pixels into an embedding, a vector of learned numbers. Those numbers can make relationships between images easier to recognize: which shapes belong together, which regions look alike, or how an object relates to its surroundings.

An embedding is not a written description. I-JEPA does not receive a label such as "dog's head" for the hidden region, and no one assigns that meaning to a particular coordinate of its output. The training task shapes the representation; downstream evaluations test whether useful information has emerged.

Why predict embeddings? Reconstructing pixels rewards getting every visible detail right. A representation-learning task may benefit more from retaining object structure than from reproducing the exact texture of grass. I-JEPA lets a learned encoder provide the prediction targets instead of fixing those targets to raw pixels.

That choice alone is not enough. The model also needs a prediction problem that encourages it to use meaningful context. The architecture and the masks work together to create one.

Follow one image through I-JEPA🔗

I-JEPA divides an image into a grid of patches, small non-overlapping squares. Its encoders use Vision Transformers, which process these patches as a sequence and allow information to move between them through attention.

There are three networks with different jobs:

The context encoder reads the visible patches. Some regions are withheld. The encoder produces features for the patches it is allowed to see.
The target encoder reads the complete image. It produces one representation per patch. The training targets are selected from these outputs at the locations of the withheld regions.
The predictor fills in the missing features. It receives the context features and position information telling it which region to predict. Its output is compared with the corresponding target-encoder features.

Figure 1. The I-JEPA training architecture, from the paper's figure 3. Matching colors identify the same target region in the two branches. The repeated predictor boxes share their weights; they are not separately trained models.

The order of operations matters. Target regions are selected after the target encoder has processed the complete image. A target feature can therefore incorporate information from elsewhere in that image. The context encoder, by contrast, never receives the hidden target patches as input.

The predictor also needs to know where to predict. Otherwise, the same visible context could be used to ask about several different missing regions. I-JEPA supplies a shared learnable mask token at each requested position, with a positional embedding added to it. These tokens specify locations; they do not reveal the missing pixels. Paper, section 3.

How the networks learn together🔗

The prediction error updates the context encoder and predictor through backpropagation. The target encoder is updated differently: its weights track an exponential moving average of the context encoder's weights.

Think of the target encoder as a slowly changing reference. Each update retains most of its previous weights and adds a small contribution from the newly trained context encoder. The targets improve as training progresses, but do not chase each individual gradient update directly.

This design addresses a risk called representation collapse. If every image produced the same vector, predicting that vector would be easy, but the representation would be useless. I-JEPA uses the asymmetry between the two branches, the predictor, and the moving-average target encoder to train without explicit negative-image pairs. The paper reports that the moving-average encoder is essential in its experiments; averaging alone is not a general mathematical guarantee against collapse.

Why the mask is part of the method🔗

Hiding a few isolated pixels can create a task solvable from nearby color and texture. Hiding almost everything can leave too little evidence to make a useful prediction. I-JEPA instead samples several substantial target regions while keeping context distributed around them.

Its default recipe is:

Sample four target blocks, each covering roughly 15-20% of the image area, with aspect ratios between 0.75 and 1.5. Target blocks may overlap one another.
Sample a square context block covering roughly 85-100% of the image area.
Remove from the context every patch that overlaps any target block.

The 85-100% figure describes the context block before those removals. It is not the fraction of image patches actually passed to the context encoder. The remaining context can have several holes and be much sparser. Likewise, four 20% targets do not necessarily hide 80% of the image, because targets can overlap.

This gives the predictor a different problem from reconstructing scattered pixels: use evidence from visible regions to infer features across several larger missing regions. The paper's masking ablations show that replacing this strategy with random patches or simple quadrants sharply reduces performance in the tested low-label linear-probe setting. Paper, figure 4 and table 6.

Does "no hand-crafted augmentations" mean no cropping?🔗

No. Here the distinction is between constructing multiple transformed views whose representations must agree and constructing a context-to-target prediction task within one image view.

The official I-JEPA pretraining configuration still uses random resized cropping and normalization. Its default recipe disables color distortion, Gaussian blur, and horizontal flipping. The masks themselves are also deliberate design choices. I-JEPA reduces reliance on the multi-view augmentation recipe used by methods such as DINO; it does not eliminate preprocessing or all assumptions about images. Official transforms, ViT-H/14 configuration.

The prediction objective, with the paper's notation

Let $M$ be the number of target blocks, normally four. For block $i$, $B_i$ is the set of patch positions inside that block. The vector $\boldsymbol{s}_{y_j}$ is the target encoder's output for patch $j$; $\hat{\boldsymbol{s}}_{y_j}$ is the predictor's estimate. The collections of vectors for a whole block are written $\boldsymbol{s}_y(i)$ and $\hat{\boldsymbol{s}}_y(i)$.

Section 3 writes the objective as follows, with the equality split across two lines here for readability:

$$ \begin{aligned} &\frac{1}{M}\sum_{i=1}^{M} D\left(\hat{\boldsymbol{s}}_y(i),\boldsymbol{s}_y(i)\right)\\ &\qquad=\frac{1}{M}\sum_{i=1}^{M}\sum_{j\in B_i} \left\lVert\hat{\boldsymbol{s}}_{y_j}-\boldsymbol{s}_{y_j}\right\rVert_2^2. \end{aligned} $$

Read the right-hand side from the inside out: take the difference between the predicted and target vectors, compute its squared Euclidean length, sum over the patches in a block, then average across the $M$ blocks. The paper's displayed expression has a block average and a patch sum; it does not include a separate division by the number of patches in each block.

The context encoder has parameters $\theta$, the predictor has parameters $\phi$, and the target encoder has parameters $\bar{\theta}$. Only $\theta$ and $\phi$ receive gradients from this objective. The moving-average update can be written as:

$$ \bar{\theta}\leftarrow m\bar{\theta}+(1-m)\theta. $$

This last equation spells out the update described in the paper's prose. The momentum $m$ starts at 0.996 and increases toward 1 during pretraining. A larger value retains more of the target encoder's previous weights. Paper, section 3 and appendix A.1.

Paper versus implementation: the released training code normalizes target features and uses smooth_l1_loss with its default mean reduction, rather than the squared-L2 expression printed above. Both train feature prediction, but they are different numerical losses. The equation here follows the paper; reproducing the released checkpoints requires following the implementation.

How it differs from MAE and DINO🔗

All three methods learn from images without needing class labels during pretraining. What changes is the training question.

Method	Training question	Prediction target
MAE	Can visible patches reconstruct the missing patches?	Pixel values, typically normalized within each patch
DINO	Can a student match its teacher across different augmented views?	The teacher's output distribution over learned features
I-JEPA	Can visible context predict features at specified hidden positions?	Patch representations from the moving-average target encoder

Figure 2. The paper's figure 2 compares architecture families. In the right-hand panel, the predictor estimates a target representation rather than a target image. For I-JEPA, the conditioning information marked $z$ specifies the positions to predict.

The middle and right panels make the central difference visible: a decoder predicts the signal; I-JEPA's predictor predicts its representation. A useful representation need not preserve every detail that a pixel reconstruction requires.

For the reconstruction side of this comparison, see the MAE article. The DINO article explains the teacher-student, multi-view approach.

What happens after pretraining?🔗

Pretraining produces features, not a ready-made classifier for every task. To evaluate those features, the paper uses the target encoder, generally averaging its patch representations into an image-level vector.

A linear probe keeps the encoder frozen and trains a simple classifier on top of its features. This asks whether the representation already makes categories easy to separate. Labels are used to train that classifier, even though they were not used to pretrain the encoder.

Fine-tuning also updates the encoder for the downstream task. It can adapt the representation more extensively, but answers a different question from linear probing. Keeping these protocols separate makes the results much easier to interpret.

Classification: useful features with fewer pretraining epochs🔗

The following results are from the paper's full-label ImageNet-1K linear evaluation, where top-1 accuracy is the percentage of images assigned the correct highest-scoring class:

Method and encoder	Pretraining epochs	Top-1 accuracy
MAE, ViT-H/14	1,600	77.2%
I-JEPA, ViT-H/14	300	79.3%
I-JEPA, ViT-H/16 at 448-pixel resolution	300	81.1%

An epoch is one pass through the training dataset. ViT-H denotes the Huge model; /14 and /16 denote the patch width in pixels. The higher-resolution row processes a different patch grid, so it is not the same computation as the standard ViT-H/14 run.

The matched ViT-H/14 comparison is the clearest result: I-JEPA reaches higher linear-probe accuracy after substantially fewer pretraining epochs. That does not make it the winner over every model in the table: DINO's different ViT-B/8 configuration reaches 80.1%. Paper, table 1.

Low-label learning and spatial information🔗

With only 1% of ImageNet's labels, I-JEPA's ViT-H/14 reaches 73.3% accuracy, compared with 71.5% for MAE's ViT-H/14. This is a separate experiment: the I-JEPA encoder is fine-tuned, not frozen. The 448-pixel I-JEPA variant reaches 77.3%. These results support label-efficient adaptation, not a claim that all downstream tasks need no fine-tuning. Paper, table 2 and appendix A.2.

The paper also tests frozen features on CLEVR, a dataset of synthetic scenes. Its counting and distance tasks check whether the representation retains information beyond object category. I-JEPA scores 86.7 on counting and 72.4 on the distance task; MAE scores 90.5 and 72.4 respectively. I-JEPA therefore retains useful spatial information, but does not surpass MAE on both tasks. These benchmarks should not be read as a demonstration of dense, real-world depth maps. Paper, table 4.

Where the compute savings come from🔗

Meta reports training the approximately 632-million-parameter ViT-H/14 on 16 A100 GPUs in under 72 hours. That is elapsed time for a multi-GPU run, not a single-GPU training budget. At 72 hours, 16 GPUs would account for 1,152 GPU-hours. Meta's June 2023 announcement.

The efficiency comes from several places. The context encoder processes only visible patches. The predictor is narrower than the encoders. The target encoder processes one complete image view rather than a collection of separately augmented views. Most importantly, useful features emerge after fewer pretraining iterations in the reported comparisons.

There is still a cost to generating targets. Section 7 reports that an I-JEPA iteration is about 7% slower than the compared MAE setup; the saving comes from needing fewer iterations. The introduction reports more than a tenfold GPU-hour advantage over its ViT-H/14 MAE reference. That is a comparison of the reported training runs, not a universal tenfold speedup for any model or dataset. Paper, section 1 and section 7.

The distinction is practical: making each training step cheaper and needing fewer steps are different routes to reducing the total bill.

What does it mean to learn a "world model"?🔗

Predicting a hidden region requires learning regularities: which parts tend to occur together, how shapes continue, and what the surrounding image makes plausible. That is the motivation behind connecting I-JEPA to learning through observation.

The evidence here is about spatial relationships in still images. I-JEPA does not learn action-conditioned future states, plan a sequence of actions, or demonstrate general common-sense reasoning. The authors describe the predictor as a restricted kind of world model, not a completed model of how the physical world works.

The image-completion visualizations need one further distinction. I-JEPA itself outputs vectors. To make those vectors visible, the researchers freeze the pretrained model and train a separate generative decoder to produce image samples from the predictor's representations. Structure shared across samples suggests what the representation captures; varying texture shows details it leaves unspecified. Those pictures are a diagnostic tool, not evidence that I-JEPA was trained to generate images. Paper, section 8, official visualizations.

The takeaway🔗

I-JEPA changes both what the model predicts and what it must use to make the prediction. A learned target encoder supplies feature-space targets, while large missing regions and distributed context make the prediction task informative.

The result is a way to learn reusable visual features without pixel reconstruction or an elaborate multi-view augmentation recipe. The paper's strongest evidence is the quality of those features under clearly defined evaluations, together with the training effort needed to obtain them.

I-JEPA addresses learning representations from unlabeled images. For a different stage of the machine-learning workflow, the QLoRA article explains how to adapt an already pretrained language model using less GPU memory. The two articles explore different efficiency questions: learning useful features versus making task-specific fine-tuning more affordable.

Original sources🔗

Assran et al., Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture, CVPR 2023. The method, equations, and tables discussed here follow arXiv version 3.
Official I-JEPA repository, with training code, configurations, and pretrained checkpoints.
Meta's June 13, 2023 announcement, which places the work in the broader JEPA research program.

Both figures are reproduced from Assran et al.'s paper and retain their original content.

🦖 DINO: Self-supervised ViTs learn strong features and semantic structure

2021-05-03T17:44:00+02:00

The DINO paper explained🔗

This article reviews Emerging Properties in Self-Supervised Vision Transformers by Caron et al., published at ICCV 2021.

DINO stands for self-DIstillation with NO labels. It is a self-supervised representation-learning method: it learns from images without human-provided class labels. A student network is optimized by gradient descent to match a teacher network's output for another view of the same image. The teacher has the same architecture but is not trained by backpropagation; its weights are an exponential moving average (EMA) of the student's weights.

The method works with both convolutional networks and Vision Transformers (ViTs), but the paper's most interesting findings concern ViTs. DINO-trained ViTs provide strong image-level features and final-layer self-attention maps that often align with foreground objects and their boundaries.

TL;DR🔗

DINO is a non-contrastive self-supervised method inspired by self-distillation and momentum-teacher methods such as BYOL and Mean Teacher. It is not simply a successor to SimCLR, MoCo, or BYOL.
The student and teacher use the same architecture but have different parameters.
The student processes all augmented views; the teacher processes only the two large global crops.
Training minimizes cross-entropy between the teacher's probability distribution for one view and the student's distribution for another view.
The teacher is updated with an EMA of the student and receives no gradients.
Centering and sharpening act in opposite directions and help prevent representation collapse in the presence of the momentum teacher.
DINO does not require negative pairs, a contrastive loss, a prediction head, a memory queue, or batch normalization when used with ViT.
A frozen DINO ViT-S/8 reaches 78.3% ImageNet top-1 accuracy with weighted k-NN. This is not zero-shot classification: the k-NN classifier uses labeled ImageNet training examples.
DINO ViT-B/8 reaches 80.1% ImageNet top-1 accuracy in linear evaluation.
Smaller ViT patches improve the reported accuracy without increasing parameter count, but they substantially reduce throughput and increase compute and memory requirements.
ViT attention maps provide evidence of emergent object localization. They are not explicit semantic segmentation predictions and do not assign semantic classes to pixels.

Figure 1. Selected attention heads often focus on foreground objects and align with object boundaries. These are attention maps, not semantic segmentation masks. (Source)

What problem does DINO address?🔗

Supervised image classification compresses the training signal for an image into one label from a fixed vocabulary. Self-supervised learning instead constructs a learning objective from the data itself. The hope is that a model trained without class labels will retain visual information useful across many downstream tasks.

The paper asks what happens when modern self-supervised methods are applied to ViTs. Its central contribution is therefore not a claim that DINO universally replaces supervised ViT training. It is an empirical study showing that:

self-supervised ViTs can learn image representations that transfer well;
their frozen features work unusually well with a simple k-NN classifier; and
their final-layer class-token attention often exposes object layout and boundaries.

The authors also simplify the training recipe. DINO combines a momentum teacher, multi-crop augmentation, cross-view prediction, centering, and sharpening. Unlike many contrastive methods, it does not compare an image against explicit negative examples.

Method🔗

Multiple views of one image🔗

For each training image, DINO creates a set of augmented views:

two global crops, typically at resolution $224 \times 224$; and
several local crops, typically at resolution $96 \times 96$.

The original implementation uses eight local crops by default. Color jitter, grayscale conversion, Gaussian blur, solarization, and horizontal flipping further alter the views.

This is not a reconstruction objective. Neither network is asked to reproduce the original pixels. Instead, the student must produce a similar high-level output for different crops of the same source image, including local crops that contain only part of the scene.

Student and teacher🔗

Let $g_{\theta_s}$ be the student and $g_{\theta_t}$ the teacher. They have the same backbone and projection-head architecture, but their parameters differ.

The student receives every global and local view. The teacher receives only the two global views. For an input view $x$, the networks produce probability distributions over $K$ learned output dimensions:

$$ P_s(x)^{(i)} = \frac{\exp(g_{\theta_s}(x)^{(i)} / \tau_s)} {\sum_{k=1}^{K} \exp(g_{\theta_s}(x)^{(k)} / \tau_s)} $$

and

$$ P_t(x)^{(i)} = \frac{\exp((g_{\theta_t}(x)^{(i)} - c^{(i)}) / \tau_t)} {\sum_{k=1}^{K} \exp((g_{\theta_t}(x)^{(k)} - c^{(k)}) / \tau_t)}. $$

Here, $c$ is the teacher-output center, while $\tau_s$ and $\tau_t$ are the student and teacher temperatures. These $K$ dimensions are learned targets, not human-defined classes.

For each teacher global view $x$, the student is trained on every other view $x'$ of the same image:

$$ L = \sum_{x \in \{x_1^g, x_2^g\}} \sum_{\substack{x' \in V \\ x' \ne x}} H(P_t(x), P_s(x')), $$

where $H(a,b)=-\sum_i a^{(i)}\log b^{(i)}$ is cross-entropy and $V$ contains all crops. The identical student-teacher view pair is excluded. Gradients flow only through the student.

Momentum teacher🔗

After each student update, the teacher parameters are updated as:

$$ \theta_t \leftarrow \lambda \theta_t + (1-\lambda)\theta_s. $$

The momentum coefficient $\lambda$ follows a cosine schedule from 0.996 toward 1. This slowly changing teacher provides more stable targets than using the current student directly. In the paper's ViT-S/16 ablation, removing the momentum teacher causes collapse and yields only 0.1% k-NN and linear accuracy.

Projection head and downstream features🔗

The projection head is a three-layer multilayer perceptron with a 2048-dimensional hidden layer, followed by $\ell_2$ normalization and a weight-normalized fully connected layer. The self-supervised loss is applied to this head.

For downstream evaluation, the authors discard the projection head and use the frozen backbone representation. For ViTs, this representation is based on the class token.

To view this video, enable JavaScript or use a browser that supports HTML5 video.

Figure 2. DINO training overview. The student sees global and local crops; the EMA teacher sees global crops. (Source)

Avoiding collapse🔗

What is representation collapse?🔗

A collapsed network maps many or all inputs to the same output. Such a constant representation can make two augmented views agree perfectly while carrying no useful information about the image.

Contrastive approaches discourage this solution using negative examples. DINO instead combines a momentum teacher with two operations on the teacher output:

Centering subtracts an EMA of the batch-average teacher logits. It prevents one output dimension from dominating, but by itself tends toward a uniform output distribution.
Sharpening uses a low teacher temperature to make the teacher distribution more selective. It counters uniform collapse, but by itself can encourage domination by a small number of dimensions.

These effects balance each other. The paper is careful about the scope of this conclusion: centering and sharpening avoid collapse when used with the momentum teacher. They are not demonstrated as a universal collapse-prevention recipe for arbitrary architectures.

The default student temperature is 0.1. For the teacher, the temperature is warmed from 0.04 to 0.07 during the first 30 epochs. The center is also updated by an EMA of teacher outputs.

What emerges in a DINO-trained ViT?🔗

Strong frozen features🔗

The authors evaluate frozen representations in two main ways:

Linear evaluation: train a supervised linear classifier on top of the frozen features.
Weighted k-NN evaluation: store frozen features from the labeled ImageNet training set and classify each validation image by a weighted vote among its nearest stored features.

The second protocol is simple, but it is still supervised at evaluation time because it uses labels from the ImageNet training set. Calling it zero-shot would be incorrect.

Backbone	Patch size	k-NN top-1	Linear top-1
ResNet-50	-	67.5%	75.3%
ViT-S	$16 \times 16$	74.5%	77.0%
ViT-S	$8 \times 8$	78.3%	79.7%
ViT-B	$16 \times 16$	76.1%	78.2%
ViT-B	$8 \times 8$	77.4%	80.1%

The paper generally uses $k=20$ for weighted k-NN. On the same ViT-S/16 architecture, DINO reaches 77.0% linear accuracy, 3.5 percentage points above the reported SwAV result. Its 74.5% k-NN accuracy is 7.9 points above the best listed BYOL result.

Smaller patches: better spatial detail, higher cost🔗

Reducing the patch size gives the transformer more spatial tokens and improves the reported representation quality. It does not add learned parameters, but self-attention becomes more expensive as the number of tokens grows.

The paper reports throughput of roughly 1007 images/s for ViT-S/16, 180 images/s for ViT-S/8, and 44 images/s for ViT-S/5 under its measurement setup. Smaller patches therefore increase, rather than reduce, running time and memory pressure.

Emergent object localization🔗

In a ViT, the final class token attends to patch tokens. DINO's final-layer attention heads often focus on coherent foreground regions. To quantify this, the authors threshold attention maps to retain 60% of their mass and compare them with object masks from PASCAL VOC 2012.

DINO ViT-S/8 reaches 44.7% Jaccard similarity, compared with 23.7% for supervised ViT-S/8 and 21.8% for a randomly initialized model. This is evidence that object layout emerges in the representation without segmentation labels.

There are important limits to the interpretation:

Attention maps are not semantic segmentation outputs.
They do not assign a category to every pixel.
Different attention heads may focus on different objects or parts.
The evaluation measures overlap with foreground masks after thresholding; it does not establish a complete segmentation system.

Video object segmentation🔗

The paper also evaluates frozen patch features on DAVIS 2017. Labels are propagated between frames using nearest-neighbor matching; the backbone itself is not fine-tuned for video segmentation. DINO ViT-S/8 obtains 69.9 mean $J\&F$, compared with 66.0 for supervised ViT-S/8. DINO ViT-B/8 reaches 71.4.

This experiment shows that local patch features encode useful correspondence. It does not mean that raw attention maps alone solve video instance segmentation.

Ablation study🔗

The ViT-S/16 ablation after 300 epochs clarifies which parts of DINO matter:

Configuration	k-NN top-1	Linear top-1
Default DINO	72.8%	76.1%
Without momentum teacher	0.1%	0.1%
Without multi-crop	67.9%	72.5%
MSE instead of cross-entropy	52.6%	62.4%
With an added predictor	71.8%	75.6%

The momentum teacher is essential in this setup, and multi-crop training provides a large improvement. Cross-entropy works substantially better than mean squared error. Unlike BYOL, DINO does not benefit from a separate predictor in this experiment.

What DINO does not establish🔗

The paper provides strong evidence for its tested architectures and benchmarks, but some broader conclusions would go beyond the experiments:

DINO is not a universally superior replacement for supervised learning.
Its transfer results are strong but not uniformly better on every dataset. For example, supervised ViT-B/16 slightly exceeds DINO on iNaturalist 2018 in the paper's transfer table.
The learned output dimensions are not automatically named semantic categories.
k-NN evaluation is not zero-shot learning.
Attention visualization is evidence of localization, not a production-ready segmentation method.
The method still requires substantial compute, careful augmentation, and large-scale image data.

Applications🔗

DINO representations are useful when labels are scarce or when a reusable visual encoder is needed. The paper demonstrates or motivates:

image classification with linear probing, k-NN, or fine-tuning;
image retrieval and copy detection;
transfer to classification datasets;
object discovery through attention analysis; and
video object segmentation through frozen-feature correspondence.

Relationship to other methods🔗

DINO belongs to a family of methods that learn by matching representations across image views.

SimCLR uses a contrastive objective with in-batch negatives.
MoCo uses a momentum encoder and a queue of negative examples.
BYOL matches an online network to a momentum target network without negatives and uses a predictor.
SwAV predicts cluster assignments between views and uses online clustering.
DINO matches centered and sharpened teacher distributions using cross-entropy, a momentum teacher, and multi-crop views, without negatives or a predictor.

DINO can train ResNet-50 as well as ViTs. The paper's distinctive observations, however, are the strong k-NN behavior and emergent spatial structure of self-supervised ViT features.

Useful resources🔗

Terminology🔗

Knowledge distillation🔗

Knowledge distillation trains a student model to match outputs produced by a teacher. In conventional distillation, the teacher is often a larger, already trained network and the goal may be model compression.

DINO uses self-distillation differently: teacher and student have the same architecture, and the teacher is created continuously as an EMA of the student. There is no separately pretrained teacher and no class-label supervision. The term describes the direction of the learning target, not compression into a smaller model.

Representation collapse🔗

Representation collapse occurs when a network maps different inputs to the same or nearly the same output. Agreement between views then becomes trivial but the features cease to distinguish images. DINO avoids this outcome through the combined training dynamics of its stop-gradient momentum teacher, centering, and sharpening.