ICLR 2027 submission / Research project

Across Viewpoints
and Time.

Trajectory-Generalizable Physical Attacks on
World Models and Video Understanding Models

Song Xia1 · Meiewen Ding1 · Chenqi Kong2 · Wenhan Yang3 · Xudong Jiang1

1 Nanyang Technological University2 National University of Singapore3 Pengcheng Lab

Read the paper Explore the trajectoriesCitation

One texture stays fixed on an object.
Its viewpoint, scale, and visibility keep changing.

TARA learns a shared printable texture through trajectory-aware contrastive alignment and task refinement, exposing vulnerabilities in video understanding and prediction under dynamic physical observation.

A single printed adversarial texture observed along three EK100 trajectories, with occlusion, deformation, scale changes, and re-entry.
A single surface-attached texture, observed across changing object–camera trajectories. Figure 1 from the submitted manuscript.
EK100 · anticipation41.1%Average validation Top-1 ASR+5.4 pp over EOT
SSV2 · recognition26.0%Average validation Top-1 ASR+5.8 pp over EOT
Physical · anticipation32.1%Print-and-recapture ASR · V-JEPA2-L+3.5 pp over printed EOT

01 / PHYSICAL EVALUATION

From rendering to print and capture.

Printed textures are attached to real objects and observed under natural motion and imaging conditions.

V-JEPA2-L · printed textures

Action anticipation

EOT28.6%
TARA32.1%
V-JEPA2-L · printed textures

Action recognition

EOT23.1%
TARA28.2%
Real-world captured data · Top-1 ASR (%) ↑
MethodAction anticipationAction recognition
V-JEPA2-LV-JEPA2.1-BV-JEPA2-LInternVideo2
Digital projection along recorded trajectories
EOT29.758.821.240.3
Patch-Fool17.623.510.635.5
UPA-RFAS18.917.611.843.5
BTC-UAP14.917.612.924.2
PhyCamo13.529.417.638.7
TARA32.458.825.850.0
Physical print and recapture
Printed EOT28.6—23.1—
Printed TARA32.1—28.2—

Paper Table 5. Digital projection uses recorded homographies and masks; physical evaluation uses printed textures and matched white-patch recordings. The two protocols are reported separately; — denotes an unavailable result.

Watch the real-world videos
TARA attack success under increasing simulated print-and-capture distortion levels, for EK100 anticipation and SSV2 recognition.
Simulated print-and-capture distortions. Brightness changes, Gaussian noise, and JPEG compression preserve relatively stable ASR, while stronger patch and capture blur suppress fine texture detail.

02 / SEE THE TRAJECTORIES

A fixed patch, seen in motion.

Start with real-world print-and-capture videos, then explore the rendered benchmarks.

Examples 1–2 of 10

Each video shows the same 32 sampled model-input frames as its contact sheet, played at 8 fps for inspection. Digital examples use rendered TARA textures; real-world examples show physically printed textures. Anticipation labels refer to the future action.

03 / THE METHOD

Align first. Refine for the task.

Heterogeneous videos and trajectories produce conflicting texture gradients. TARA builds a more compatible starting point for shared optimization.

01

Render along a trajectory

Attach one canonical texture to a surface. Frame-wise homographies and visibility masks account for motion, perspective, and occlusion.

γi = {(Hit, Mit)}t=1T
02

Align attacked representations

Bring attacked features from different video–trajectory instances toward a shared region, while separating them from their own clean features.

Lcon = meani≠j softplus[(s(zia, zic) − s(zia, zja)) / τ]
03

Refine toward prediction errors

Optimize the model's aggregated task prediction from the aligned texture. Visibility-aware weights balance different levels of texture exposure.

Lref = Σi wi log pθ(yi | X̃i(δ; γi))
TARA framework: black texture initialization, trajectory-aware contrastive alignment, then trajectory-conditioned task refinement.
The two-stage TARA framework. For future frame generation, refinement instead targets reconstruction degradation.

04 / RESULTS

Three tasks. Six video models.

Results transcribed from the submitted paper. Switch tasks and splits to inspect the comparisons.

Top-1 ASR (%) is computed on initially clean-correct examples. “Avg.” is the unweighted mean across models.

Generalization beyond training observations

Frozen textures under independent changes in content and observation trajectory.

TARAEOT

Top-1 ASR on 74 eligible clip instances per condition (Appendix Table 7). Unseen-content instances come from 35 independently captured recordings; frozen textures receive no further adaptation.

05 / THE BENCHMARK

Grounded in surfaces and trajectories.

Temporally consistent geometry makes texture comparisons reproducible under the same observation conditions.

Geometry-grounded trajectory benchmark statistics from paper Table 1
DatasetTaskClipsTrain / ValAnnotated framesHomography coverage
EK100Anticipation605356 / 24919,25173.94%
SSV2Recognition328184 / 14410,20489.78%
Real-worldRecognition / Anticipation3480 / 3488,64581.27%

Dataset-level statistics from paper Table 1. Model-specific attack cohorts, temporal inputs, and geometric validity rules are detailed in the appendix.

Human-guided surface trajectory construction: select an attachment region, estimate homography, estimate visibility, and manually review uncertain projections.
Human review anchors the geometry pipeline: select a feasible surface, track its projection, estimate visible support, and inspect drift or inconsistent occlusion.
Read the abstract

World models and video understanding models are increasingly deployed in dynamic visual environments, yet their robustness to physical adversarial threats remains largely underexplored. We study whether a single physical adversarial texture can mislead the victim model across different real-world scenarios where observing viewpoints vary along uncontrolled object–camera trajectories. We identify that heterogeneous video semantics and trajectories induce poorly aligned gradients for optimizing the adversarial texture. To address that, we propose the Trajectory-aware adversarial Alignment and Refinement Attack (TARA), which incorporates trajectory-aware optimization to capture the evolving geometric variations of a printed texture under real-world scenarios. To improve cross-instance optimization compatibility, TARA incorporates trajectory-aware contrastive alignment to encourage attacked representations from diverse video–trajectory instances to approach a shared adversarial region and then performs trajectory-aware task refinement to translate the aligned representation into task-level prediction errors. For reproducible evaluation, we develop a geometry-grounded tracking pipeline that provides physically plausible adversarial regions with temporally consistent trajectories on EPIC-KITCHENS-100 and Something-Something V2. We further collect real-world print-and-recapture videos to validate attack effectiveness under physical deployment conditions. Together, the annotated benchmarks and real-world captured data form a reproducible testbed for future evaluation. Notably, our results reveal that a single printable adversarial texture can persistently disrupt world models and video understanding models under physical conditions, highlighting the fundamental security challenge.

REFERENCE

Citation

@misc{xia2026tara,
  title = {Across Viewpoints and Time: Trajectory-Generalizable Physical
           Attacks on World Models and Video Understanding Models},
  author = {Xia, Song and Ding, Meiewen and Kong, Chenqi
            and Yang, Wenhan and Jiang, Xudong},
  year = {2026},
  note = {Manuscript submitted to ICLR 2027},
  url = {https://xiasong0501.github.io/TARA/}
}