Student Researcher

Highlights in full

• Built an evaluation harness for text-conditioned video generation that scores physical plausibility separately from semantic fidelity, so a model that renders a beautiful scene with impossible motion loses points on exactly one axis instead of averaging the failure away.
• Designed a frame-sequence rubric for rigid-body motion: does a falling object accelerate rather than drift, does momentum carry through a collision, does an occluded object reappear on the trajectory it left on.
• Measured inter-rater agreement on a held-out subset before trusting a single aggregate score, because a rubric two people apply differently is a preference, not a measurement.

Prompt design

• Wrote minimal-pair prompts where exactly one clause changes (a ball rolls off the table against a ball is placed on the table) so a score gap isolates the causal claim rather than the scene.
• Mapped a prompt taxonomy across event ordering, counterfactual conditions, object permanence and multi-agent interaction, so a weakness lands in a named category instead of a general impression.

Scoring

• Held generation fixed across models: identical seeds, resolution, frame count and sampler settings, so a comparison measures the model and nothing around it.
• Automated frame extraction and per-frame annotation in PyTorch and Diffusers, storing per-clip artifacts so any score traces back to the frames that produced it.
• Scored trajectories against a simple analytic baseline rather than by eye, flagging a clip when observed acceleration departs from constant-gravity motion beyond a stated tolerance.

Findings

• Derived a failure taxonomy from the annotated clips: broken object permanence, non-conserved mass, contact that teleports, and physics that silently resets at a scene cut.
• Found counterfactual prompts degrade faster than descriptive ones at matched length, so apparent fluency on a benchmark prompt set overstates a model's grasp of causal structure.
• Reported the uncomfortable correlation: fluency and physical-plausibility scores track each other weakly, so a model ranked first on human preference can rank last on causality.