Lost in Adaptation: Layer-Selective Recovery of Temporal Reasoning in Video-Language Models

Multimodal adaptation can erode temporal reasoning (TR) in video-language models (VLMs), leaving models able to perceive salient events yet unable to infer their temporal and causal structure. We introduce MERIT, a gradient-free framework that repairs this capability through layer-selective model merging. MERIT assigns each self-attention layer a VLM-dominant or LLM-dominant interpolation and uses the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to search the resulting combinatorial space under an objective that rewards TR gains while penalizing temporal perception (TP) degradation. Across three VLM families and five video benchmarks, MERIT consistently improves TR while preserving TP; recipes selected on a compact diagnostic set transfer to four unseen benchmarks, with relative gains of up to 27.8%. Interventional masking and frame-level attribution further show that the selected layers are functionally important for reasoning and that MERIT shifts decisions toward temporally distributed, causally relevant evidence. These results establish layer-selective merging as a practical post-hoc mechanism for repairing video temporal reasoning degraded during multimodal adaptation, without learning new parameters.