Article · Вестник КазАТК · August 31, 2026

A PARTIALLY CONTROLLED EVALUATION PROTOCOL FOR TRAJECTORY FORECASTING MODELS: RETHINKING COMPARABILITY IN DEEP LEARNING BENCHMARKS

Indira Bainazarova, Zhanat Rysbayevna Umarova, Assel Botasheva, Pernekul Kozhabekova, Yusmadi Jusoh

Abstract

The comparison of trajectory prediction models is complicated by the fact that architectural differences are coupled with variations in input representation and training procedures. This study introduces a partially controlled evaluation protocol in which datasets, temporal task configuration, evaluation metrics, and computational constraints are fixed, while preserving architecture-specific input representations and individualized model tuning. In contrast to conventional approaches, the analysis is conducted not only on aggregated metric values but also at the level of scene-wise error distributions. Experiments on the ETH/UCY, Argoverse 2, and Waymo Open Motion datasets demonstrate that differences between models in terms of mean ADE and FDE remain limited and are accompanied by comparable intra-model variability. Statistical testing reveals significant differences (Friedman test, p < 0.001); however, the effect size remains modest (Kendall’s W ≈ 0.17–0.19), indicating weak agreement in model rankings. The findings suggest that the interpretation of model comparisons is strongly dependent on the experimental setup and the level of metric aggregation. The proposed protocol enables a clearer separation between the influence of experimental conditions and intrinsic model properties, thereby complementing—rather than replacing—existing benchmarking practices.

Autonomous Vehicle Technology and SafetyTraffic Prediction and Management TechniquesTraffic and Road Safetytrajectory predictionbenchmarkingmodel comparison

A PARTIALLY CONTROLLED EVALUATION PROTOCOL FOR TRAJECTORY FORECASTING MODELS: RETHINKING COMPARABILITY IN DEEP LEARNING BENCHMARKS

Abstract

More by Zhanat Umarova