Comparative Evaluation of Corruption Robustness and Confidence Calibration in Real-Time Object Detectors for UAV Imagery
Keywords:
unmanned aerial vehicle, object detection, corruption robustness, confidence calibration, small object detectionAbstract
Detectors for unmanned aerial vehicles are usually selected on clean-image average precision, yet an airborne camera delivers blurred, noisy, hazy and compressed frames. Four detectors, three lightweight convolutional designs and one real-time transformer, were trained under a matched data exposure and epoch budget on the UAVDT vehicle benchmark and evaluated on temporally subsampled frames of the official test sequences under a COCO-style protocol: clean, six controlled corruptions at five severities, and across the weather, altitude and camera-view attributes of the benchmark, with all partitions formed from whole video sequences and all uncertainty from a sequence-level cluster bootstrap. The transformer reference reached 16.8 per cent average precision against 11.1 to 12.7 per cent for the lightweight detectors, among which no significant difference was detected. The detectors retained 65.3 to 71.4 per cent of their clean accuracy across the corruption benchmark, so relative robustness separated them far less than absolute accuracy did, while small objects lost about 40.6 per cent against about 16.7 per cent for large objects. Contrary to the familiar overconfidence of image classifiers, the three lightweight detectors were underconfident and the gap widened with severity, while the transformer reference was close to calibrated on clean images. A calibrator fitted only on clean held-out sequences reduced the detection expected calibration error by 67 per cent on clean images and by 42 per cent across the corruption benchmark, without changing average precision. All four detectors are non-dominated once accuracy, robustness, calibration, latency and size are considered jointly.
References
[1] L. P. Osco et al., "A review on deep learning in UAV remote sensing," Int. J. Appl. Earth Obs. Geoinf., vol. 102, Art. no. 102456, Oct. 2021, doi: 10.1016/j.jag.2021.102456.
[2] P. Mittal, R. Singh and A. Sharma, "Deep learning-based object detection in low-altitude UAV datasets: A survey," Image Vis. Comput., vol. 104, Art. no. 104046, Dec. 2020, doi: 10.1016/j.imavis.2020.104046.
[3] X. Zhu, S. Lyu, X. Wang and Q. Zhao, "TPH-YOLOv5: Improved YOLOv5 based on transformer prediction head for object detection on drone-captured scenarios," in Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshops (ICCVW), Montreal, QC, Canada, 2021, pp. 2778–2788, doi: 10.1109/ICCVW54120.2021.00312.
[4] D. Du et al., "The unmanned aerial vehicle benchmark: Object detection and tracking," in Proc. Eur. Conf. Comput. Vis. (ECCV), Munich, Germany, 2018, pp. 375–391, doi: 10.1007/978-3-030-01249-6_23.
[5] P. Zhu et al., "Detection and tracking meet drones challenge," IEEE Trans. Pattern Anal. Mach. Intell., vol. 44, no. 11, pp. 7380–7399, Nov. 2022, doi: 10.1109/TPAMI.2021.3119563.
[6] G. Cheng et al., "Towards large-scale small object detection: Survey and benchmarks," IEEE Trans. Pattern Anal. Mach. Intell., vol. 45, no. 11, pp. 13467–13488, 2023, doi: 10.1109/TPAMI.2023.3290594.
[7] M. Kisantal, Z. Wojna, J. Murawski, J. Naruniec and K. Cho, "Augmentation for small object detection," in Proc. 9th Int. Conf. Adv. Comput. Inf. Technol. (ACITY), Sydney, Australia, 2019, pp. 119–133, doi: 10.5121/csit.2019.91713.
[8] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You only look once: Unified, real-time object detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Las Vegas, NV, USA, 2016, pp. 779–788, doi: 10.1109/CVPR.2016.91.
[9] W. Liu et al., "SSD: Single shot MultiBox detector," in Proc. Eur. Conf. Comput. Vis. (ECCV), Amsterdam, The Netherlands, 2016, pp. 21–37, doi: 10.1007/978-3-319-46448-0_2.
[10] Z. Tian, C. Shen, H. Chen and T. He, "FCOS: Fully convolutional one-stage object detection," in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Seoul, South Korea, 2019, pp. 9626–9635, doi: 10.1109/ICCV.2019.00972.
[11] T.-Y. Lin, P. Goyal, R. Girshick, K. He and P. Dollár, "Focal loss for dense object detection," in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), Venice, Italy, 2017, pp. 2999–3007, doi: 10.1109/ICCV.2017.324.
[12] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan and S. Belongie, "Feature pyramid networks for object detection," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Honolulu, HI, USA, 2017, pp. 936–944, doi: 10.1109/CVPR.2017.106.
[13] C.-Y. Wang, A. Bochkovskiy and H.-Y. M. Liao, "YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Vancouver, BC, Canada, 2023, pp. 7464–7475, doi: 10.1109/CVPR52729.2023.00721.
[14] Z. Ge, S. Liu, F. Wang, Z. Li and J. Sun, "YOLOX: Exceeding YOLO series in 2021," arXiv:2107.08430, 2021, doi: 10.48550/arXiv.2107.08430.
[15] A. Wang et al., "YOLOv10: Real-time end-to-end object detection," in Adv. Neural Inf. Process. Syst. (NeurIPS), Vancouver, BC, Canada, 2024, pp. 107984–108011, doi: 10.52202/079017-3429.
[16] R. Khanam and M. Hussain, "YOLOv11: An overview of the key architectural enhancements," arXiv:2410.17725, 2024, doi: 10.48550/arXiv.2410.17725.
[17] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov and S. Zagoruyko, "End-to-end object detection with transformers," in Proc. Eur. Conf. Comput. Vis. (ECCV), Glasgow, U.K., 2020, pp. 213–229, doi: 10.1007/978-3-030-58452-8_13.
[18] X. Zhu, W. Su, L. Lu, B. Li, X. Wang and J. Dai, "Deformable DETR: Deformable transformers for end-to-end object detection," in Proc. Int. Conf. Learn. Represent. (ICLR), 2021.
[19] Y. Zhao et al., "DETRs beat YOLOs on real-time object detection," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Seattle, WA, USA, 2024, pp. 16965–16974, doi: 10.1109/CVPR52733.2024.01605.
[20] D. Hendrycks and T. Dietterich, "Benchmarking neural network robustness to common corruptions and perturbations," in Proc. Int. Conf. Learn. Represent. (ICLR), New Orleans, LA, USA, 2019.
[21] C. Michaelis et al., "Benchmarking robustness in object detection: Autonomous driving when winter is coming," arXiv:1907.07484, 2019, doi: 10.48550/arXiv.1907.07484.
[22] Y. Dong et al., "Benchmarking robustness of 3D object detection to common corruptions in autonomous driving," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Vancouver, BC, Canada, 2023, pp. 1022–1032, doi: 10.1109/CVPR52729.2023.00105.
[23] S. Dodge and L. Karam, "Understanding how image quality affects deep neural networks," in Proc. 8th Int. Conf. Qual. Multimedia Exper. (QoMEX), Lisbon, Portugal, 2016, pp. 1–6, doi: 10.1109/QOMEX.2016.7498955.
[24] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann and W. Brendel, "ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness," in Proc. Int. Conf. Learn. Represent. (ICLR), New Orleans, LA, USA, 2019.
[25] C. Kamann and C. Rother, "Benchmarking the robustness of semantic segmentation models with respect to common corruptions," Int. J. Comput. Vis., vol. 129, no. 2, pp. 462–483, Feb. 2021, doi: 10.1007/s11263-020-01383-2.
[26] H. He, J. Ding, B. Xu and G.-S. Xia, "On the robustness of object detection models on aerial images," IEEE Trans. Geosci. Remote Sens., vol. 63, Art. no. 5600512, pp. 1–12, 2025, doi: 10.1109/TGRS.2024.3514741.
[27] A. Arsenos et al., "Common corruptions for evaluating and enhancing robustness in air-to-air visual object detection," IEEE Robot. Autom. Lett., vol. 9, no. 7, pp. 6688–6695, Jul. 2024, doi: 10.1109/LRA.2024.3408485.
[28] X. Liu, Y. Feng, S. Hu, X. Yuan and H. Fan, "Benchmarking the robustness of UAV tracking against common corruptions," in Proc. IEEE 7th Int. Conf. Multimedia Inf. Process. Retrieval (MIPR), San Jose, CA, USA, 2024, pp. 465–470, doi: 10.1109/MIPR62202.2024.00079.
[29] J. Liu et al., "Benchmarking object detection robustness against real-world corruptions," Int. J. Comput. Vis., vol. 132, no. 10, pp. 4398–4416, Oct. 2024, doi: 10.1007/s11263-024-02096-6.
[30] D. Feng, A. Harakeh, S. L. Waslander and K. Dietmayer, "A review and comparative study on probabilistic object detection in autonomous driving," IEEE Trans. Intell. Transp. Syst., vol. 23, no. 8, pp. 9961–9980, Aug. 2022, doi: 10.1109/TITS.2021.3096854.
[31] C. Guo, G. Pleiss, Y. Sun and K. Q. Weinberger, "On calibration of modern neural networks," in Proc. 34th Int. Conf. Mach. Learn. (ICML), Sydney, Australia, 2017, pp. 1321–1330.
[32] F. Kuppers, J. Kronenberger, A. Shantia and A. Haselhoff, "Multivariate confidence calibration for object detection," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. Workshops (CVPRW), Seattle, WA, USA, 2020, pp. 1322–1330, doi: 10.1109/CVPRW50498.2020.00171.
[33] M. P. Naeini, G. Cooper and M. Hauskrecht, "Obtaining well calibrated probabilities using Bayesian binning," in Proc. AAAI Conf. Artif. Intell., Austin, TX, USA, 2015, pp. 2901–2907, doi: 10.1609/aaai.v29i1.9602.
[34] K. Oksuz, T. Joy and P. K. Dokania, "Towards building self-aware object detectors via reliable uncertainty quantification and calibration," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Vancouver, BC, Canada, 2023, pp. 9263–9274, doi: 10.1109/CVPR52729.2023.00894.
[35] B. Pathiraja, M. Gunawardhana and M. H. Khan, "Multiclass confidence and localization calibration for object detection," in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Vancouver, BC, Canada, 2023, pp. 19734–19743, doi: 10.1109/CVPR52729.2023.01890.
[36] T. Popordanoska, A. Tiulpin and M. B. Blaschko, "Beyond classification: Definition and density-based estimation of calibration in object detection," in Proc. IEEE/CVF Winter Conf. Appl. Comput. Vis. (WACV), Waikoloa, HI, USA, 2024, pp. 574–583, doi: 10.1109/WACV57701.2024.00064.
[37] S. Kuzucu, K. Oksuz, J. Sadeghi and P. K. Dokania, "On calibration of object detectors: Pitfalls, evaluation and baselines," in Proc. Eur. Conf. Comput. Vis. (ECCV), Milan, Italy, 2024, pp. 185–204, doi: 10.1007/978-3-031-72664-4_11.
[38] D. Huseljic, M. Herde, P. Hahn, M. Müjde and B. Sick, "Systematic evaluation of uncertainty calibration in pretrained object detectors," Int. J. Comput. Vis., vol. 133, no. 3, pp. 1033–1047, Mar. 2025, doi: 10.1007/s11263-024-02219-z.
[39] B. Zadrozny and C. Elkan, "Transforming classifier scores into accurate multiclass probability estimates," in Proc. 8th ACM SIGKDD Int. Conf. Knowl. Discovery Data Mining, Edmonton, AB, Canada, 2002, pp. 694–699, doi: 10.1145/775047.775151.
[40] A. Niculescu-Mizil and R. Caruana, "Predicting good probabilities with supervised learning," in Proc. 22nd Int. Conf. Mach. Learn. (ICML), Bonn, Germany, 2005, pp. 625–632, doi: 10.1145/1102351.1102430.
[41] T.-Y. Lin et al., "Microsoft COCO: Common objects in context," in Proc. Eur. Conf. Comput. Vis. (ECCV), Zurich, Switzerland, 2014, pp. 740–755, doi: 10.1007/978-3-319-10602-1_48.
[42] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn and A. Zisserman, "The Pascal visual object classes (VOC) challenge," Int. J. Comput. Vis., vol. 88, no. 2, pp. 303–338, Jun. 2010, doi: 10.1007/s11263-009-0275-4.
[43] G. W. Brier, "Verification of forecasts expressed in terms of probability," Mon. Weather Rev., vol. 78, no. 1, pp. 1–3, Jan. 1950, doi: 10.1175/1520-0493(1950)078<0001:VOFEIT>2.0.CO;2.
[44] H. R. Künsch, "The jackknife and the bootstrap for general stationary observations," Ann. Statist., vol. 17, no. 3, pp. 1217–1241, Sep. 1989, doi: 10.1214/aos/1176347265.
[45] B. Efron, "Bootstrap methods: Another look at the jackknife," Ann. Statist., vol. 7, no. 1, pp. 1–26, Jan. 1979, doi: 10.1214/aos/1176344552.
[46] M. Friedman, "The use of ranks to avoid the assumption of normality implicit in the analysis of variance," J. Amer. Statist. Assoc., vol. 32, no. 200, pp. 675–701, Dec. 1937, doi: 10.1080/01621459.1937.10503522.
[47] F. Wilcoxon, "Individual comparisons by ranking methods," Biometrics Bull., vol. 1, no. 6, pp. 80–83, Dec. 1945, doi: 10.2307/3001968.
[48] S. Holm, "A simple sequentially rejective multiple test procedure," Scand. J. Statist., vol. 6, no. 2, pp. 65–70, 1979.
[49] J. Demšar, "Statistical comparisons of classifiers over multiple data sets," J. Mach. Learn. Res., vol. 7, pp. 1–30, 2006.
