AAAI

Association for the Advancement of Artificial Intelligence

Home > Proceedings / Proceedings of the AAAI Conference on Artificial Intelligence, 36 > No. 7: AAAI-22 Technical Tracks 7

Towards a Rigorous Evaluation of Time-Series Anomaly Detection

Authors

Siwon Kim

Data Science and AI Laboratory, Seoul National University, Korea


Kukjin Choi

Data Science and AI Laboratory, Seoul National University, Korea DIT Center, Samsung Electronics, Korea


Hyun-Soo Choi

Department of CSE and Education Research Team for Medical Big-data Convergence, Kangwon National University, Korea Ziovision


Byunghan Lee

Department of Electronic and IT Media Engineering, Seoul National University of Science and Technology, Korea


Sungroh Yoon

Data Science and AI Laboratory, Seoul National University, Korea Department of ECE and Interdisciplinary Program in AI, Seoul National University, Korea AIIS, ASRI, and INMC, Seoul National University, Korea


Proceedings:

No. 7: AAAI-22 Technical Tracks 7

Volume

Issue:

Proceedings of the AAAI Conference on Artificial Intelligence, 36

Track:

AAAI Technical Track on Machine Learning II

Downloads:

Download PDF

Abstract:

In recent years, proposed studies on time-series anomaly detection (TAD) report high F1 scores on benchmark TAD datasets, giving the impression of clear improvements in TAD. However, most studies apply a peculiar evaluation protocol called point adjustment (PA) before scoring. In this paper, we theoretically and experimentally reveal that the PA protocol has a great possibility of overestimating the detection performance; even a random anomaly score can easily turn into a state-of-the-art TAD method. Therefore, the comparison of TAD methods after applying the PA protocol can lead to misguided rankings. Furthermore, we question the potential of existing TAD methods by showing that an untrained model obtains comparable detection performance to the existing methods even when PA is forbidden. Based on our findings, we propose a new baseline and an evaluation protocol. We expect that our study will help a rigorous evaluation of TAD and lead to further improvement in future researches.

DOI:

10.1609/aaai.v36i7.20680


AAAI

Proceedings of the AAAI Conference on Artificial Intelligence, 36