The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. ex. Some numerals are expressed as "XNUMX".
Copyrights notice
The original paper is in English. Non-English content has been machine-translated and may contain typographical errors or mistranslations. Copyrights notice
시스템 로그는 대규모 네트워크의 상태를 이해하고 오류를 감지하는 데 유용합니다. 그러나 이러한 로그의 다양성과 양으로 인해 로그 분석에는 많은 시간과 노력이 필요합니다. 본 논문에서는 전처리 및 특징 추출이 필요 없는 대규모 네트워크에 대한 로그 이벤트 이상 탐지 방법을 제안한다. 핵심 아이디어는 잠재 변수를 사용하여 대량의 다양한 데이터를 숨겨진 상태에 삽입하는 것입니다. 우리는 일본의 전국 학술 네트워크에서 얻은 12개월 간의 시스템 로그를 사용하여 방법을 평가합니다. Kleinberg의 단변량 버스트 검출과 전통적인 다변량 분석(즉, PCA)과의 비교를 통해 제안한 방법이 PCA보다 14.5% 더 높은 재현율과 3% 더 높은 정밀도를 달성한다는 것을 입증합니다. 사례 연구에서는 감지된 이상 현상이 네트워크 시스템 오류 문제를 해결하는 데 효과적인 정보임을 보여줍니다.
Kazuki OTOMO
the University of Tokyo
Satoru KOBAYASHI
National Institute of Informatics
Kensuke FUKUDA
National Institute of Informatics,Sokendai
Hiroshi ESAKI
the University of Tokyo
The copyright of the original papers published on this site belongs to IEICE. Unauthorized use of the original or translated papers is prohibited. See IEICE Provisions on Copyright for details.
부
Kazuki OTOMO, Satoru KOBAYASHI, Kensuke FUKUDA, Hiroshi ESAKI, "Latent Variable Based Anomaly Detection in Network System Logs" in IEICE TRANSACTIONS on Information,
vol. E102-D, no. 9, pp. 1644-1652, September 2019, doi: 10.1587/transinf.2018OFP0007.
Abstract: System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.
URL: https://global.ieice.org/en_transactions/information/10.1587/transinf.2018OFP0007/_p
부
@ARTICLE{e102-d_9_1644,
author={Kazuki OTOMO, Satoru KOBAYASHI, Kensuke FUKUDA, Hiroshi ESAKI, },
journal={IEICE TRANSACTIONS on Information},
title={Latent Variable Based Anomaly Detection in Network System Logs},
year={2019},
volume={E102-D},
number={9},
pages={1644-1652},
abstract={System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.},
keywords={},
doi={10.1587/transinf.2018OFP0007},
ISSN={1745-1361},
month={September},}
부
TY - JOUR
TI - Latent Variable Based Anomaly Detection in Network System Logs
T2 - IEICE TRANSACTIONS on Information
SP - 1644
EP - 1652
AU - Kazuki OTOMO
AU - Satoru KOBAYASHI
AU - Kensuke FUKUDA
AU - Hiroshi ESAKI
PY - 2019
DO - 10.1587/transinf.2018OFP0007
JO - IEICE TRANSACTIONS on Information
SN - 1745-1361
VL - E102-D
IS - 9
JA - IEICE TRANSACTIONS on Information
Y1 - September 2019
AB - System logs are useful to understand the status of and detect faults in large scale networks. However, due to their diversity and volume of these logs, log analysis requires much time and effort. In this paper, we propose a log event anomaly detection method for large-scale networks without pre-processing and feature extraction. The key idea is to embed a large amount of diverse data into hidden states by using latent variables. We evaluate our method with 12 months of system logs obtained from a nation-wide academic network in Japan. Through comparisons with Kleinberg's univariate burst detection and a traditional multivariate analysis (i.e., PCA), we demonstrate that our proposed method achieves 14.5% higher recall and 3% higher precision than PCA. A case study shows detected anomalies are effective information for troubleshooting of network system faults.
ER -