TY - GEN
T1 - An Extensible Parsing Pipeline for Unstructured Data Processing
AU - Jain, Shubham
AU - De Buitleir, Amy
AU - Fallon, Enda
N1 - Publisher Copyright:
© 2021 Global IT Research Institute (GiRI).
PY - 2021/2/7
Y1 - 2021/2/7
N2 - Network monitoring and diagnostics systems depict the running system's state and generate enormous amounts of unstructured data through log files, print statements, and other reports. It is not feasible to manually analyze all these files due to limited resources and the need to develop custom parsers to convert unstructured data into desirable file formats. Prior research focuses on rule-based and relationship-based parsing methods to parse unstructured data into structured file formats; these methods are labor-intensive and need large annotated datasets. This paper presents an unsupervised text processing pipeline that analyses such text files, removes extraneous information, identifies tabular components, and parses them into a structured file format. The proposed approach is resilient to changes in the data structure, does not require training data, and is domain-independent. We experiment and compare topic modeling and clustering approaches to verify the accuracy of the proposed technique. Our findings indicate that combining similarity and clustering algorithms to identify data components had better accuracy than topic modeling.
AB - Network monitoring and diagnostics systems depict the running system's state and generate enormous amounts of unstructured data through log files, print statements, and other reports. It is not feasible to manually analyze all these files due to limited resources and the need to develop custom parsers to convert unstructured data into desirable file formats. Prior research focuses on rule-based and relationship-based parsing methods to parse unstructured data into structured file formats; these methods are labor-intensive and need large annotated datasets. This paper presents an unsupervised text processing pipeline that analyses such text files, removes extraneous information, identifies tabular components, and parses them into a structured file format. The proposed approach is resilient to changes in the data structure, does not require training data, and is domain-independent. We experiment and compare topic modeling and clustering approaches to verify the accuracy of the proposed technique. Our findings indicate that combining similarity and clustering algorithms to identify data components had better accuracy than topic modeling.
KW - Clustering
KW - Information Extraction
KW - Topic Modeling
KW - Unsupervised Data Mining
UR - http://www.scopus.com/inward/record.url?scp=85102855258&partnerID=8YFLogxK
U2 - 10.23919/ICACT51234.2021.9370654
DO - 10.23919/ICACT51234.2021.9370654
M3 - Conference contribution
AN - SCOPUS:85102855258
SN - 9791188428069
T3 - International Conference on Advanced Communication Technology, ICACT
SP - 312
EP - 318
BT - 23rd International Conference on Advanced Communication Technology
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 23rd International Conference on Advanced Communication Technology, ICACT 2021
Y2 - 7 February 2021 through 10 February 2021
ER -