A topical VAEGAN-IHMM approach for automatic story segmentation

Jia Yu, Huiling Peng, Guoqiang Wang, Nianfeng Shi

Research output: Contribution to journalArticlepeer-review

Abstract

Feature representations with rich topic information can greatly improve the performance of story segmentation tasks. VAEGAN offers distinct advantages in feature learning by combining variational autoencoder (VAE) and generative adversarial network (GAN), which not only captures intricate data representations through VAE's probabilistic encoding and decoding mechanism but also enhances feature diversity and quality via GAN's adversarial training. To better learn topical domain representation, we used a topical classifier to supervise the training process of VAEGAN. Based on the learned feature, a segmentor splits the document into shorter ones with different topics. Hidden Markov model (HMM) is a popular approach for story segmentation, in which stories are viewed as instances of topics (hidden states). The number of states has to be set manually but it is often unknown in real scenarios. To solve this problem, we proposed an infinite HMM (IHMM) approach which utilized an HDP prior on transition matrices over countably infinite state spaces to automatically infer the state's number from the data. Given a running text, a Blocked Gibbis sampler labeled the states with topic classes. The position where the topic changes was a story boundary. Experimental results on the TDT2 corpus demonstrated that the proposed topical VAEGAN-IHMM approach was significantly better than the traditional HMM method in story segmentation tasks and achieved state-of-the-art performance.

Original languageEnglish
Pages (from-to)6608-6630
Number of pages23
JournalMathematical Biosciences and Engineering
Volume21
Issue number7
DOIs
Publication statusPublished - 2024

Keywords

  • generative adversarial network
  • HDP
  • hidden Markov model
  • story segmentation
  • variational autoencoder

Fingerprint

Dive into the research topics of 'A topical VAEGAN-IHMM approach for automatic story segmentation'. Together they form a unique fingerprint.

Cite this