The detection of distributional discrepancy for language GANs
A pre-trained neural language model (LM) is usually used to generate texts. Due to exposure bias, the generated text is not as good as real text. Many researchers claimed they employed the Generative Adversarial Nets (GAN) to alleviate this issue by feeding reward signals from a discriminator to upd...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Taylor & Francis Group
2022-12-01
|
| Series: | Connection Science |
| Subjects: | |
| Online Access: | http://dx.doi.org/10.1080/09540091.2022.2080182 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850160110014300160 |
|---|---|
| author | Xingyuan Chen Peng Jin Ping Cai Hongjun Wang Xinyu Dai Jiajun Chen |
| author_facet | Xingyuan Chen Peng Jin Ping Cai Hongjun Wang Xinyu Dai Jiajun Chen |
| author_sort | Xingyuan Chen |
| collection | DOAJ |
| description | A pre-trained neural language model (LM) is usually used to generate texts. Due to exposure bias, the generated text is not as good as real text. Many researchers claimed they employed the Generative Adversarial Nets (GAN) to alleviate this issue by feeding reward signals from a discriminator to update the LM (generator). However, some researchers argued that GAN did not work by evaluating the generated texts with a quality-diversity metric such as Bleu versus self-Bleu, and language model score versus reverse language model score. Unfortunately, these two-dimension metrics are not reliable. Furthermore, the existing methods only assessed the final generated texts, thus neglecting the dynamic evaluating the adversarial learning process. Different from the above-mentioned methods, we adopted the most recent metric functions, which measure the distributional discrepancy between real and generated text. Besides that, we design a comprehensive experiment to investigate the performance during the learning process. First, we evaluate a language model with two functions and identify a large discrepancy. Then, several methods with the detected discrepancy signal to improve the generator were tried. Experimenting with two language GANs on two benchmark datasets, we found that the distributional discrepancy increases with more adversarial learning rounds. Our research provides convicted evidence that the language GANs fail. |
| format | Article |
| id | doaj-art-b1041dc081324d6bb55d84e7e51c2f52 |
| institution | OA Journals |
| issn | 0954-0091 1360-0494 |
| language | English |
| publishDate | 2022-12-01 |
| publisher | Taylor & Francis Group |
| record_format | Article |
| series | Connection Science |
| spelling | doaj-art-b1041dc081324d6bb55d84e7e51c2f522025-08-20T02:23:15ZengTaylor & Francis GroupConnection Science0954-00911360-04942022-12-013411736175010.1080/09540091.2022.20801822080182The detection of distributional discrepancy for language GANsXingyuan Chen0Peng Jin1Ping Cai2Hongjun Wang3Xinyu Dai4Jiajun Chen5Nanjing UniversitySchool of Electronic Engineering and Artificial Intelligence, Leshan Normal UniversitySchool of Information Science and Technology, Southwest Jiaotong UniversitySchool of Information Science and Technology, Southwest Jiaotong UniversityNanjing UniversityNanjing UniversityA pre-trained neural language model (LM) is usually used to generate texts. Due to exposure bias, the generated text is not as good as real text. Many researchers claimed they employed the Generative Adversarial Nets (GAN) to alleviate this issue by feeding reward signals from a discriminator to update the LM (generator). However, some researchers argued that GAN did not work by evaluating the generated texts with a quality-diversity metric such as Bleu versus self-Bleu, and language model score versus reverse language model score. Unfortunately, these two-dimension metrics are not reliable. Furthermore, the existing methods only assessed the final generated texts, thus neglecting the dynamic evaluating the adversarial learning process. Different from the above-mentioned methods, we adopted the most recent metric functions, which measure the distributional discrepancy between real and generated text. Besides that, we design a comprehensive experiment to investigate the performance during the learning process. First, we evaluate a language model with two functions and identify a large discrepancy. Then, several methods with the detected discrepancy signal to improve the generator were tried. Experimenting with two language GANs on two benchmark datasets, we found that the distributional discrepancy increases with more adversarial learning rounds. Our research provides convicted evidence that the language GANs fail.http://dx.doi.org/10.1080/09540091.2022.2080182text generationgenerative adversarial netsdistributional discrepancy |
| spellingShingle | Xingyuan Chen Peng Jin Ping Cai Hongjun Wang Xinyu Dai Jiajun Chen The detection of distributional discrepancy for language GANs Connection Science text generation generative adversarial nets distributional discrepancy |
| title | The detection of distributional discrepancy for language GANs |
| title_full | The detection of distributional discrepancy for language GANs |
| title_fullStr | The detection of distributional discrepancy for language GANs |
| title_full_unstemmed | The detection of distributional discrepancy for language GANs |
| title_short | The detection of distributional discrepancy for language GANs |
| title_sort | detection of distributional discrepancy for language gans |
| topic | text generation generative adversarial nets distributional discrepancy |
| url | http://dx.doi.org/10.1080/09540091.2022.2080182 |
| work_keys_str_mv | AT xingyuanchen thedetectionofdistributionaldiscrepancyforlanguagegans AT pengjin thedetectionofdistributionaldiscrepancyforlanguagegans AT pingcai thedetectionofdistributionaldiscrepancyforlanguagegans AT hongjunwang thedetectionofdistributionaldiscrepancyforlanguagegans AT xinyudai thedetectionofdistributionaldiscrepancyforlanguagegans AT jiajunchen thedetectionofdistributionaldiscrepancyforlanguagegans AT xingyuanchen detectionofdistributionaldiscrepancyforlanguagegans AT pengjin detectionofdistributionaldiscrepancyforlanguagegans AT pingcai detectionofdistributionaldiscrepancyforlanguagegans AT hongjunwang detectionofdistributionaldiscrepancyforlanguagegans AT xinyudai detectionofdistributionaldiscrepancyforlanguagegans AT jiajunchen detectionofdistributionaldiscrepancyforlanguagegans |