Broad learning system based on attention mechanism and tracking differentiator
Broad learning system (BLS) has advantages such as a simple model structure, high training efficiency, and easy interpretability. However, it also has drawbacks such as insufficient feature learning capability and unstable generalization performance. To alleviate these problems, broad learning syste...
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Science Press (China Science Publishing & Media Ltd.)
2024-09-01
|
| Series: | Shenzhen Daxue xuebao. Ligong ban |
| Subjects: | |
| Online Access: | https://journal.szu.edu.cn/en/#/digest?ArticleID=2668 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Summary: | Broad learning system (BLS) has advantages such as a simple model structure, high training efficiency, and easy interpretability. However, it also has drawbacks such as insufficient feature learning capability and unstable generalization performance. To alleviate these problems, broad learning system based on attention mechanism and tracking differentiator (TD), abbreviated as A-TD-BLS, was proposed. In terms of model structure, A-TD-BLS introduced self-attention mechanism to the original BLS, and further fused and transformed the extracted features through attention weighting to improve the feature learning ability.In terms of model training methods, a weight optimization algorithm based on tracking differentiator was designed.This method effectively alleviates the overfitting phenomenon of the original BLS by limiting the size of the weight values, significantly reduces the influence of the number of hidden layer nodes on model performance and makes the generalization performance more stable.Moreover, the training algorithm was extended to the BLS incremental learning framework, so that the model can improve performance by dynamically adding hidden layer nodes.Multiple experiments conducted on some benchmark datasets show that compared to the original BLS, the classification accuracy of A-TD-BLS is increased by 1.27% on average on classification datasets and the root mean square error of A-TD-BLS is reduced by 0.53 on average on regression datasets.Besides, A-TD-BLS is less affected by the number of hidden layer nodes and has more stable generalization performance. Based on the above experimental results, it can be concluded that A-TD-BLS enhances the stability of generalization performance of the original BLS model, reduces the sensitivity of the model's generalization performance to hyperparameters, and effectively suppresses the phenomenon of overfitting. |
|---|---|
| ISSN: | 1000-2618 |