Equivalence of Informations Characterizes Bregman Divergences
Bregman divergences form a class of distance-like comparison functions which plays fundamental roles in optimization, statistics, and information theory. One important property of Bregman divergences is that they generate agreement between two useful formulations of information content (in the sense...
Saved in:
| Main Author: | |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
MDPI AG
2025-07-01
|
| Series: | Entropy |
| Subjects: | |
| Online Access: | https://www.mdpi.com/1099-4300/27/7/766 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850078060281331712 |
|---|---|
| author | Philip S. Chodrow |
| author_facet | Philip S. Chodrow |
| author_sort | Philip S. Chodrow |
| collection | DOAJ |
| description | Bregman divergences form a class of distance-like comparison functions which plays fundamental roles in optimization, statistics, and information theory. One important property of Bregman divergences is that they generate agreement between two useful formulations of information content (in the sense of variability or non-uniformity) in weighted collections of vectors. The first of these is the Jensen gap information, which measures the difference between the mean value of a strictly convex function evaluated on a weighted set of vectors and the value of that function evaluated at the centroid of that collection. The second of these is the divergence information, which measures the mean divergence of the vectors in the collection from their centroid. In this brief note, we prove that the agreement between Jensen gap and divergence informations in fact characterizes the class of Bregman divergences; they are the only divergences that generate this agreement for arbitrary weighted sets of data vectors. |
| format | Article |
| id | doaj-art-d21fab4ccb734dfb9e859cfd037bea65 |
| institution | DOAJ |
| issn | 1099-4300 |
| language | English |
| publishDate | 2025-07-01 |
| publisher | MDPI AG |
| record_format | Article |
| series | Entropy |
| spelling | doaj-art-d21fab4ccb734dfb9e859cfd037bea652025-08-20T02:45:38ZengMDPI AGEntropy1099-43002025-07-0127776610.3390/e27070766Equivalence of Informations Characterizes Bregman DivergencesPhilip S. Chodrow0Department of Computer Science, Middlebury College, Middlebury, VT 05753, USABregman divergences form a class of distance-like comparison functions which plays fundamental roles in optimization, statistics, and information theory. One important property of Bregman divergences is that they generate agreement between two useful formulations of information content (in the sense of variability or non-uniformity) in weighted collections of vectors. The first of these is the Jensen gap information, which measures the difference between the mean value of a strictly convex function evaluated on a weighted set of vectors and the value of that function evaluated at the centroid of that collection. The second of these is the divergence information, which measures the mean divergence of the vectors in the collection from their centroid. In this brief note, we prove that the agreement between Jensen gap and divergence informations in fact characterizes the class of Bregman divergences; they are the only divergences that generate this agreement for arbitrary weighted sets of data vectors.https://www.mdpi.com/1099-4300/27/7/766Bregman divergencemutual informationcharacterization theoremconvexity |
| spellingShingle | Philip S. Chodrow Equivalence of Informations Characterizes Bregman Divergences Entropy Bregman divergence mutual information characterization theorem convexity |
| title | Equivalence of Informations Characterizes Bregman Divergences |
| title_full | Equivalence of Informations Characterizes Bregman Divergences |
| title_fullStr | Equivalence of Informations Characterizes Bregman Divergences |
| title_full_unstemmed | Equivalence of Informations Characterizes Bregman Divergences |
| title_short | Equivalence of Informations Characterizes Bregman Divergences |
| title_sort | equivalence of informations characterizes bregman divergences |
| topic | Bregman divergence mutual information characterization theorem convexity |
| url | https://www.mdpi.com/1099-4300/27/7/766 |
| work_keys_str_mv | AT philipschodrow equivalenceofinformationscharacterizesbregmandivergences |