Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inference
The significance of statistical physics concepts such as entropy extends far beyond classical thermodynamics. We interpret the similarity between partitions in statistical mechanics and partitions in Bayesian inference as an articulation of a result by Jaynes (1957), who clarified that thermodynamic...
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Maynooth Academic Publishing
2025-03-01
|
| Series: | The Open Journal of Astrophysics |
| Online Access: | https://doi.org/10.33232/001c.131858 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1849414758407602176 |
|---|---|
| author | Rebecca Maria Kuntz Heinrich von Campe Tobias Röspel Maximilian Philipp Herzog Björn Malte Schäfer |
| author_facet | Rebecca Maria Kuntz Heinrich von Campe Tobias Röspel Maximilian Philipp Herzog Björn Malte Schäfer |
| author_sort | Rebecca Maria Kuntz |
| collection | DOAJ |
| description | The significance of statistical physics concepts such as entropy extends far beyond classical thermodynamics. We interpret the similarity between partitions in statistical mechanics and partitions in Bayesian inference as an articulation of a result by Jaynes (1957), who clarified that thermodynamics is in essence a theory of information. In this, every sampling process has a mechanical analogue. Consequently, the divide between ensembles of samplers in parameter space and sampling from a mechanical system in thermodynamic equilibrium would be artificial. Based on this realisation, we construct a continuous modelling of a Bayes update akin to a transition between thermodynamic ensembles. This leads to an information theoretic interpretation of Jazinsky's equality, relating the expenditure of work to the influence of data via the likelihood. We propose one way to transfer the vocabulary and the formalism of thermodynamics (energy, work, heat) and statistical mechanics (partition functions) to statistical inference, starting from Bayes' law. Different kinds of inference processes are discussed and relative entropies are shown to follow from suitably constructed partitions as an analytical formulation of sampling processes. Lastly, we propose an effective dimension as a measure of system complexity. A numerical example from cosmology is put forward to illustrate these results. |
| format | Article |
| id | doaj-art-8bdf2b1cb97b45cb882f15c5492c66da |
| institution | Kabale University |
| issn | 2565-6120 |
| language | English |
| publishDate | 2025-03-01 |
| publisher | Maynooth Academic Publishing |
| record_format | Article |
| series | The Open Journal of Astrophysics |
| spelling | doaj-art-8bdf2b1cb97b45cb882f15c5492c66da2025-08-20T03:33:43ZengMaynooth Academic PublishingThe Open Journal of Astrophysics2565-61202025-03-01810.33232/001c.131858Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inferenceRebecca Maria KuntzHeinrich von CampeTobias RöspelMaximilian Philipp HerzogBjörn Malte SchäferThe significance of statistical physics concepts such as entropy extends far beyond classical thermodynamics. We interpret the similarity between partitions in statistical mechanics and partitions in Bayesian inference as an articulation of a result by Jaynes (1957), who clarified that thermodynamics is in essence a theory of information. In this, every sampling process has a mechanical analogue. Consequently, the divide between ensembles of samplers in parameter space and sampling from a mechanical system in thermodynamic equilibrium would be artificial. Based on this realisation, we construct a continuous modelling of a Bayes update akin to a transition between thermodynamic ensembles. This leads to an information theoretic interpretation of Jazinsky's equality, relating the expenditure of work to the influence of data via the likelihood. We propose one way to transfer the vocabulary and the formalism of thermodynamics (energy, work, heat) and statistical mechanics (partition functions) to statistical inference, starting from Bayes' law. Different kinds of inference processes are discussed and relative entropies are shown to follow from suitably constructed partitions as an analytical formulation of sampling processes. Lastly, we propose an effective dimension as a measure of system complexity. A numerical example from cosmology is put forward to illustrate these results.https://doi.org/10.33232/001c.131858 |
| spellingShingle | Rebecca Maria Kuntz Heinrich von Campe Tobias Röspel Maximilian Philipp Herzog Björn Malte Schäfer Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inference The Open Journal of Astrophysics |
| title | Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inference |
| title_full | Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inference |
| title_fullStr | Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inference |
| title_full_unstemmed | Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inference |
| title_short | Partition function approach to non-Gaussian likelihoods: information theory and state variables for Bayesian inference |
| title_sort | partition function approach to non gaussian likelihoods information theory and state variables for bayesian inference |
| url | https://doi.org/10.33232/001c.131858 |
| work_keys_str_mv | AT rebeccamariakuntz partitionfunctionapproachtonongaussianlikelihoodsinformationtheoryandstatevariablesforbayesianinference AT heinrichvoncampe partitionfunctionapproachtonongaussianlikelihoodsinformationtheoryandstatevariablesforbayesianinference AT tobiasrospel partitionfunctionapproachtonongaussianlikelihoodsinformationtheoryandstatevariablesforbayesianinference AT maximilianphilippherzog partitionfunctionapproachtonongaussianlikelihoodsinformationtheoryandstatevariablesforbayesianinference AT bjornmalteschafer partitionfunctionapproachtonongaussianlikelihoodsinformationtheoryandstatevariablesforbayesianinference |