The Geometry of Concepts: Sparse Autoencoder Feature Structure

The Geometry of Concepts: Sparse Autoencoder Feature Structure

Sparse autoencoders have recently produced dictionaries of high-dimensional vectors corresponding to the universe of concepts represented by large language models. We find that this concept universe has interesting structure at three levels: (1) The “atomic” small-scale structure contains “crystals”...

Full description

Saved in:
Bibliographic Details
Main Authors: Yuxiao Li, Eric J. Michaud, David D. Baek, Joshua Engels, Xiaoqing Sun, Max Tegmark
Format: Article
Language:English
Published: MDPI AG 2025-03-01
Series:Entropy
Subjects:
sparse coding
mechanistic interpretability
neural networks
large language models
clustering
Online Access:https://www.mdpi.com/1099-4300/27/4/344
Tags: Add Tag
No Tags, Be the first to tag this record!

Similar Items

Similar Items