Proto is not affiliated with Biohub. This toolkit is open source and builds on the implementation produced by this organization. Product names, logos, and trademarks are the property of their respective owners.

```bibtex theme={null} @misc{candido2026language, title={Language Modeling Materializes a World Model of Protein Biology}, author={Candido, Salvatore and Hayes, Thomas and Derry, Alexander and Rao, Roshan and Lin, Zeming and Verkuil, Robert and Wu, Bryan and Lee, Jin Sub and Bruguera, Elise S. and Keval, Jehan A. and Kopylov, Mykhailo and Pak, John E. and Wu, Wesley and Thomas, Neil and Mataraso, Samson and Hsu, Alvin and Trotman-Grant, Ashton C. and Fatras, Kilian and dos Santos Costa, Allan and Badkundri, Rohil and Ak{\i}n, Halil and Oktay, Deniz and Deaton, Jonathan and Montabana, Elizabeth and Sitwala, Hrishita and Yu, Yue and Wiggert, Marius and Carlin, Dylan Alexander and Goering, Anthony W. and Blazejewski, Tomasz and Sandora, McCullen and Hla, Michael and Jia, Tina Z. and Kloker, Leon H. and Sofroniew, Nicholas J. and Uehara, Masatoshi and Pannu, Jassi and Bachas, Sharrol and Liu, Daniel S. and Sercu, Tom and Rives, Alexander}, year={2026}, url={https://www.biorxiv.org/content/10.64898/2026.06.03.729735}, note={Preprint} } @software{evolutionaryscale_2024, author={{EvolutionaryScale Team}}, title={evolutionaryscale/esm}, year={2024}, publisher={Zenodo}, doi={10.5281/zenodo.14219303}, url={https://doi.org/10.5281/zenodo.14219303} } ```

Copy citation

### ESM C SAE Features (`esmc-sae-features`) Runs each sequence through the ESM C backbone once with SAEs attached to the requested layers, and returns the active codebook features at each residue, ordered by descending magnitude. Start and end tokens are stripped so positions align with the input sequence: `feature_indices[0]` holds the features for residue 1, and the `position` column of an exported CSV is 1-indexed, matching the rest of proto-tools. #### API Reference

Source Protein sequence(s) to process. Can be provided as:

Source ESM C backbone whose activations are decomposed. The SAE must match the backbone it was trained on. Available options: `esmc_300m`, `esmc_600m`, `esmc_6b` Backbone layers to attach SAEs to. `None` uses the \~75%-depth layer Biohub sweeps (300M: 23, 600M: 27, 6B: 60). Each layer adds a download and GPU memory. Which activations the SAE was trained on. Hidden states give a global view; MLP outputs isolate one layer's computation. Available options: `hidden_states`, `mlp_outputs` Active features per residue. Fixed in the SAE's weights, so this selects a model rather than a threshold; only `64` was trained against every layer, and other values exist solely at the sweep layer. Available options: `16`, `32`, `64`, `128`, `256`, `512` Total features the SAE can represent, also fixed in its weights. Larger codebooks split concepts more finely; which sizes exist depends on `model_checkpoint` and `sae_target`. Available options: `8192`, `16384`, `32768`, `65536`, `131072` Which ESM C implementation supplies the activations the SAE reads. `"transformers"` matches the published SAE documentation. `"esm"` reads the `esmc` toolkit's weights instead, avoiding a second backbone download at the cost of \~1% disagreement in active features. Available options: `transformers`, `esm` Sequences per forward pass. Verbosity level (0=quiet, 1=info, 2=debug, 3=raw subprocess stderr). `True` is coerced to `1` and `False` to `0`. Device to run the model on. Maximum execution time in seconds. `None` waits indefinitely. Random seed. When set, tools run reproducibly up to small GPU float noise (see `BaseToolOutput.approx_equal`), and the seed participates in cache keys. When None, cacheable seed-sensitive tools skip cache until seeded.

Source Per-sequence SAE features, index-parallel with the input sequences. The input sequence these features were computed from, echoed so an exported row can name the residue a feature fired on. One entry per requested layer, ascending.

#### Applications Feature activations show which concepts the model recognizes at each residue, which supports interpreting what drives an embedding, locating functional sites without supervision, and comparing how proteins are represented internally. Because features are sparse and indexed, activations are directly comparable across proteins: within one SAE, a feature index always denotes the same learned concept. Indices are not comparable between different SAEs, including different layers of the same backbone, since each is trained separately and orders its codebook arbitrarily. The `ESMC-6B-sae-layer60-k64-codebook16384` SAE additionally has agent-generated natural-language descriptions for its codebook, available through the ESM Atlas. #### Usage Tips * **`layers` selects which activations are decomposed.** The default is the \~75%-depth layer Biohub sweeps (300M: 23, 600M: 27, 6B: 60), where representations transfer best to downstream tasks. Each extra layer adds a download and GPU memory. * **`sae_target` picks what the SAE reads.** `hidden_states` (the default) decomposes the accumulated residual stream after a block, so features reflect everything the model has built up to that depth; it is what the ESM Atlas and the published feature descriptions use. `mlp_outputs` decomposes only that block's own MLP contribution before the residual add, which attributes a feature to one layer's computation. MLP-output SAEs are published only at `codebook_size=131072`. * **`k` and `codebook_size` are only free at the sweep layer.** All-layer SAEs exist at `k=64` and one codebook size, so varying either requires `layers` to be exactly the sweep layer. The config rejects unpublished combinations and names the valid alternatives. * **Only requested layers are downloaded, and layer size tracks `codebook_size`.** Each layer file holds an encoder and decoder of `d_model x codebook_size` weights, so a hidden-state layer is 0.13 GB on 300M and 0.34 GB on 6B, while an MLP-output layer (131072 codebook) is 1.0 GB and 2.7 GB respectively. Requesting every layer of the 6B MLP collection would pull roughly 217 GB; the tool logs a warning past 10 GB rather than refusing, since a deliberate multi-layer sweep is legitimate. * **Rank features by normalized activation, not raw magnitude.** The largest raw activations belong to features that fire on nearly every protein and say little. Biohub's published statistics correct for this: `(activation / uniref90_max_activation) * uniref90_idf` scales a feature to \[0, 1] and upweights rare ones. `describe_sae_features` in `helpers.py` returns both statistics alongside each feature's label, for the one SAE with published descriptions (`ESMC-6B-sae-layer60-k64-codebook16384`, which the 6B defaults resolve to). * **Output size scales with `k` times sequence length.** Each residue carries `k` indices and `k` magnitudes, so a 300-residue protein at `k=64` yields 19,200 pairs per layer.