Proto is not affiliated with Biohub. This toolkit is open source and builds on the implementation produced by this organization. Product names, logos, and trademarks are the property of their respective owners.

```bibtex theme={null} @article{hayes2025esm3, title={Simulating 500 million years of evolution with a language model}, author={Hayes, Thomas and Rao, Roshan and Akin, Halil and Sofroniew, Nicholas J. and Oktay, Deniz and Lin, Zeming and Verkuil, Robert and Tran, Vincent Q. and Deaton, Jonathan and Wiggert, Marius and Badkundri, Rohil and Shafkat, Irhum and Gong, Jun and Derry, Alexander and Molina, Raul S. and Thomas, Neil and Khan, Yousuf A. and Mishra, Chetan and Kim, Carolyn and Bartie, Liam J. and Nemeth, Matthew and Hsu, Patrick D. and Sercu, Tom and Candido, Salvatore and Rives, Alexander}, journal={Science}, volume={387}, number={6736}, pages={850--858}, year={2025}, publisher={American Association for the Advancement of Science}, doi={10.1126/science.ads0018} } @software{evolutionaryscale_2024, author={{EvolutionaryScale Team}}, title={evolutionaryscale/esm}, year={2024}, publisher={Zenodo}, doi={10.5281/zenodo.14219303}, url={https://doi.org/10.5281/zenodo.14219303} } ```

Copy citation

### ESM3 Embeddings (`esm3-embedding`) Runs a single forward pass over ESM3 and mean-pools the per-residue hidden states into a fixed-length sequence descriptor. Per-position amino-acid logits are returned on request. #### API Reference

Source Protein sequence(s) to process. Can be provided as:

Source ESM3 weights variant. Currently `"esm3_sm_open_v1"` is the only public open-weights checkpoint. Include per-position logits in the output (large; disable to save memory). Transformer layer index for embeddings. `-1` returns the post-norm last-block output (matches ESM2/ESMC `-1` semantics); other indices select pre-norm per-block hiddens. Both are captured via a forward hook on `model.transformer` since `ESM3.forward` discards them. Print status messages during model execution. Device to run the model on. Maximum execution time in seconds. `None` waits indefinitely. Random seed. When set, tools run reproducibly up to small GPU float noise (see `BaseToolOutput.approx_equal`), and the seed participates in cache keys. When None, cacheable seed-sensitive tools skip cache until seeded. Number of sequences to process in parallel. Larger batches improve throughput but require more GPU memory.

Source Per-sequence embedding results. Each `SequenceEmbedding` contains: Mean-pooled embedding vector for one sequence. Binary mask indicating valid positions (1) vs padding (0). Optional per-position amino acid logits for one sequence. Optional 2D coordinate from a UMAP projection of all embeddings in the same call. Populated when `n_sequences >= 4`; `None` otherwise (single-point or 2-3-point UMAP is meaningless).

#### Applications The mean-pooled embedding is a learned protein representation for downstream supervised tasks such as clustering, classification, and property regression, and powers similarity search through cosine similarity on the mean vector. #### Usage Tips * **`repr_layer` selects which transformer layer is mean-pooled.** The default `-1` returns the post-norm output of the last block (matching ESM-2/ESMC `-1` semantics); other indices select pre-norm per-block hidden states, captured via a forward hook because `ESM3.forward` discards them. * **Per-position logits are large.** Enabling `return_logits` adds a per-position vocabulary-sized float tensor per sequence, dominating wall time and memory on long inputs. Leave it `False` unless the per-position distribution is needed.

### ESM3 Sampling (`esm3-sample`) Selects positions via a configurable masking strategy, masks them, and resamples from ESM3's predicted distribution. `single_pass` fills every masked position in one forward pass; `iterative_refinement` dispatches to ESM3's native `batch_generate` for multi-round commitment. Positions can also be pre-masked directly with `_` in the input string, or a masking strategy can be used. #### API Reference

Source Protein sequence(s) to process. Can be provided as:

Source Positions to mask before sampling. Temperature for position selection. \< 1.0 is greedy, 1.0 uses scores as-is, > 1.0 is more uniform. Only affects model-based methods. Scoring method for position selection. `"random"`: uniform random, `"entropy"`: highest model uncertainty, `"max-logit"`: lowest model confidence. Available options: `random`, `entropy`, `max-logit` Exact number of positions to mask per sequence. Fraction of designable positions to mask (e.g. 0.15 for \~15%). 1-indexed positions that must NOT be masked. Applied uniformly to all sequences. ESM3 weights variant. "single\_pass" fills every mask in one forward; "iterative\_refinement" dispatches to `model.batch_generate` and uses the five GenerationConfig settings below. Available options: `single_pass`, `iterative_refinement` Softmax temperature. Nucleus threshold (iterative only). Refinement steps (iterative only). Unmask schedule (iterative only). Available options: `cosine`, `linear` Per-round commit selection (iterative only). Available options: `random`, `entropy` Anneal toward 0 across rounds (iterative only). Include per-position logits. Verbosity level (0=quiet, 1=info, 2=debug, 3=raw subprocess stderr). `True` is coerced to `1` and `False` to `0`. Device to run on. Maximum execution time in seconds. `None` waits indefinitely. Random seed. When set, tools run reproducibly up to small GPU float noise (see `BaseToolOutput.approx_equal`), and the seed participates in cache keys. When None, cacheable seed-sensitive tools skip cache until seeded. Sequences per GPU forward pass.

Source One entry per input sequence, in input order, each holding the sampled sequence and its optional per-position logits. The sampled or restored protein sequence. Per-position amino acid logits for this sequence, shape `[seq_len, 20]`. Present only when the tool's config sets `return_logits=True`.

#### Applications This tool drives guided point mutation, variant generation, and infilling at designable sites. Resampling masked positions from a protein language model is the core operation behind directed-evolution proposals and antibody affinity maturation. Which positions are resampled is set by the [masking strategy](https://github.com/evo-design/proto-tools/blob/main/proto_tools/transforms/masking/README.md); see its README for the available selection methods and tuning parameters. #### Usage Tips * **`iterative_refinement` produces more coherent joint samples than `single_pass`.** It runs ESM3's `batch_generate` over `num_steps` rounds (cosine or linear unmask schedule) instead of filling every mask independently in one pass; it is roughly `num_steps×` slower. Default to it when masking more than a handful of sites. * **`masking_strategy` controls which positions get masked before sampling.** See the [masking strategy README](https://github.com/evo-design/proto-tools/blob/main/proto_tools/transforms/masking/README.md) for the available selection methods and tuning parameters. As an alternative to passing a strategy, pre-mask exact positions with `_` directly in the input string and the masking strategy is skipped entirely. * **`temperature` scales the per-position logits before sampling.** Values of 0.5 to 0.7 yield conservative mutations close to the input; values above 1.0 broaden exploration of the model's distribution.

### ESM3 Scoring (`esm3-score`) Computes masked-language-model pseudo-perplexity for each input sequence. Each position is masked individually and the model's log-probability of the true amino acid under bidirectional context is recorded, then aggregated into per-sequence log-likelihood, average log-likelihood, and perplexity. #### API Reference

Source Protein sequence(s) to process. Can be provided as:

Source ESM3 weights variant. Print status messages during scoring. Device to run the model on. Maximum execution time in seconds. `None` waits indefinitely. Random seed. When set, tools run reproducibly up to small GPU float noise (see `BaseToolOutput.approx_equal`), and the seed participates in cache keys. When None, cacheable seed-sensitive tools skip cache until seeded. Masked variants per forward pass, pooled across all input sequences. Larger batches improve throughput but use more memory. Include per-position logits in the output (large; disable to save memory).

Source List of scoring outputs, one per input sequence. Each entry is a `Metrics` subclass with scalar metrics (accessed via `score.perplexity` or `score["perplexity"]`) plus declared `logits` / `vocab` fields that carry raw model outputs when requested. Per-position logits array `(seq_len, vocab_size)`. `None` unless `return_logits=True`. Token ordering for `logits`. Name of the metric that best summarizes the result overall (e.g. `"avg_plddt"` for AlphaFold2). Used by downstream UI and reporting to pick a headline value. Concrete Metrics subclass tag; enables typed reconstruction after a serialization round-trip. **Metrics** (one set per `scores` item) | Metric | Type | Range | Availability | | -------------------- | ----- | ----- | ------------ | | `log_likelihood` | float | ≤ 0.0 | always | | `avg_log_likelihood` | float | ≤ 0.0 | always | | `perplexity` | float | ≥ 1.0 | always |

#### Applications ESM3 pseudo-perplexity is a fitness proxy for ranking variants, filtering generated sequences for naturalness, or comparing engineered constructs against wild type. The masked log-likelihood difference between wild-type and mutant residues is a zero-shot baseline for variant-effect prediction. #### Usage Tips * **Pseudo-perplexity is a relative score, not an absolute fitness.** It is measured against the model's training distribution and is sensitive to length, so it is most useful for comparing closely related sequences of similar length. * **Ambiguous residues are excluded.** Perplexity is computed only over the 20 canonical amino acids; `X`, `B`, `Z`, and similar are dropped from both the log-likelihood sum and the position count.