ESM3

License: ESM3 is open source and free for academic and commercial use under an MIT license. Please refer to the license for full terms.

Proto is not affiliated with Biohub. This toolkit is open source and builds on the implementation produced by this organization. Product names, logos, and trademarks are the property of their respective owners.

GitHub GitHub

HuggingFace

HuggingFace Website Website Publication Publication Cite Cite Tool Source Tool Source Open as Notebook Open as Notebook Run Locally Run Locally

Biohub/esm

View repo

biohub/esm3-sm-open-v1

View model

biohub.ai

Visit website

Simulating 500 million years of evolution with a language model

Thomas Hayes, Roshan Rao, … Alexander Rives

Science (2025)

Read paper

@article{hayes2025esm3,
  title={Simulating 500 million years of evolution with a language model},
  author={Hayes, Thomas and Rao, Roshan and Akin, Halil and Sofroniew, Nicholas J. and Oktay, Deniz and Lin, Zeming and Verkuil, Robert and Tran, Vincent Q. and Deaton, Jonathan and Wiggert, Marius and Badkundri, Rohil and Shafkat, Irhum and Gong, Jun and Derry, Alexander and Molina, Raul S. and Thomas, Neil and Khan, Yousuf A. and Mishra, Chetan and Kim, Carolyn and Bartie, Liam J. and Nemeth, Matthew and Hsu, Patrick D. and Sercu, Tom and Candido, Salvatore and Rives, Alexander},
  journal={Science},
  volume={387},
  number={6736},
  pages={850--858},
  year={2025},
  publisher={American Association for the Advancement of Science},
  doi={10.1126/science.ads0018}
}

@software{evolutionaryscale_2024,
  author={{EvolutionaryScale Team}},
  title={evolutionaryscale/esm},
  year={2024},
  publisher={Zenodo},
  doi={10.5281/zenodo.14219303},
  url={https://doi.org/10.5281/zenodo.14219303}
}

Copy citation

evo-design/proto-tools/proto_tools/tools/masked_models/esm3

View source

Open Notebook

Open notebook

Run locally with proto-tools

pip install git+https://github.com/evo-design/proto-tools.git

Toolkit contributors

Function	Description
`run_esm3_embeddings()`	Extract protein sequence embeddings and logits using ESM3 (GPU)	Docs Source
`run_esm3_sample()`	Sample masked positions in protein sequences using ESM3 language model (GPU)	Docs Source
`run_esm3_score()`	Score protein sequences using ESM3 language model (GPU)	Docs Source

Background

In 2025, Hayes et al. introduced ESM3, a generative model from EvolutionaryScale that departs from the encoder-only design of the ESM-1/ESM-2 line. ESM3 is a masked generative transformer that represents a protein across three simultaneous tracks (amino-acid sequence, discrete structure tokens, and function annotation). Training masks spans across all three tracks, so a single model can be prompted with any combination of partial sequence, structure, and function and asked to complete the rest. The flagship 98B-parameter model (esm3-large-2024-03) is available through the Biohub Platform API (also offered via AWS SageMaker); the publicly released open checkpoint, esm3_sm_open_v1, is the small 1.4B-parameter variant. ESM3 is the multimodal successor to ESM-2 (Lin et al., 2023). Where ESM-2 is a sequence-only masked language model, ESM3 adds structure and function tracks and a generative objective. For pure sequence-embedding workloads ESM-2 remains lighter and faster; ESM3 is the choice when masked generative editing matters. This toolkit exposes only the sequence-track operations (embeddings, masked sampling, scoring) over supplied sequences.

Tools

ESM3 Embeddings (`esm3-embedding`)

Runs a single forward pass over ESM3 and mean-pools the per-residue hidden states into a fixed-length sequence descriptor. Per-position amino-acid logits are returned on request.

API Reference

Source

Input: MaskedModelInput

List[string]

required

Protein sequence(s) to process. Can be provided as:

Source

Config: ESM3EmbeddingsConfig

string

default:"esm3_sm_open_v1"

ESM3 weights variant. Currently "esm3_sm_open_v1" is the only public open-weights checkpoint.

boolean

default:"False"

Include per-position logits in the output (large; disable to save memory).

integer

default:"-1"

Transformer layer index for embeddings. -1 returns the post-norm last-block output (matches ESM2/ESMC -1 semantics); other indices select pre-norm per-block hiddens. Both are captured via a forward hook on model.transformer since ESM3.forward discards them.

integer

default:"0"

Print status messages during model execution.

string

default:"cuda"

Device to run the model on.

integer

default:"3600"

Maximum execution time in seconds. None waits indefinitely.

integer

Random seed. When set, tools run reproducibly up to small GPU float noise (see BaseToolOutput.approx_equal), and the seed participates in cache keys. When None, cacheable seed-sensitive tools skip cache until seeded.

integer

default:"8"

Number of sequences to process in parallel. Larger batches improve throughput but require more GPU memory.

Source

Output: ESM3EmbeddingsOutput

List[SequenceEmbedding]

required

Per-sequence embedding results. Each SequenceEmbedding contains:

Show SequenceEmbedding

List[number]

required

Mean-pooled embedding vector for one sequence.

List[integer]

required

Binary mask indicating valid positions (1) vs padding (0).

array

Optional per-position amino acid logits for one sequence.

Projection2D

Optional 2D coordinate from a UMAP projection of all embeddings in the same call. Populated when n_sequences >= 4; None otherwise (single-point or 2-3-point UMAP is meaningless).

Applications

The mean-pooled embedding is a learned protein representation for downstream supervised tasks such as clustering, classification, and property regression, and powers similarity search through cosine similarity on the mean vector.

Usage Tips

repr_layer selects which transformer layer is mean-pooled. The default -1 returns the post-norm output of the last block (matching ESM-2/ESMC -1 semantics); other indices select pre-norm per-block hidden states, captured via a forward hook because ESM3.forward discards them.
Per-position logits are large. Enabling return_logits adds a per-position vocabulary-sized float tensor per sequence, dominating wall time and memory on long inputs. Leave it False unless the per-position distribution is needed.

ESM3 Sampling (`esm3-sample`)

Selects positions via a configurable masking strategy, masks them, and resamples from ESM3’s predicted distribution. single_pass fills every masked position in one forward pass; iterative_refinement dispatches to ESM3’s native batch_generate for multi-round commitment. Positions can also be pre-masked directly with _ in the input string, or a masking strategy can be used.

API Reference

Source

Input: MaskedModelInput

List[string]

required

Protein sequence(s) to process. Can be provided as:

Source

Config: ESM3SampleConfig

MaskingStrategy

Positions to mask before sampling.

Show MaskingStrategy

number

default:"1.0"

Temperature for position selection. < 1.0 is greedy, 1.0 uses scores as-is, > 1.0 is more uniform. Only affects model-based methods.

enum

default:"random"

Scoring method for position selection. "random": uniform random, "entropy": highest model uncertainty, "max-logit": lowest model confidence.Available options: random, entropy, max-logit

integer

Exact number of positions to mask per sequence.

number

Fraction of designable positions to mask (e.g. 0.15 for ~15%).

array

1-indexed positions that must NOT be masked. Applied uniformly to all sequences.

string

default:"esm3_sm_open_v1"

ESM3 weights variant.

enum

default:"single_pass"

“single_pass” fills every mask in one forward; “iterative_refinement” dispatches to model.batch_generate and uses the five GenerationConfig settings below.Available options: single_pass, iterative_refinement

number

default:"1.0"

Softmax temperature.

number

default:"1.0"

Nucleus threshold (iterative only).

integer

default:"20"

Refinement steps (iterative only).

enum

default:"cosine"

Unmask schedule (iterative only).Available options: cosine, linear

enum

default:"random"

Per-round commit selection (iterative only).Available options: random, entropy

boolean

default:"True"

Anneal toward 0 across rounds (iterative only).

boolean

default:"False"

Include per-position logits.

integer

default:"0"

Verbosity level (0=quiet, 1=info, 2=debug, 3=raw subprocess stderr). True is coerced to 1 and False to 0.

string

default:"cuda"

Device to run on.

integer

default:"3600"

Maximum execution time in seconds. None waits indefinitely.

integer

default:"8"

Sequences per GPU forward pass.

Source

Output: ESM3SampleOutput

List[MaskedModelSample]

required

One entry per input sequence, in input order, each holding the sampled sequence and its optional per-position logits.

Show MaskedModelSample

string

required

The sampled or restored protein sequence.

array

Per-position amino acid logits for this sequence, shape [seq_len, 20]. Present only when the tool’s config sets return_logits=True.

Applications

This tool drives guided point mutation, variant generation, and infilling at designable sites. Resampling masked positions from a protein language model is the core operation behind directed-evolution proposals and antibody affinity maturation. Which positions are resampled is set by the masking strategy; see its README for the available selection methods and tuning parameters.

Usage Tips

iterative_refinement produces more coherent joint samples than single_pass. It runs ESM3’s batch_generate over num_steps rounds (cosine or linear unmask schedule) instead of filling every mask independently in one pass; it is roughly num_steps× slower. Default to it when masking more than a handful of sites.
masking_strategy controls which positions get masked before sampling. See the masking strategy README for the available selection methods and tuning parameters. As an alternative to passing a strategy, pre-mask exact positions with _ directly in the input string and the masking strategy is skipped entirely.
temperature scales the per-position logits before sampling. Values of 0.5 to 0.7 yield conservative mutations close to the input; values above 1.0 broaden exploration of the model’s distribution.

ESM3 Scoring (`esm3-score`)

Computes masked-language-model pseudo-perplexity for each input sequence. Each position is masked individually and the model’s log-probability of the true amino acid under bidirectional context is recorded, then aggregated into per-sequence log-likelihood, average log-likelihood, and perplexity.

API Reference

Source

Input: MaskedModelInput

List[string]

required

Protein sequence(s) to process. Can be provided as:

Source

Config: ESM3ScoringConfig

string

default:"esm3_sm_open_v1"

ESM3 weights variant.

integer

default:"0"

Print status messages during scoring.

string

default:"cuda"

Device to run the model on.

integer

default:"3600"

Maximum execution time in seconds. None waits indefinitely.

integer

default:"8"

Masked variants per forward pass, pooled across all input sequences. Larger batches improve throughput but use more memory.

boolean

default:"False"

Include per-position logits in the output (large; disable to save memory).

Source

Output: MaskedModelScoringOutput

List[MaskedModelScoringMetrics]

required

List of scoring outputs, one per input sequence. Each entry is a Metrics subclass with scalar metrics (accessed via score.perplexity or score["perplexity"]) plus declared logits / vocab fields that carry raw model outputs when requested.

Show MaskedModelScoringMetrics

array

Per-position logits array (seq_len, vocab_size). None unless return_logits=True.

array

Token ordering for logits.

string

Name of the metric that best summarizes the result overall (e.g. "avg_plddt" for AlphaFold2). Used by downstream UI and reporting to pick a headline value.

string

Concrete Metrics subclass tag; enables typed reconstruction after a serialization round-trip.

Metrics (one set per scores item)

Metric	Type	Range	Availability
`log_likelihood`	float	≤ 0.0	always
`avg_log_likelihood`	float	≤ 0.0	always
`perplexity`	float	≥ 1.0	always

Applications

ESM3 pseudo-perplexity is a fitness proxy for ranking variants, filtering generated sequences for naturalness, or comparing engineered constructs against wild type. The masked log-likelihood difference between wild-type and mutant residues is a zero-shot baseline for variant-effect prediction.

Usage Tips

Pseudo-perplexity is a relative score, not an absolute fitness. It is measured against the model’s training distribution and is sensitive to length, so it is most useful for comparing closely related sequences of similar length.
Ambiguous residues are excluded. Perplexity is computed only over the 20 canonical amino acids; X, B, Z, and similar are dropped from both the log-likelihood sum and the position count.

Toolkit Notes

These apply to every ESM3 tool in this toolkit (esm3-embedding, esm3-sample, esm3-score).

ESM3 is larger than many ESM-2 variants. For sequence-embedding-only workloads, smaller ESM-2 variants are faster; consider reaching for ESM3 when you want masked generative editing. This toolkit takes only amino-acid sequences as input and does not expose the structure or function tracks.
batch_size controls memory usage across the toolkit. Lower it if you OOM; raise it for short-sequence throughput. For esm3-score, batch_size counts masked variants pooled across all input sequences rather than sequences themselves (each input contributes one masked variant per position).

Example notebook: See the full working example for a copy-paste-ready walkthrough.

Infrastructure Guides

The following guides cover how to run tools efficiently and at scale.

Tool Persistence

Keep a tool’s model warm across calls instead of reloading it every invocation.

Device Management

How GPUs are allocated to tools and how to target specific devices.

Parallel Execution

Fan a batch of inputs out across multiple GPUs.

Tools

Guides

Tool Catalog

Concepts

Implementing a Tool

Background

Tools

ESM3 Embeddings (`esm3-embedding`)

API Reference

Applications

Usage Tips

ESM3 Sampling (`esm3-sample`)

API Reference

Applications

Usage Tips

ESM3 Scoring (`esm3-score`)

API Reference

Applications

Usage Tips

Toolkit Notes

Infrastructure Guides

Tool Persistence

Device Management

Parallel Execution

​Background

​Tools

​ESM3 Embeddings (esm3-embedding)

​API Reference

​Applications

​Usage Tips

​ESM3 Sampling (esm3-sample)

​API Reference

​Applications

​Usage Tips

​ESM3 Scoring (esm3-score)

​API Reference

​Applications

​Usage Tips

​Toolkit Notes

​Infrastructure Guides

Tool Persistence

Device Management

Parallel Execution

Background

Tools

ESM3 Embeddings (`esm3-embedding`)

API Reference

Applications

Usage Tips

ESM3 Sampling (`esm3-sample`)

API Reference

Applications

Usage Tips

ESM3 Scoring (`esm3-score`)

API Reference

Applications

Usage Tips

Toolkit Notes

Infrastructure Guides