Computational Antibody Papers

Filter by tags
language models
Filter by published year
All
TitleKey points

  • 2024-12-18

    Bilingual language model for protein sequence and structure

    • language models
    • Prost5 : novel language model using the FoldSeek structural representation to introduce structural dimension to the model.
    • The foldseek 3Di representation is used to encode 3D protein structures as 1D token sequences, enabling seamless translation between amino acid sequences and structural representations.
    • The model was fine-tuned on 17 million AlphaFoldDB structures using ProtT5 as a base, with bi-directional translation tasks to map between amino acid (AA) and 3Di sequences.
    • ProstT5 achieves 3600-fold faster remote homology detection compared to AlphaFold-based methods, while maintaining near-experimental accuracy and improving fold classification tasks like CATH.
    • ProstT5 embeddings outperform ProtT5, ESM-1b, and Ankh for structure-related tasks and show competitive performance in inverse folding, generating diverse sequences with preserved structural similarity. Though in most cases ProteinMPNN still performs better for inverse folding.

  • 2024-12-18

    IgBlend: Unifying 3D Structures and Sequences in Antibody Language Models

    • language models
    • nanobodies
    • Novel language model for antibodies, blending sequence and structural information.
    • The model encodes sequence ‘as usual’ and uses GVP-GNN (like esm-if) for structural representation. Only the three backbone atoms (C,N,Ca) are taken per residue to get the structural representation.
    • The data is a mix of sequence data and X-ray structures. The sequence datasets were modeled using ImmuneBuilder to increase structural coverage.
    • The model has an MLM objective on sequence & structure with three losses - sequence only, sequence + structure and structure only.
    • On sequence infilling IgBLEND performs better than other methods (e.g. AbLang, Nanobert), though arguably CDR-H3 predictions look very ‘close’ across the board.
    • On inverse folding the method performs quite a stretch better with large gaps in CDR-H3 with notable improvements for nanobodies - that other methods like ESM-IF or AntiFold did not handle natively.

  • 2024-12-18

    Focused learning by antibody language models using preferential masking of non-templated regions

    • language models
    • Novel masking scheme for antibody sequences with applications to Vh-Vl pairing and specificity prediction.
    • Since antibodies have intrinsically biased mutation patterns in favor of CDRs, the authors questioned the canonical 15% uniform masking procedure in antibody language models.
    • They focused the masking on the CDR3 regions during training which resulted in faster convergence.
    • They tested pairing prediction of Vh/Vl and they noticed 60% random vs non-random pairing accuracy.

  • 2024-11-22

    Structure Language Models for Protein Conformation Generation

    • generative methods
    • language models
    • non-antibody stuff
    • Novel model for sampling the structural space of proteins - applied to nanobodies.
    • Protein structures are encoded into latent tokens using a discrete variational auto-encoder (dVAE), which captures residue-level local geometric features in a roto-translation invariant manner.
    • The model combines a dVAE encoder-decoder with a Structure Language Model (SLM) to model sequence-to-structure relationships.
    • One can sample alternate conformations by providing an amino acid sequence as input and use the SLM to generate latent tokens representing potential conformations. Decode these latent tokens back into 3D structures using the dVAE decoder to generate diverse structural ensembles.
    • Altogether, though they develop ESMdiff in the context of the paper, it is rather the structural language in the context of the broader framework of language models that forms the core of the message, rather than any single method.
    • The model was evaluated on tasks like generating equilibrium dynamics, fold-switching, and intrinsically disordered proteins, using metrics like JS-divergence, TM-score, and RMSD. It outperformed existing methods in speed and accuracy, generating structures 20-100× faster.

  • 2024-11-22

    AlphaBind, a Domain-Specific Model to Predict and Optimize Antibody-Antigen Binding Affinity

    • binding prediction
    • language models
    • AlphaBind, a deep learning model designed to optimize antibody sequences, by leveraging large-scale pre-trained affinity datasets and fine-tuning on experimental data.
    • AlphaBind was pre-trained on a dataset of 7.5 million antibody-antigen affinity measurements, which includes data from yeast display systems and diverse antibody libraries obtained from multiple experimental sources, focusing on quantitative affinity measurements.
    • The model utilizes transformer-based architecture with protein sequence embeddings generated using ESM-2nv (Evolutionary Scale Model) embeddings. The model consists of 4 attention heads, 7 layers, and about 15 million parameters.
    • The model fine-tunes on specific antibody-antigen systems using AlphaSeq data, then performs stochastic greedy optimization by generating sequence mutations (using ESM-2nv logits) to explore sequence space and predict binding affinity. This process generates thousands of candidate sequences, which are filtered based on affinity predictions and developability metrics before in vitro validation.
    • The novel sequences based on three systems were verified in experimentally.

  • 2024-10-24

    ESM3- Simulating 500 million years of evolution with a language model

    • language models
    • generative methods
    • An evolution of the ESM model family that scales up in terms of parameters, data, and computational power compared to ESM2, which allows it to improve on sequence, structure, and function representations of proteins.
    • ESM3 is a multimodal, bidirectional transformer that models sequence, structure, and function using discrete token representations for each modality. It merges these representations into a single latent space and is trained with a masked language model objective, allowing it to generate and predict across different modalities.
    • ESM3's largest model has 98 billion parameters.
    • The model was trained with 1.07 × 10²⁴ floating point operations (FLOPs) over a dataset of 771 billion tokens from 2.78 billion proteins.
    • Structural tokens in ESM3 are encoded by a discrete autoencoder that compresses three-dimensional protein structures into a sequence of discrete tokens. This is done by encoding local atomic environments around each amino acid and representing them in a simplified form that captures geometric properties.
    • There are a total of 4096 structural tokens to be had.
    • The structural autoencoder tokenizes protein structures by encoding local neighborhoods around each amino acid into discrete tokens. It uses a geometric attention mechanism that operates in local reference frames, based on bond geometry. This mechanism encodes and reconstructs the atomic structure, supervised by a geometric loss that preserves distances and orientations of bonds and atoms.
    • ESM3 can be used to generate novel sequences/proteins. generates protein sequences and structures by prompting the model with sequence or structural tokens. It uses iterative sampling, starting from a masked context, where tokens are predicted and unmasked progressively until a full sequence or structure is generated. This allows the model to create novel proteins that respect the given prompts or constraints.
    • The model was verified experimentally by generating novel proteins, including a green fluorescent protein, that was synthesized and tested for fluorescence in laboratory conditions. The novel protein had a sequence identity of 58% to the nearest known fluorescent protein.

  • 2024-10-24

    Benchmarking antibody generative models

    • generative methods
    • language models
    • binding prediction
    • Study evaluates a number of generative models on datasets of antibodies with reported affinities.
    • The methods tested were: MEAN, dyMEAN, IgBLEND, Ablang, Ablang2, AntiBerty, ESM, Antifold, ESM-IF, AbX, Diffab + their own version of Diffab.
    • Datasets used were the Absci HER2 datasets (100s of binders) and a number of datasets with tens of binders each.
    • All models have some correlation with the affinity data, though weak.
    • Adding epitope information is not a game changer, showing that information that is mostly captured is fitness of antibody first and antigen second, if at all.
    • Employing structural information helps as compared to purely sequence approaches.

  • 2024-10-08

    AntiFormer: graph enhanced large language model for binding affinity prediction

    • language models
    • Novel language model applied to predicting antibody binding affinity in antigen-less manner.
    • AntiFormer is a graph-based large language model that combines sequence information with graph structures to predict antibody binding affinity. Its dual-flow architecture includes a transformer-based encoder for sequence features and a graph convolutional network (GCN) for capturing structural relationships (from sequence!), offering enhanced prediction accuracy.
    • AntiFormer was compared against advanced models like AntiBERTy and AntiBERTa, as well as basic transformer models with 6 and 12 layers, demonstrating superior performance across all evaluation metrics. It shows a better performance but not by a huge margin.
    • The model's performance was evaluated using affinity datasets, including the Observed Antibody Space (OAS) database and an additional dataset containing 104,972 antibody sequences with annotated affinity values, highlighting its accuracy and efficiency.

  • 2024-09-04

    Adapting protein language models for structure-conditioned design

    • language models
    • binding prediction
    • protein prediction
    • Novel language model incorporating structural information, with demonstrated experimental ability to improve design of therapeutic antibodies.
    • The new language model, ProseLM, builds upon Progen family of models from the same authors.
    • Structural information in the form of structural adapter layers after language model layers, encoding backbone and associated functional annotations.
    • Models with more parameters achieve much better perplexity. There is also some improvement by adding tangential context information such as ligands etc.
    • They trained an antibody-specific version of ProseLM, only on SABDAB data and it does much better on sequence recovery even than the larger models.
    • They use the model to propose mutations for Nivolumab ad Secukinumab, with mutations both in CDRs and Frameworks. THey used structures from the PDB as the basis for designs.
    • They found better binders, however if CDRs were re-designed the overall success rate of maintaining binding was lower (25% for Nivolumab) than when frameworks were redesigned (92%).

  • 2024-09-04

    p-IgGen: A Paired Antibody Generative Language Model

    • generative methods
    • language models
    • developability
    • Novel generative model for antibody sequences that supports Vh/Vl pairing and generation of developable sequences.
    • Three models were created, IgGen (unpaired model), p-IgGen (unpaired fine-tuned on pairs) and developable p-IgGen (paired fine-tuned on developable sequences).
    • They used ca. 250m unpaired sequences and 1.8m paired sequences for training.
    • The model is based on GPT-2 but with rotary position embedding.
    • Developable sequences were defined as structural models of the 1.8m that had good TAP metrics (900,000 in total).
    • The model is much smaller than many of the models out there, (17m params), so it is more lightweight in training and application.
    • The model performs better on immunogenicity prediction than other models but worse on expression prediction.