All benchmarks

Label Projection

Release v2.0.0 v1.0.0

Automated cell type annotation from rich, labeled reference data

9 methods
2 control methods
8 datasets
3 metrics
3 releases

A major challenge for integrating single cell datasets is creating matching cell type annotations for each cell. One of the most common strategies for annotating cell types is referred to as "cluster-then-annotate" whereby cells are aggregated into clusters based on feature similarity and then manually characterized based on differential gene expression or previously identified marker genes. Recently, methods have emerged to build on this strategy and annotate cells using known marker genes. However, these strategies pose a difficulty for integrating atlas-scale datasets as the particular annotations may not match.

To ensure that the cell type labels in newly generated datasets match existing reference datasets, some methods align cells to a previously annotated reference dataset and then project labels from the reference to the new dataset.

Here, we compare methods for annotation based on a reference dataset. The datasets consist of two or more samples of single cell profiles that have been manually annotated with matching labels. These datasets are then split into training and test batches, and the task of each method is to train a cell type classifer on the training set and project those labels onto the test set.

Contributors

  • Nikolay Markov
    authormaintainer
  • Scott Gigante
    author
  • Robrecht Cannoodt
    author

Leaderboard

Methods ranked by scaled overall mean. Each cell encodes a score from 0 to 1 by size and intensity.

QC: Normalisation Visualisation 3 plots

Per metric: points placed by control-anchored scaled score (x); dashed lines mark scaled 0 and 1 (worst/best control); the lower axis shows the raw score. Points beyond [-0.2, 1.2] are clamped to the edge as triangles. Hover a dot or line to highlight it and read details.

methodcontrol
  • Accuracyhigher better
    True LabelsTrue LabelsLogistic regression (…Logistic regression (log CP10k)Seurat reference mapp…Seurat reference mapping (SCTransform)Multilayer perceptron…Multilayer perceptron (log scran)XGBoost (log CP10k)XGBoost (log CP10k)Multilayer perceptron…Multilayer perceptron (log CP10k)XGBoost (log scran)XGBoost (log scran)Logistic regression (…Logistic regression (log scran)K-neighbors classifie…K-neighbors classifier (log CP10k)K-neighbors classifie…K-neighbors classifier (log scran)scANVI (All genes)scANVI (All genes)scANVI (Seurat v3 200…scANVI (Seurat v3 2000 HVG)scArches+scANVI (All …scArches+scANVI (All genes)scArches+scANVI (Seur…scArches+scANVI (Seurat v3 2000 HVG)Majority VoteMajority VoteRandom LabelsRandom Labels0.0140.2610.5070.754100.250.50.751rawscaled
  • F1 scorehigher better
    True LabelsTrue LabelsSeurat reference mapp…Seurat reference mapping (SCTransform)Logistic regression (…Logistic regression (log CP10k)Multilayer perceptron…Multilayer perceptron (log scran)Multilayer perceptron…Multilayer perceptron (log CP10k)XGBoost (log CP10k)XGBoost (log CP10k)XGBoost (log scran)XGBoost (log scran)Logistic regression (…Logistic regression (log scran)K-neighbors classifie…K-neighbors classifier (log CP10k)K-neighbors classifie…K-neighbors classifier (log scran)scANVI (All genes)scANVI (All genes)scANVI (Seurat v3 200…scANVI (Seurat v3 2000 HVG)scArches+scANVI (All …scArches+scANVI (All genes)scArches+scANVI (Seur…scArches+scANVI (Seurat v3 2000 HVG)Random LabelsRandom LabelsMajority VoteMajority Vote0.0170.2620.5080.754100.250.50.751rawscaled
  • Macro F1 scorehigher better
    True LabelsTrue LabelsLogistic regression (…Logistic regression (log CP10k)Seurat reference mapp…Seurat reference mapping (SCTransform)XGBoost (log CP10k)XGBoost (log CP10k)XGBoost (log scran)XGBoost (log scran)Multilayer perceptron…Multilayer perceptron (log CP10k)Multilayer perceptron…Multilayer perceptron (log scran)K-neighbors classifie…K-neighbors classifier (log CP10k)K-neighbors classifie…K-neighbors classifier (log scran)Logistic regression (…Logistic regression (log scran)scANVI (All genes)scANVI (All genes)scANVI (Seurat v3 200…scANVI (Seurat v3 2000 HVG)scArches+scANVI (All …scArches+scANVI (All genes)scArches+scANVI (Seur…scArches+scANVI (Seurat v3 2000 HVG)Random LabelsRandom LabelsMajority VoteMajority Vote2.2e-30.2520.5010.751100.250.50.751rawscaled
QC: Indicator table all clear

Automated checks on the benchmark run and its results: missing values, score scaling, metric ranges and similar. Errors are high-severity issues that usually need a maintainer's attention; warnings are lower-severity signals. Findings that are expected for this task are listed separately as silenced.

No high-severity issues. 149 of 149 checks passed.

Method info 9

K-neighbors classifier uses the "k-nearest neighbours" approach, which is a popular machine learning algorithm for classification and regression tasks. The assumption underlying KNN in this context is that cells with similar gene expression profiles tend to belong to the same cell type. For each unlabelled cell, this method computes the $k$ labelled cells (in this case, 5) with the smallest distance in PCA space, and assigns that cell the most common cell type among its $k$ nearest neighbors.

parameter sets tested log CP10klog scran

Logistic Regression estimates parameters of a logistic function for multivariate classification tasks. Here, we use 100-dimensional whitened PCA coordinates as independent variables, and the model minimises the cross entropy loss over all cell type classes.

parameter sets tested log CP10klog scran

Assignment of all predicted labels as the most common label in the training data

Multilayer perceptroncode ↗source ↗Hinton, 1989

MLP or "Multi-Layer Perceptron" is a type of artificial neural network that consists of multiple layers of interconnected neurons. Each neuron computes a weighted sum of all neurons in the previous layer and transforms it with nonlinear activation function. The output layer provides the final prediction, and network weights are updated by gradient descent to minimize the cross entropy loss. Here, the input data is 100-dimensional whitened PCA coordinates for each cell, and we use two hidden layers of 100 neurons each.

parameter sets tested log CP10klog scran

scANVI or "single-cell ANnotation using Variational Inference" is a semi-supervised variant of the scVI(Lopez et al. 2018) algorithm. Like scVI, scANVI uses deep neural networks and stochastic optimization to model uncertainty caused by technical noise and bias in single - cell transcriptomics measurements. However, scANVI also leverages cell type labels in the generative modelling. In this approach, scANVI is used to predict the cell type labels of the unlabelled test data.

parameter sets tested All genesSeurat v3 2000 HVG

scArches+scANVI or "Single-cell architecture surgery" is a deep learning method for mapping new datasets onto a pre-existing reference model, using transfer learning and parameter optimization. It first uses scANVI to build a reference model from the training data, and then apply scArches to map the test data onto the reference model and make predictions.

parameter sets tested All genesSeurat v3 2000 HVG

scArches+scANVI or "Single-cell architecture surgery" is a deep learning method for mapping new datasets onto a pre-existing reference model, using transfer learning and parameter optimization. It first uses scANVI to build a reference model from the training data, and then apply scArches to map the test data onto the reference model and make predictions.

parameter sets tested All genesSeurat v3 2000 HVG
Seurat reference mappingcode ↗source ↗Hao et al., 2021

Seurat reference mapping is a cell type label transfer method provided by the Seurat package. Gene expression counts are first normalised by SCTransform before computing PCA. Then it finds mutual nearest neighbours, known as transfer anchors, between the labelled and unlabelled part of the data in PCA space, and computes each cell’s distance to each of the anchor pairs. Finally, it uses the labelled anchors to predict cell types for unlabelled cells based on these distances.

parameter sets tested SCTransform

XGBoost is a gradient boosting decision tree model that learns multiple tree structures in the form of a series of input features and their values, leading to a prediction decision, and averages predictions from all its trees. Here, input features are normalised gene expression values.

parameter sets tested log CP10klog scran
Control method info 2

Random assignment of predicted labels proportionate to label abundance in training data

Perfect assignment of the predicted labels from the test labels

Metric info 3
Accuracyhigher is betterGrandini et al., 2020

Average number of correctly applied labels.

F1 scorehigher is betterGrandini et al., 2020

The F1 score is a weighted average of the precision and recall over all class labels, where an F1 score reaches its best value at 1 and worst score at 0, where each class contributes to the score relative to its frequency in the dataset.

Macro F1 scorehigher is betterGrandini et al., 2020

The macro F1 score is an unweighted F1 score, where each class contributes equally, regardless of its frequency.

Dataset info 8
CeNGEN (random split) unlinked

100k FACS-isolated C. elegans neurons from 17 experiments sequenced on 10x Genomics. Split into train/test randomly. Dimensions: 100955 cells, 22469 genes. 169 cell types avg. 597±800 cells per cell type).

CeNGEN (split by batch) unlinked

100k FACS-isolated C. elegans neurons from 17 experiments sequenced on 10x Genomics. Split into train/test by experimental batch. Dimensions: 100955 cells, 22469 genes. 169 cell types (avg. 597±800 cells per cell type).

Pancreas (by batch) unlinked

Human pancreatic islet scRNA-seq data from 6 datasets across technologies (CEL-seq, CEL-seq2, Smart-seq2, inDrop, Fluidigm C1, and SMARTER-seq). Split into train/test by experimental batch. Dimensions: 16382 cells, 18771 genes. 14 cell types (avg. 1170±1703 cells per cell type).

Pancreas (random split with label noise) unlinked

Human pancreatic islet scRNA-seq data from 6 datasets across technologies (CEL-seq, CEL-seq2, Smart-seq2, inDrop, Fluidigm C1, and SMARTER-seq). Split into train/test randomly with 20% label noise. Dimensions: 16382 cells, 18771 genes. 14 cell types (avg. 1170±1703 cells per cell type).

Pancreas (random split) unlinked

Human pancreatic islet scRNA-seq data from 6 datasets across technologies (CEL-seq, CEL-seq2, Smart-seq2, inDrop, Fluidigm C1, and SMARTER-seq). Split into train/test randomly. Dimensions: 16382 cells, 18771 genes. 14 cell types (avg. 1170±1703 cells per cell type).

Tabula Muris Senis Lung (random split) unlinked

All lung cells from Tabula Muris Senis, a 500k cell-atlas from 18 organs and tissues across the mouse lifespan. Split into train/test randomly. Dimensions: 24540 cells, 17985 genes. 39 cell types (avg. 629±999 cells per cell type).

90k cells from zebrafish embryos throughout the first day of development, with and without a knockout of chordin, an important developmental gene. Split into train/test by laboratory. Dimensions: 26022 cells, 25258 genes. 24 cell types (avg. 1084±1156 cells per cell type).

Zebrafish (random split) unlinked

90k cells from zebrafish embryos throughout the first day of development, with and without a knockout of chordin, an important developmental gene. Split into train/test randomly. Dimensions: 26022 cells, 25258 genes. 24 cell types (avg. 1084±1156 cells per cell type).

References

  1. Chen, T., & Guestrin, C. (2016). XGBoost. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 10.1145/2939672.2939785 ↗
  2. Open Problems for Single Cell Analysis Consortium. (2022). Open Problems. link ↗
  3. Cover, T., & Hart, P. (1967). Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 21–27. 10.1109/tit.1967.1053964 ↗
  4. Grandini, M., Bagli, E., & Visani, G. (2020). Metrics for Multi-Class Classification: an Overview. arXiv. 10.48550/arxiv.2008.05756 ↗
  5. Hao, Y., Hao, S., Andersen-Nissen, E., Mauck, W. M., Zheng, S., Butler, A., Lee, M. J., Wilk, A. J., Darby, C., Zager, M., Hoffman, P., Stoeckius, M., Papalexi, E., Mimitou, E. P., Jain, J., Srivastava, A., Stuart, T., Fleming, L. M., Yeung, B., … Satija, R. (2021). Integrated analysis of multimodal single-cell data. Cell, 184(13), 3573-3587.e29. 10.1016/j.cell.2021.04.048 ↗
  6. Hinton, G. E. (1989). Connectionist learning procedures. Artificial Intelligence, 40(1–3), 185–234. 10.1016/0004-3702(89)90049-0 ↗
  7. Hosmer Jr, D. W., Lemeshow, S., & Sturdivant, R. X. (2013). Applied logistic regression (Vol. 398). John Wiley & Sons.
  8. Lotfollahi, M., Naghipourfar, M., Luecken, M. D., Khajavi, M., Büttner, M., Avsec, Z., Misharin, A. V., & Theis, F. J. (2020). Query to reference single-cell integration with transfer learning. bioRxiv. 10.1101/2020.07.16.205997 ↗
  9. Xu, C., Lopez, R., Mehlman, E., Regier, J., Jordan, M. I., & Yosef, N. (2021). Probabilistic harmonization and annotation of single-cell transcriptomics data with deep generative models. Molecular Systems Biology, 17(1). 10.15252/msb.20209620 ↗