Meta-learning Representations for Learning from Multiple Annotators

Atsutoshi Kumagai; Taishi Nishiyama; Tomoharu Iwata; Yasuhiro Fujiwara; Yasutoshi Ida

REVIEW 2 major objections 5 minor 16 references

Meta-learning Representations for Learning from Multiple Annotators

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that meta-learning a neural embedding with pseudo-annotations plus EM adaptation of a Gaussian mixture with annotator-specific confusion matrices produces classifiers that outperform 13 existing methods when only a few…

desk verdict Solid meta-learning paper with a genuinely useful twist: pseudo-annotation during meta-training conditions the embedding on the downstream crowd-learning objective; the experiments are thorough and the central claim holds up. read the letter →

arxiv 2506.10259 v1 pith:VDFBORYB submitted 2025-06-12 cs.LG stat.ML

Atsutoshi Kumagai , Tomoharu Iwata , Taishi Nishiyama , Yasutoshi Ida , Yasuhiro Fujiwara This is my paper

classification cs.LGstat.ML

keywords meta-learningcrowdsourcingmultipleannotatorsnoisylabelsGaussianmixturemodelexpectation-maximizationconfusionmatrixpseudo-annotation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a classifier can be learned from very few noisy labels supplied by multiple annotators, provided the model was meta-trained on clean labeled data from related tasks. To do this, it treats the ground-truth labels as hidden variables, models each annotator by a class-dependent but input-independent confusion matrix, and fits a Gaussian mixture model to a neural embedding of the support examples. The embedding is meta-learned by simulating noisy annotators during the inner adaptation loop, so that at test time both the label estimates and the annotator-ability estimates are reliable from a handful of examples. The paper reports that this method outperforms all 13 comparison methods on Omniglot, Miniimagenet, LabelMe, and CIFAR-10H in the small-support regime, and that the pseudo-annotation step, not just the meta-learning, is what accounts for most of the gain.

What carries the argument

The load-bearing mechanism is a probabilistic latent-variable model placed on the neural embedding: a Gaussian mixture model in which each class is an isotropic Gaussian component with a prototype mean, and each annotator is a confusion matrix governing the probability of an observed noisy label given the true class. MAP estimation with conjugate Gaussian and Dirichlet priors yields closed-form E-step responsibilities and M-step updates for prototypes, class priors, and confusion matrices. Because every update is differentiable in the embedding parameters, the EM adaptation loop can be unrolled for a few steps and differentiated through, enabling end-to-end meta-learning of the embedding by ordinary gradient descent.

What would settle it

Construct a target task in which annotator errors are input-dependent — for example, blur or distort some images so the same annotator confuses the same class pair only on the hard images, while labeling all images in the support set. Because the EM updates are derived under input-independent confusion matrices, the estimated responsibilities and prototypes would be systematically biased on the hard subset; if, with small support sets, the method's accuracy falls below an instance-dependent confusion baseline such as the sample-wise label fusion method compared in the appendix, the load-bearing assumption fails in exactly the regime the paper targets.

Watch

Extended reading notes

Core claim

The central claim is that a neural embedding meta-learned end-to-end with a differentiable EM inner loop produces accurate classifiers from small amounts of multi-annotator noisy data, even when the target annotators differ from the ones simulated in training. The ground-truth label of each support example is a latent variable; the embedded examples form a Gaussian mixture whose components are prototypes of each class, and each annotator is described by a confusion matrix. Maximum-a-posteriori estimation of these task-specific parameters via EM has closed-form, differentiable updates, so the query-set loss can be backpropagated through the entire adaptation to train the embedding network. In experiments, the method outperformed all 13 baselines across all support sizes and annotator counts on three primary datasets, and the ablation without pseudo-annotation shows the simulated-noise step is essential to the result.

Load-bearing premise

The method assumes each annotator's error pattern depends only on the true class, not on the particular example, and that each class forms an isotropic (spherical) Gaussian cluster in the learned embedding; if annotators err more on hard inputs, the estimated abilities and labels become biased.

Editorial extensions

If this is right

With only 1, 3, or 5 labeled examples per class, test accuracy exceeds all 13 baselines on all three primary datasets, with the largest gaps in the hardest regimes.
Removing the pseudo-annotation step drops average accuracy from 0.892 to 0.758 on Omniglot and from 0.542 to 0.449 on Miniimagenet, showing that clean-label meta-training alone does not transfer to noisy-annotator targets.
Meta-training time stays close to that of prototypical networks (1361 s vs. 1281 s on Omniglot) and well below MAML (3499 s), because the EM inner loop needs no second-order derivatives.
Under clean labels, uniform class priors, and zero precision parameter, the adapted classifier reduces exactly to a prototypical-network classifier, so the method is a strict generalization of prototype-based few-shot learning.
The generative model allows source and target tasks to have different numbers of classes, which is what makes meta-training on Miniimagenet and testing on LabelMe or CIFAR-10H work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the fitted confusion matrices already quantify each annotator's reliability per class, the same differentiable-EM machinery could be repurposed for active learning from multiple annotators, directing the next annotation request to the example and annotator that would most reduce label uncertainty; the paper lists active learning only as a future direction.
The robustness to annotator-type shifts between meta-training and test suggests the pseudo-annotator distribution is a tunable design choice: learning it from a small amount of unlabeled target-annotator behavior, or making it adaptive during meta-training, could further close the gap when target annotators are unusually adversarial.
The input-independent confusion matrix is what keeps the EM updates closed-form; replacing it with an instance-dependent model would break the differentiability trick, but an amortized confusion predictor could restore closed-form M-steps per mini-batch and extend the method to annotators who err more on hard examples.
If source tasks themselves contain noisy labels, the paper suggests pre-cleaning them with existing denoising methods; an untested alternative consistent with the paper's machinery is to treat source noise inside the same EM loop rather than cleaning it beforehand.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this paper deserves peer review and I'd bet it gets accepted after some polish. The central idea is a nice twist on meta-learning for multiple noisy annotators: instead of meta-learning a feature extractor on clean data and then bolting on a crowd model, the authors meta-learn the embedding while simulating noisy annotators, so the embedding is optimized for the downstream EM-based classifier. The closed-form, differentiable EM steps make the inner loop cheap, and the ablation against the no-pseudo-annotation variant is exactly the right experiment: it shows the pseudo-annotation is doing real work, not just the GMM. The math is solid. The EM derivation in Appendix F is standard and correct, with proper conjugate priors; the M-step equations check out. The experiments are more thorough than most: three synthetic-noise benchmarks plus two real crowdsourcing datasets (LabelMe and CIFAR-10H), standard errors, ablations on the number of EM steps, and cross-dataset transfer. They even report the one dataset where a competing method (LNL on Omniglot) beats them, which is the kind of honesty we don't see enough. The soft spots are minor. Baselines had their hyperparameters chosen using test performance; the proposed method used validation. That's a protocol inconsistency, but it works against the proposed method, so it doesn't explain the wins. The model assumes isotropic Gaussian classes and input-independent confusion matrices; those are strong assumptions, but the paper acknowledges them and shows empirically that more flexible example-dependent models do worse with few labels. The pseudo-annotator distribution used during meta-training is close to one of the target test distributions, which is a legitimate concern, but the experiments with different annotator types (from Appendix M.4) show the method is robust to that mismatch. The paper isn't going to reshape the field, but it solves a real problem with a clean, well-tested method. Anyone working on few-shot learning with noisy labels or crowdsourcing should read it. I'd send it to review.

Referee Report

2 major / 5 minor

Summary. This paper proposes a meta-learning method for classification when only a few noisy labels from multiple annotators are available. The method embeds examples with a neural network, fits a Gaussian mixture model in the embedding space while estimating annotator-specific confusion matrices via a closed-form EM algorithm, and meta-learns the embedding by backpropagating through the EM iterations using pseudo-annotators. Experiments on Omniglot, MiniImageNet, LabelMe, and CIFAR-10H compare the method with 13 baselines and show improved accuracy in most settings, including cross-dataset transfer.

Significance. The contribution is a principled integration of probabilistic crowdsourcing models with meta-learning, yielding a differentiable inner loop that is computationally efficient. The paper's strengths include detailed and correct EM derivations in Appendix F, extensive ablations (EM steps, Dirichlet hyperparameter, class imbalance, annotator types, number of annotators), and real-world crowdsourcing evaluations. The cross-dataset transfer experiments (MiniImageNet to LabelMe and CIFAR-10H) are particularly valuable. The results are reported with standard errors and paired t-tests, and limitations are honestly acknowledged. If the results are reproducible, the method would be a practical advance for few-shot learning with noisy multi-annotator data.

major comments (2)

[Section 4.2 / Section L] The evaluation protocol is inconsistent: for all comparison methods, the best test results are reported from their hyperparameter candidates, while the proposed method selects hyperparameters on a validation set. This gives the baselines an advantage (test-set tuning) and makes the paired t-test comparisons in Tables 1 and 2 statistically invalid as a fair comparison. The empirical evidence for the central claim must be obtained under a consistent protocol; please rerun baselines with validation-based hyperparameter selection and update the tables and statistical tests accordingly.
[Section 4.3] The claim 'The proposed method outperformed the other methods for all cases' is not supported by the reported tables, which contain several ties. For example, Table 1(b), NS=4, R=7 lists Ours=0.432 and PrMV=0.432, both marked as comparable; Table 1(a), NS=20, R=7 lists Ours=0.982 and w/o PA=0.981, again both bold. The paper's own definition of boldface is 'best and comparable,' so the text should be revised to say 'best or statistically comparable' rather than 'outperformed.'

minor comments (5)

[Section 3.2, Eq. (4)] The Dirichlet priors in Eq. (4) are written with exponent b for π_k and normalization Γ(K(b+1))/Γ(b+1)^K, which corresponds to concentration parameter b+1, but the M-step in Eq. (7) uses b in the numerator and Kb in the denominator, corresponding to concentration parameter b. The same mismatch occurs for c in the confusion-matrix prior. Please clarify which parameterization is intended.
[Section 3.2, Eq. (6)] The denominator in the E-step is written as p(u_n, Y, M, π, A); it should be p(u_n, Y | M, π, A), the probability of the observed variables conditioned on the parameters.
[Section 3.3, Eq. (9)] The notation E_{d, E(\bar S,Q)∼D_d} is unclear. Please describe the sampling process more explicitly, for example, first sample a task d, then sample disjoint support and query sets from the task's data.
[Section M.12] There is a typo: 'MCLAL' should be 'MCNAL' in the sentence about omitted methods.
[Reproducibility Statement] The reproducibility statement provides hyperparameters and architectures, but no code is made available. A public implementation would strengthen the reproducibility of the results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the central claim is supported by held-out query evaluation and independent ablations.

full rationale

The paper's derivation chain is a standard bi-level meta-learning objective. In the inner loop, task-specific parameters M, pi, and A are estimated from the noisy support set by maximizing the posterior in Eq. (2) via the EM updates in Eqs. (6) and (7), and the adapted classifier is given by Eq. (8). In the outer loop, the neural embedding parameters theta are meta-learned by minimizing the negative log-likelihood of clean query labels in Eqs. (9) and (10). No fitted target-task parameter is reused as evidence for the claim: the support-set adaptation is evaluated on held-out query examples whose ground-truth labels are not used during adaptation. Hyperparameters for the proposed method are selected on validation accuracy, while baseline methods report their best test results, so any evaluation bias favors the baselines rather than the proposed method. The pseudo-annotator distribution used during meta-training matches one of the four target distributions averaged in Table 1, but this is an experimental design choice, not a fitted quantity, and Figure 3 and Section M.4 show the method remains competitive when target annotator types and distributions are varied. The self-citations (Iwata 2021, 2023; Iwata and Kumagai 2020) are related-work context and are not load-bearing for the central derivation; no uniqueness theorem or ansatz is imported from them. The acknowledged assumptions of input-independent confusion matrices and isotropic Gaussian components are explicit modeling choices, and Section M.9 directly tests the alternative of example-dependent confusion matrices. Therefore no step in the derivation reduces by construction to its own inputs, and no circularity can be exhibited.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim relies on standard probabilistic modeling assumptions and hand-chosen hyperparameters, not on new physical entities or constants fitted to the target result.

free parameters (5)

Dirichlet hyperparameter b for class prior = selected from {1,10,100}
Controls the class prior; higher b makes prior more uniform; performance robust, b=100 often best in experiments (Section M.5).
Gaussian prior precision tau = 1.0
Set by hand for all experiments; shrinks prototype estimates toward origin.
Dirichlet hyperparameter c for confusion matrices = 1.0
Set by hand for all experiments; smooths confusion matrix estimates.
Number of EM steps J = 2 (chosen from {1,2,3,4,5,10})
Small J is sufficient; selected on validation.
Pseudo-annotator distribution p(B) = (0.1,0.7,0.2) with accuracy ranges
Design choice for simulating annotators during meta-training; not fitted to target data.

assumptions (5)

domain assumption Source tasks contain only clean ground-truth labels
Assumed throughout (Section 3.1 and Appendix A); source tasks contain only clean labels.
domain assumption Target and source classes are disjoint, and feature space is the same across tasks
Standard meta-learning assumption stated in Section 3.1.
domain assumption Embedded examples from each class follow an isotropic Gaussian distribution
Model assumption in Eq. (1) and Section 3.2; enables closed-form EM.
domain assumption Each annotator's confusion matrix is input-independent
Model assumption in Eq. (1) and Section 3.2; acknowledged as limitation in conclusion.
standard math Conjugate priors are used for tractability
Used to derive closed-form EM steps; see Appendix F.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Meta-learning Representations for Learning from Multiple Annotators." pith.science (2026). https://pith.science/paper/VDFBORYB

@misc{pith2026250610259,
  author       = {Pith},
  title        = {Pith review of: Meta-learning Representations for Learning from Multiple Annotators},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VDFBORYB}},
  note         = {Machine review of arXiv:2506.10259}
}

read the original abstract

We propose a meta-learning method for learning from multiple noisy annotators. In many applications such as crowdsourcing services, labels for supervised learning are given by multiple annotators. Since the annotators have different skills or biases, given labels can be noisy. To learn accurate classifiers, existing methods require many noisy annotated data. However, sufficient data might be unavailable in practice. To overcome the lack of data, the proposed method uses labeled data obtained in different but related tasks. The proposed method embeds each example in tasks to a latent space by using a neural network and constructs a probabilistic model for learning a task-specific classifier while estimating annotators' abilities on the latent space. This neural network is meta-learned to improve the expected test classification performance when the classifier is adapted to a given small amount of annotated data. This classifier adaptation is performed by maximizing the posterior probability via the expectation-maximization (EM) algorithm. Since each step in the EM algorithm is easily computed as a closed-form and is differentiable, the proposed method can efficiently backpropagate the loss through the EM algorithm to meta-learn the neural network. We show the effectiveness of our method with real-world datasets with synthetic noise and real-world crowdsourcing datasets.

Figures

Figures reproduced from arXiv: 2506.10259 by the authors.

**Figure 1.** Our problem setting. In the (meta-)training phase, our model is meta-learned with multiple source tasks. In the testing phase of the meta-learned model, a classifier is obtained using the meta-learned model adapted to noisy labeled data in a target task. 2. Related Work Many methods have been proposed for learning from multiple noisy annotators (Sheng & Zhang, 2019; Zhang et al., 2022). Early methods focused on est… view at source ↗

**Figure 2.** Overview of our meta-learning procedure. (1) For each training iteration, we randomly sample a few labeled data, called support set, and labeled data, called query set, from a randomly selected source task. (2) We generate pseudo-labels on the basis of a support set to simulate data obtained from multiple noisy annotators. (3) All examples are embedded to a latent space by a neural network, and (4) the task-specific… view at source ↗

**Figure 3.** Average and standard errors of accuracies when changing the ratio of spammers on target tasks. modeling annotators, it is interesting to extend our framework to handle input example-dependent confusion matrices as in (Gao et al., 2022; Guo et al., 2023; Li et al., 2024). In addition, it is also interesting to extend our framework to other crowdsourcing problems such as active learning from multiple annotators. We a… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Graphical model representation of the proposed model in the latent space of a task. The gray and non-gray nodes represent observe and unobserved variables, respectively. Embedded example un depends on neural network parameter θ since un := f(xn; θ). et al., 2022). Ther…

**Figure 5.** Figure 5: Average and standard errors of accuracies with the proposed method when changing the numbers of EM steps J. calculated with clean labeled data (query set), when using the model adapted to a few clean labeled data (support set) in source tasks. Then, they fine-tune the …

**Figure 6.** Figure 6: t-SNE (Van der Maaten & Hinton, 2008) visualization of embedded support and query examples obtained by the proposed method (Ours) and w/o PA in Omniglot. Each color represents a true class label. Darker and lighter colors represent support and query examples, respectiv…

**Figure 7.** Figure 7: Estimated confusion matrices for annotators on a target task by the proposed method on Omniglot. We used four-class classification problem: three support examples per class and five annotators. Vertical and horizontal axes represent estimated and true class indexes, re…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages

[3]

The number of classes in each task is four, and the number of support data per class (shot) was one, three, and five

Average test accuracy with different numbers of support data and annotators on Omniglot with normal, pair-wise flippers, and class-wise spammers on target tasks. The number of classes in each task is four, and the number of support data per class (shot) was one, three, and five. Boldface denotes the best and comparable methods according to the paired t-te...

work page arXiv
[4]

Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing

Li, H. and Yu, B. Error rate bounds and iterative weighted majority voting for crowdsourcing.arXiv preprint arXiv:1411.4086,

work page Pith review arXiv
[7]

in Section F. D. Our Graphical Model Figure 4 shows the graphical model representation of our model in the latent space of a task. E. Extended Related Work There are some meta-learning methods for learning from a few noisy labeled data without considering multiple annotators (Liang et al., 2022; Mazumder et al., 2021; Chen et al., 2022; Killamsetty et al....

work page 2022
[8]

The gray and non-gray nodes represent observe and unobserved variables, respectively

Graphical model representation of the proposed model in the latent space of a task. The gray and non-gray nodes represent observe and unobserved variables, respectively. Embedded example un depends on neural network parameter θ since un :=f(x n;θ) . et al., 2022). Therefore, this paper considers modeling multiple annotators. We experimentally compared the...

work page 2022
[10]

The number of classes in each task is ten, and the number of support data per class is one, three, and five

Average test accuracies with different numbers of support data NS on CIFAR-10H. The number of classes in each task is ten, and the number of support data per class is one, three, and five. Boldface denotes the best and comparable methods according to the paired t-test (p= 0.05). NS Ours LRMV LRDS RFMV RFDS CL CNAL PrMV PrDS MaMV MaDS MCL MCNAL w/o PA 100....

work page arXiv
[11]

Here, methods with the symbol ‘MV’ used majority voting for determining the label of each support example

in Sections M.8 and M.9. Here, methods with the symbol ‘MV’ used majority voting for determining the label of each support example. Methods with the symbol ‘DS’ used the DS model for estimating labels of support data (Dawid & Skene, 1979). Although the DS model is simple, it has been reported to perform better than many existing methods (Zheng et al., 201...

work page 1979
[12]

We used four-class classification problem: three support examples per class and five annotators

Estimated confusion matrices for annotators on a target task by the proposed method on Omniglot. We used four-class classification problem: three support examples per class and five annotators. Vertical and horizontal axes represent estimated and true class indexes, respectively. Each caption represents a true annotator type and a numerical value in the b...

work page 2019
[15]

Boldface denotes the best and comparable methods according to the paired t-test (p= 0.05)

Average test accuracy over different numbers of support data and annotators on each dataset. Boldface denotes the best and comparable methods according to the paired t-test (p= 0.05). PrMV PrDS MaMV MaDS MCL MCNAL Data Ours w/ PA w/ PA w/ PA w/ PA w/ PA w/ PA Omniglot0.8920.853 0.856 0.809 0.808 0.827 0.751 Miniimagenet0.5420.484 0.504 0.419 0.424 0.447 0...

work page 2022

Show all 16 references

[100]

Tables 3 and 4 show the average test accuracy with different numbers of support data and annotators on Omniglot and Miniimagenet, respectively

Omniglot 0.884 0.890 0.892 Miniimagenet 0.534 0.540 0.542 LabelMe 0.503 0.512 0.520 each meta-training iteration from the annotator’s distribution (p(E), p(H), p(S)) = (0.1,0.7,0.2), where E, H, S represent experts, hammers, and spammers, respectively. Tables 3 and 4 show the ...

2017
[2015]

Few-shot learning for topic modeling.arXiv preprint arXiv:2104.09011,

Iwata, T. Few-shot learning for topic modeling.arXiv preprint arXiv:2104.09011,

arXiv
[2016]

A survey on programmatic weak supervision.arXiv preprint arXiv:2202.05433,

Zhang, J., Hsieh, C.-Y ., Yu, Y ., Zhang, C., and Ratner, A. A survey on programmatic weak supervision.arXiv preprint arXiv:2202.05433,

arXiv
[2017]

11 Meta-learning Representations for Learning from Multiple Annotators A. Remarks Annotators’ modelingThe proposed method assumes input example-independent confusion matrices for modeling annotators as in previous works (Chu et al., 2021; Rodrigues & Pereira, 2018; Raykar et a...

2021
[2020]

Crowdsourcing with multiple-source knowledge trans- fer

Han, G., Tu, J., Yu, G., Wang, J., and Domeniconi, C. Crowdsourcing with multiple-source knowledge trans- fer. InIJCAI, 2021a. Han, G., Yu, G., Cui, L., Domeniconi, C., and Zhang, X. Crowdsourcing with meta-workers: a new way to save the budget.arXiv preprint arXiv:2111.04068,...

2004 arXiv
[2021]

We also evaluated other recent methods (Liang et al., 2022; Gao et al.,

(CNAL), its meta-learning variant (MCNAL), and the proposed method without pseudo-annotators (w/o PA). We also evaluated other recent methods (Liang et al., 2022; Gao et al.,

2022
[2022]

Kingma, D. P. and Ba, J. Adam: a method for stochastic optimization.arXiv preprint arXiv:1412.6980,

arXiv
[2024]

highway”, “inside city

are more appropriate for these situations. I. Data Details In the main paper, we used three real-world datasets: Omniglot, Miniimagenet, and LabelMe. Omniglot and Miniimagenet are commonly used in meta-learning studies (Snell et al., 2017; Finn et al., 2017; Rajeswaran et al.,...

2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

The number of classes in each task is four, and the number of support data per class (shot) was one, three, and five

Average test accuracy with different numbers of support data and annotators on Omniglot with normal, pair-wise flippers, and class-wise spammers on target tasks. The number of classes in each task is four, and the number of support data per class (shot) was one, three, and five. Boldface denotes the best and comparable methods according to the paired t-te...

work page arXiv

[2] [4]

Error Rate Bounds and Iterative Weighted Majority Voting for Crowdsourcing

Li, H. and Yu, B. Error rate bounds and iterative weighted majority voting for crowdsourcing.arXiv preprint arXiv:1411.4086,

work page Pith review arXiv

[3] [7]

in Section F. D. Our Graphical Model Figure 4 shows the graphical model representation of our model in the latent space of a task. E. Extended Related Work There are some meta-learning methods for learning from a few noisy labeled data without considering multiple annotators (Liang et al., 2022; Mazumder et al., 2021; Chen et al., 2022; Killamsetty et al....

work page 2022

[4] [8]

The gray and non-gray nodes represent observe and unobserved variables, respectively

Graphical model representation of the proposed model in the latent space of a task. The gray and non-gray nodes represent observe and unobserved variables, respectively. Embedded example un depends on neural network parameter θ since un :=f(x n;θ) . et al., 2022). Therefore, this paper considers modeling multiple annotators. We experimentally compared the...

work page 2022

[5] [10]

The number of classes in each task is ten, and the number of support data per class is one, three, and five

Average test accuracies with different numbers of support data NS on CIFAR-10H. The number of classes in each task is ten, and the number of support data per class is one, three, and five. Boldface denotes the best and comparable methods according to the paired t-test (p= 0.05). NS Ours LRMV LRDS RFMV RFDS CL CNAL PrMV PrDS MaMV MaDS MCL MCNAL w/o PA 100....

work page arXiv

[6] [11]

Here, methods with the symbol ‘MV’ used majority voting for determining the label of each support example

in Sections M.8 and M.9. Here, methods with the symbol ‘MV’ used majority voting for determining the label of each support example. Methods with the symbol ‘DS’ used the DS model for estimating labels of support data (Dawid & Skene, 1979). Although the DS model is simple, it has been reported to perform better than many existing methods (Zheng et al., 201...

work page 1979

[7] [12]

We used four-class classification problem: three support examples per class and five annotators

Estimated confusion matrices for annotators on a target task by the proposed method on Omniglot. We used four-class classification problem: three support examples per class and five annotators. Vertical and horizontal axes represent estimated and true class indexes, respectively. Each caption represents a true annotator type and a numerical value in the b...

work page 2019

[8] [15]

Boldface denotes the best and comparable methods according to the paired t-test (p= 0.05)

Average test accuracy over different numbers of support data and annotators on each dataset. Boldface denotes the best and comparable methods according to the paired t-test (p= 0.05). PrMV PrDS MaMV MaDS MCL MCNAL Data Ours w/ PA w/ PA w/ PA w/ PA w/ PA w/ PA Omniglot0.8920.853 0.856 0.809 0.808 0.827 0.751 Miniimagenet0.5420.484 0.504 0.419 0.424 0.447 0...

work page 2022