Submission to Transactions on Machine Learning Research (TMLR)
Dear Editors,
We submit A Controlled Synthetic Benchmark for Educational Aspect-Based Sentiment Analysis for consideration at TMLR. The paper studies a problem that is becoming central as practitioners lean on large language models to synthesize labeled training data: the generated labels are often unfaithful to the text, and there is no easy way to tell whether a synthetic benchmark carries genuine learnable signal or merely reproduces label frequencies. We study both problems in educational ABSA, a setting where real aspect-labeled student feedback is private and costly to annotate, and we contribute a methodology that applies wherever models are trained on LLM-synthesized labels.
Synthetic supervision is now a default response to label scarcity, yet quality control for LLM-generated labels remains ad hoc. Two questions recur across applications: is a synthetic benchmark learnable rather than a memorized prior, and which generated rows are faithful enough to train on. Educational feedback analysis sharpens both questions because the real data are scarce and sensitive, and because a useful classifier must separate fine-grained pedagogical aspects rather than report overall sentiment.
The central difficulty is establishing trust in synthetic supervision without circularity. We address it with measured evidence rather than assertion. Permuting the labels collapses detection to the trivial floor (0.182 versus 0.276 micro-F1); accuracy scales monotonically with training size; threshold-free ranking (macro AUROC 0.681) shows the detector discriminates well above chance. The faithfulness audit that drives the filter agrees with human annotators at Cohen's kappa 0.56 (precision 0.88) on two human-annotated corpora, so the filter tracks human judgment rather than only other models. Faithfulness-aware filtering lowers transferred sentiment error on real data (paired 95% bootstrap CI excludes zero), and a lowest-faithfulness control collapses transfer across two architectures and two real benchmarks. Trained on synthetic data alone, the model recovers about 60% of a real-trained reference with no real labels, and synthetic pre-training followed by real fine-tuning exceeds real-only training.
The audit-filter-validate recipe is a general, low-cost tool for quality-controlling LLM-supervised data, and the released corpus gives the educational-NLP community a reproducible benchmark in a domain where public aspect-labeled data are scarce. The corpus and code are released so that both the resource and the method can be reused and extended.
The paper's evaluation is deliberately rigorous, and we have tried to keep each claim close to what the experiments show. Results are reported with multi-seed bootstrap confidence intervals and accompanied by negative and size-matched controls, replication across two model architectures and two real transfer corpora, threshold-free ranking metrics, and a validation of the faithfulness audit against human labels. We think the work might be of interest to the TMLR audience, since quality control for LLM-synthesized training data recurs across many machine-learning settings, and since the released educational benchmark and the audit-filter-validate method can each be reused on their own by readers working on synthetic data, evaluation, or applied NLP.
This manuscript is original, is not under review at any other venue, and has not been published elsewhere. The submission is prepared for double-blind review; the corpus and code are provided as anonymized supplementary material, with public release on acceptance. The authors declare no competing interests. We are happy to assist with the assignment of an Action Editor whose expertise covers synthetic data generation, evaluation methodology, or aspect-based sentiment analysis.
Sincerely,
Yehudit Aperstein and Alexander Apartsin