A Controlled Synthetic Benchmark for Educational Aspect-Based Sentiment Analysis (TMLR)
We thank the reviewer for a detailed and technically precise report. We ran three new experiments (a targeted regeneration of the truncated rows, a four-family generator-robustness study, and a distributional realism analysis) and revised the manuscript accordingly. Each point is answered below with the specific evidence and its location.
Requested: 841 of 10,000 samples truncated at the token cap; full-corpus length-band adherence 0.6819 undercuts the "controlled corpus" claim.
Done, with a new experiment. We isolated the cause: the shortfall is entirely a token-budget artifact. On the 9,159 rows that never hit the cap, length-band adherence is 0.7304; the 841 truncated rows account for the whole gap below that level. Regenerating exactly those 841 rows with the same generator (gpt-5-nano) and prompt at a raised output cap eliminates every truncation (incomplete rate 8.41% to 0%) and recovers full-corpus adherence to 0.7125, close to but still below the never-truncated rate of 0.7304. The released corpus is unchanged; this is reported as an output-control diagnostic. We also note that the length band is a soft, sampled stylistic target, whereas the corpus's controlled variables are the per-review aspect-sentiment labels, which truncation does not affect. Finally, the generation length budget is itself realistic: real course-feedback platforms cap how much a student can write, and the retained reviews match the length profile of real full-length course reviews (median 122 words versus 104 on the public OMSCS reviews, with 92% inside the real 10th-to-90th-percentile band).
paper/n1_regenerate_truncated.py.
Requested: the worst-scoring quartile inflates sentiment error (Table 12) and seems to hold little training value.
Agreed; this supports the method. This is precisely the paper's filtering result rather than a counterpoint to it. The bottom bucket is defined purely as the rows the audit scores lowest, and training on it degrades transfer on every architecture and target we tested; the faithfulness-aware filter exists to remove exactly these rows. Discarding the bottom 50% by audit score matches full-corpus sentiment fidelity at half the training cost, and discarding the bottom 25% is shown to be an active liability. The reviewer's observation is the evidence that the audit score is an actionable training-data filter.
Section 5.7 (Table 12 and surrounding discussion).Requested: only 42 of 100 declared aspect-sentiment pairs find support; over half appear to conflict with the text.
Clarified. The 0.42 figure is a strict, conservative, per-aspect exact-polarity match required by the strictest auditor (gpt-5.2, explicitly instructed to be strict) on the raw, unfiltered corpus, so it is a lower bound on faithfulness, not a statement that "over half the sentiments conflict." A partly-correct multi-aspect row scores zero on any aspect whose polarity is merely understated. This raw diagnostic is exactly the signal the Section 5.7 filter exploits: the low-scoring rows are demonstrably worse training data and are removed by audit score. We made this reading explicit in the text.
Requested: kappa 0.56 is moderate, so the audit cannot be a fully reliable proxy for human judgement.
Addressed. We agree the audit is a useful signal rather than a ground-truth oracle, and the paper frames it that way; we do not claim it is a fully reliable proxy for human judgement. Three points support its use despite moderate agreement. First, its precision is 0.88 (a label it accepts as faithful agrees with the human annotation 88% of the time), and precision, not the symmetric kappa, is what governs a filter that removes low-scoring rows. Second, the audit is validated behaviorally rather than only by agreement: the lowest-scored bucket is defined purely as the rows the audit scores lowest, and training on it degrades transfer on every architecture and both real targets, while removing the bottom half matches full-corpus quality at half the data, so the score demonstrably selects better training rows regardless of whether any single verdict matches a human label. Third, a fully independent model family (Google gemini-2.5-flash) reproduces the human faithfulness judgments at Cohen's kappa 0.62, matching and slightly exceeding the same-family auditor, so the signal is human-grounded rather than a same-family artifact. We further characterize where the moderate agreement comes from (new Appendix A.22): decomposing the 2,482 decisions by review shape shows the audit's recall on genuine aspects rises with review length while its specificity against fabricated aspects falls as a review packs more aspects, so agreement is highest at moderate length and aspect count and lowest on the shortest and most aspect-dense reviews. The moderate overall kappa is thus a predictable, bounded property of the review distribution rather than unstructured noise. In short, the audit does not need to be a perfect human proxy to be a validated training-data filter, which is the role it plays.
Requested: LLM-as-judge realism does not establish statistically that synthetic reviews are indistinguishable from real feedback; the full dimensional label set lacks comprehensive human re-annotation.
Reframed, with analysis. We agree, and we now state plainly that perceptual indistinguishability under an adversarial frontier-model detector is not achieved and is not claimed; our own diagnostics confirm that current LLM-generated text of any family is separable by such a detector, so this is not the operative bar for a labeled training benchmark. The realism that matters here is functional: a model trained on the synthetic corpus recovers real-review aspect and polarity signal on two independent external corpora (Herath and EduRABSA). On the task-relevant distributional axes, review length (after the regeneration above) and punctuation rate are statistically indistinguishable from real OMSCS reviews. At the pool level a sentence-level MAUVE places synthetic-to-OMSCS at 0.23, below the 0.98 real-to-real ceiling and the 0.63 between two independent real corpora (OMSCS and Herath), so the synthetic sentence distribution remains separable in aggregate; this gap is style rather than course or platform identity (it survives scrubbing entities). Most directly, we show the detectability is a document-structure effect, not a sentence-realism one: judged sentence by sentence, individual synthetic sentences are near-indistinguishable from real ones (the same judge is only 60% accurate, close to the 50% chance floor, versus 93% on whole reviews, and calls 63% of synthetic sentences real). A control confirms the tell is a cumulative property of extended synthetic text rather than multi-aspect coverage: assembling real sentences into equally multi-aspect documents leaves them judged real (19%, versus 97.5% for synthetic reviews). The detectable signal is therefore orthogonal to the sentence-level aspect-sentiment labels the corpus supplies (new Appendix A.23). On human re-annotation, we scope the claim: 9 of 20 aspects are externally checked and full-schema human re-annotation remains the natural extension, which we state as a limitation.
Section 6.1 (limitation 2); Section 6.1 (limitation 1).Requested: testing does not extend beyond a single provider's GPT family and a single structured prompting methodology.
Done, with a new experiment. We ran a generator-robustness study across four families spanning four providers via OpenRouter: OpenAI gpt-5-nano, Google gemini-2.5-flash, Zhipu glm-4.6, and Meta llama-3.3-70b. The same 150 label-conditioned prompts were generated by each family and scored by the identical gpt-5.2 label-fidelity audit. All four produce auditable, label-faithful reviews (aspect support 0.921 to 0.968; per-aspect sentiment match 0.759 to 0.904), and the two non-GPT families match or exceed the same-family generator (Gemini 0.857, GLM 0.904, versus gpt-5-nano 0.759). This shows the generate-audit-filter pipeline is not specific to the GPT family; non-GPT generators, both closed and open-weight, produce auditable, comparably faithful data. On the audit side, the checker is already validated across three families (gpt-4o-mini, Claude, and Gemini in the audit-judge ablation and the cross-family human validation).
paper/n3_generator_fidelity.py.
Requested: refine the formatting of these items.
Done. Figures A2 and A3 were re-rendered with larger, selectable text and cleaner axes (the previous versions baked labels into glyph paths, which caused the small-text appearance). Figure 1 was redrawn with uniform boxes, uniform title and subtitle fonts, and straight orthogonal arrows. Table 5 was restructured with concise, scannable cells (replacing the prose-heavy originals) and brought up to date, adding the second transfer corpus, the multi-family prompted baseline, and a generator/output-control robustness row.
Figure 1; Table 5; Figures A2 and A3.Requested: the real reference data are small and drawn from a narrow domain, so conclusions apply to a narrow scope.
Clarified and scoped. The 32 OMSCS reviews are not the benchmark's real-data scope; they are only the blinded reference pool for the realism-judge loop, never used for training or evaluation, and that pool is freely enlargeable from public sources (thousands of RateMyProfessor comments are available). The actual external evaluation uses two independent annotated real corpora under two different annotation schemes: Herath et al. (2,829 reviews, 9-aspect overlap) and EduRABSA (7-aspect overlap), so the transfer evidence rests on thousands of real reviews across two schemes, not on 32 reviews. We revised the text so this distinction is explicit, and we continue to state that broader claims about the full 20-aspect schema require additional real-data tests and that high-stakes decisions require human-in-the-loop review. We also add the broader context that motivates this work: a recent systematic review of educational ABSA finds that only two of fifteen studies released any annotated data and that no public corpus provides full-review (rather than sentence-level) aspect labels. The narrowness the reviewer identifies is therefore a field-wide data scarcity, which is precisely the gap a controlled synthetic corpus is built to fill; we now state this explicitly and cite it.
Section 6.1 (limitations 1 and 2); Section 5 (external validation).We are grateful for a report that pushed us to quantify the truncation recovery, test the pipeline across generator families, and state the realism claim precisely. We are happy to provide any further detail.
Sincerely,
The authors