2026-08-29
Restructured draft. Numbers are current best estimates from the pipeline as of 2026-08-28. TODO markers name what is still missing.
Every large system logs discrete operational events: errors, retries,
task failures, maintenance actions. Frequent-pattern mining (FP-Growth,
PrefixSpan) surfaces recurrent event patterns from these logs, but
frequency does not imply predictiveness. Many mined patterns occur
equally often before failures and in normal operation. This paper’s
central object is the predictive-versus-frequent
separation across operational-log traces, produced by three
statistical safeguards: (i) an entity-disjoint
discovery/inference split so that the mining step’s
label-dependent selection does not invalidate the significance test on
selected patterns; (ii) a count-preserving order
comparator that measures the pure ordering effect of a sequence
pattern against a permuted-multiset null, isolating order from event
multiplicity; and (iii) a matched conditional logistic
estimator stratified by risk set for right-censored traces,
following the Prentice-Breslow nested-case- control formulation. We
characterise four operational traces spanning three domains: synthetic
industrial per-machine (Azure PdM), real cloud per-job (Alibaba v2018),
real HPC per-rack (LLNL Blue Gene/L syslogs), and real automotive
per-vehicle (SCANIA Component X). On Azure and Alibaba, 45-100% of
frequent itemsets pass post-selection- valid BY q<0.05 significance
on the inference half. Order effect under the count-preserving null: on
Azure, real sequence lift exceeds the count-preserving-shuffle lift by
+0.52 (last5) and +1.09 (last10); on Alibaba
the residual order effect is essentially zero (-0.02 to +0.24), showing
that the earlier naive “order gain” on Alibaba was largely a
multiplicity artefact rather than an ordering signal. On SCANIA
Component X, matched conditional logistic stratified by risk set gives
top per-pattern hazard ratio 1.73 (95% CI [1.53, 1.96], p = 1e-17), with
121/200 top patterns significant; a pooled 2x2 odds ratio on the same
data would inflate the effect to 2.72 by discarding the matched
structure. On BGL, essentially no non-alert precursor pattern survives
the post- selection-valid test, consistent with the target class being a
self-triggering alert cascade. Concrete Azure predictive signature
software_error:error2 → software_error:error3 reaches
sequence lift 3.73 with a genuine order effect after multiplicity
control. Adding surviving patterns to a temporally-held-out logistic
regression as binary features improves failure prediction by +5.6 AUROC
on Azure last5 over itemset-only features. Mined patterns,
matched-control windows, discovery/inference splits, and matched
hazard-ratio outputs are released as reproducible parquet artefacts
alongside the code repository.
Operational logs from datacenters, industrial fleets, and cloud platforms carry a rich stream of discrete events: software errors, task failures, retries, eviction notices, maintenance actions, component replacements. These events are usually consumed one at a time by alerting systems and dashboards. We ask a different question: do RECURRENT ORDERED SEQUENCES of these events precede failures systematically enough to serve as early-warning signatures?
Two mining families answer this shape of question. Frequent itemset mining (Apriori, FP-Growth) treats each pre-failure window as an unordered set of events. Sequential pattern mining (PrefixSpan, SPADE, GSP) preserves temporal order. The paper’s central question is whether the second family finds anything the first does not.
We contribute:
The two mining families this paper compares are well-established. Association-rule mining introduced Apriori (Agrawal and Srikant 1994) and FP-Growth (Han, Pei, and Yin 2000) over unordered transactions, the latter avoiding candidate generation through a compressed frequent-pattern tree. Sequential pattern mining extended this to ordered event streams: GSP (Srikant and Agrawal 1996) generalises Apriori-style level-wise generation to sequences; SPADE (Zaki 2001) uses a vertical id-list format; PrefixSpan (Pei et al. 2001) uses prefix-projected pattern growth, and is the algorithm we use in this paper.
Later variants sharpen the raw output. CM-SPAM / CM-SPADE (Fournier-Viger,
Gomariz, et al. 2014) speed vertical mining through co-occurrence
pruning; VMSP (Fournier-Viger, Wu, et al.
2014) returns only maximal sequences, compressing the pattern set
without losing coverage. A recent survey (Fournier-Viger et al.
2022) summarises open problems in the area. All of these
algorithms are implemented in the SPMF library (Fournier-Viger et al. 2016), which
we call from Python for the sequence-mining half of our pipeline; the
itemset half uses mlxtend (Raschka 2018).
Two families of methods dominate the literature: (i) explicit frequent-pattern mining on parsed log templates, close to what this paper does, and (ii) deep learning over log sequences.
Pattern mining. Ren et al. (Ren et al. 2021) apply Spark FP-Growth to failure prediction on BlueGene/L, LANL-HPC, and CMRI-Hadoop logs, using event-density sliding windows over long-tail event vocabularies. İfraz and Ersöz (İfraz and Ersöz 2024) run PrefixSpan and Apriori side-by-side on a bus-fleet maintenance log, showing that sequence mining recovers “errors → replacement” trajectories that itemset mining misses. Both studies stop short of the post-selection-valid matched-control + BY-corrected + predictive-utility comparison used here, and neither touches Alibaba PdM or Azure.
Discriminative and statistically-significant pattern mining. Dong and Li (Dong and Li 1999) introduced emerging patterns as itemsets whose support differs substantially between classes; Bay and Pazzani (Bay and Pazzani 1999) formulated the parallel notion of contrast sets in the same year. This class-contrast literature is a direct ancestor of the paper’s predictive-vs-frequent-noise separation. Statistically significant pattern mining sharpens the criterion: Terada et al. (Terada et al. 2013) give an exact family-wise-error control for combinatorial regulations. All of these methods share the ordinary post-selection-inference concern (Fithian, Sun and Taylor (Fithian, Sun, and Taylor 2014)) that mining and testing on the same sample invalidates marginal p-values; we address it via entity-disjoint discovery/inference splits (§4.5).
Deep learning on log sequences. DeepLog (Du et al. 2017) frames system-log anomaly detection as next-template prediction with a stacked LSTM, and remains the canonical DL baseline. LogAnomaly (Meng et al. 2019) adds unsupervised quantitative-anomaly detection alongside sequential anomalies. LogRobust (Zhang et al. 2019) adds an attention Bi-LSTM to survive log-template drift, and PLELog (Yang et al. 2021) introduces semi-supervised label estimation for the weakly-labelled setting. Recent transformer approaches (LogBERT (Haixuan Guo, Yuan, and Wu 2021); LogFormer (Hongcheng Guo et al. 2024)) pre-train on unlabelled logs and fine-tune for anomaly detection. These methods generally outperform classical pattern miners on held-out AUROC when trained on enough data, but produce opaque per-line anomaly scores rather than interpretable pre-failure signatures. Our contribution is orthogonal: we ask whether explicit ordered patterns add signal over their itemset counterparts, with every mined pattern human-readable and independently significance-tested.
The raw text of most system logs must first be converted into event templates before either family of methods applies. The Loghub collection (Zhu et al. 2023) curates parsed versions of BGL, HDFS, Thunderbird, and 13 other benchmark log datasets; the parsing benchmark of Zhu et al. (Zhu et al. 2019) compares Drain and alternative parsers on the same corpora. We take BGL from Loghub directly and use its native label field, avoiding the parsing step as a confound.
Cheng et al. (Cheng, Chai, and Anwar 2018) provide the standard characterisation of the Alibaba 2018 trace, reporting failure statistics and co-location effects but not extracting sequential patterns. Luo et al. (Luo et al. 2021) extend this to the microservice trace with focus on dependency and latency rather than fault prediction. Oliner and Stearley (Oliner and Stearley 2007) introduce the BGL log we use, along with four other HPC logs, and characterise their alert statistics.
The broader predictive-maintenance literature works mostly on continuous sensor telemetry. NASA C-MAPSS (Saxena et al. 2008) is the de-facto Remaining-Useful-Life benchmark; recent surveys (Serradilla et al. 2022) catalogue the deep-learning methods trained on it. Automotive predictive maintenance has historically used per-vehicle sensor snapshots (SCANIA APS Failure via the IDA 2016 industrial challenge (Costa and Nascimento 2016)); the SCANIA Component X release (Kharazian et al. 2025) we adopt extends this to a per-vehicle longitudinal readout stream. Our work sits between these traditions: we take discrete-event traces where possible, and derive discrete tokens (per §3.4) where we must.
We apply Benjamini-Hochberg FDR correction (Benjamini and Hochberg 1995) to the one-sided hypergeometric p-values on the inference half of the discovery/inference split (§4.5). For arbitrary dependence between p-values (justified when mined patterns share items), we also report the more conservative Benjamini-Yekutieli correction (Benjamini and Yekutieli 2001). The Prentice-Breslow retrospective-cohort framework (Prentice and Breslow 1978) and the Langholz-Goldstein risk-set-sampling review (Langholz and Goldstein 1996) motivate the matched conditional logistic estimator we use for SCANIA (§4.6).
To our knowledge no peer-reviewed study applies FP-Growth and
PrefixSpan head-to-head to Alibaba batch_task status
transitions or to Azure PdM errorID → failure sequences
with the matched-control design used here, then evaluates the resulting
patterns as binary features against event-count and
deep-learning-adjacent alternatives on a temporally-held-out split. The
four-trace regime-of-validity study in §7.4 is likewise, to our
knowledge, unprecedented in the pattern-mining log-analysis
literature.
We use four public event-log traces covering three domains.
100 machines, 2015-01-01 to 2016-01-01, hourly telemetry.
PdM_errors (3,919 non-fatal errors, five error codes),
PdM_maint (3,286 maintenance actions, four components),
PdM_failures (761 component replacements). We join
PdM_maint and PdM_failures on (machineID,
datetime, comp) to distinguish maintenance from
component_replacement. Failures at exactly 2015-01-02 03:00
(18 rows) do not match any PdM_maint record; they are a
bootstrap seed batch planted by the synthetic generator and are excluded
from BOTH anchors and event streams so they do not contaminate windows
for subsequent real failures.
Event vocabulary: software_error,
maintenance, component_replacement,
terminal_failure, each with a subtype
(error1..error5, comp1..comp4). Entity is
machineID. Source: (Microsoft 2016).
batch_task.csv from the public Alibaba trace (Alibaba
2018; Lu et al. 2017), 14,295,731 tasks across 4,201,014 jobs,
8.9 days (2018-01-01 through 2018-01-09 by trace clock). 83,207 jobs
contain at least one Failed task.
batch_instance.csv (21 GB compressed) is not used in this
pass; the per-job analysis on batch_task alone is
sufficient to answer the ordering question.
Event vocabulary: task_failure,
task_success, task_waiting,
task_running, each with subtype = task_name letter prefix
(M, R, J, task,
MergeTask, L). Entity is
job_name.
4,747,963 syslog messages from LLNL Blue Gene/L, 214.7 days
(2005-06-03 to 2006-01-04), from the Loghub archive [TODO:cite
oliner2007bgl]. 913,594 messages remain after dropping INFO-level noise;
348,189 (7.34%) are labeled alerts. Entity is the rack (top-level
R## prefix of the node ID); 64 racks. Event vocabulary:
terminal_alert (labeled alerts with 30+ alert codes such as
KERNMNTF, APPTO, KERNSTOR), system_error (non-alert FATAL /
ERROR / SEVERE / FAILURE), system_warning. Component (RAS,
KERNEL, APP, MMCS, …) is used as an additional subtype axis.
Real fleet telematics dataset released 2025 [TODO:cite kharazian2025], 23,550 trucks over 1.5 years (2019-01 through 2020-05 in study clock), 1,122,452 readouts of 105 numeric counter and histogram features. 2,272 vehicles (9.65%) undergo a component X repair during the study.
Because features are numeric counters rather than native discrete
events, we derive tokens: for each (vehicle, feature) we compute
inter-readout DELTAS and emit a counter_surprise token per
readout whenever the absolute delta exceeds the vehicle’s own
90th-percentile threshold for that feature. Per-vehicle normalisation
controls for baseline usage variation across the fleet. Entity is
vehicle_id. The failure event is a synthetic
terminal_repair marker placed at the last readout timestamp
of each repair-labeled vehicle.
For every terminal failure event on an entity (machine on Azure, job on Alibaba, rack on BGL, vehicle on SCANIA), we build a failure window covering the K events (or the time horizon T) strictly before the failure timestamp. Matched controls come from two designs depending on the trace:
BGL alerts are additionally grouped into episodes (>= 1h inter-arrival gap) and windows are anchored on the first alert of each episode, so anchor-per-alert double-counting inside a cascade is avoided.
Horizons studied: 1h, 6h, 24h,
last5, last10 on Azure; last3,
last5, last10 on Alibaba; last5,
last10, last20 on BGL and SCANIA (time-based
horizons are not meaningful for short per-job or per-episode
observations).
Items are event_type:event_subtype strings. FP-Growth
(mlxtend.frequent_patterns.fpgrowth) mines frequent
itemsets on failure windows at minimum support 0.05. PrefixSpan (SPMF
v2.64 (Fournier-Viger et al. 2016) via
subprocess) mines frequent ordered sequences at the same support.
For each mined pattern P we compute support in failure windows
(support_failure), support in control windows (support_control), lift =
support_failure / pooled_support(P), and relative risk = P(failure | P)
/ P(failure | ¬P). Every mined sequence is also scored against the
ITEMSET COUNTERPART of the same event set; the difference
order_gain = sequence_lift - itemset_lift quantifies how
much preserving order contributes above co-occurrence.
Every phase carries pre-declared invariants whose expected outcome is stated up front. Itemset mining checks that a random-label permutation at the same min_support does not yield a top lift within a factor of 1.5× of the real top lift; a violation would indicate either data leakage or an over-sensitive support threshold. Sequence mining checks that a within-window random order permutation preserves top itemset lift (unchanged, by construction) but strictly reduces top sequence lift on rich horizons (windows with >= 3 events on average). Both invariants pass on the two traces where the method yields wins; the boundary traces are diagnosed via these invariants rather than by post-hoc justification.
The matched-control design in §4.1 assumes we can define a “no-failure” control window on the same or another entity at the anchor time. For right-censored survival-style data (traces where entities exit observation upon repair or dropout), naive matching biases scoring because “did we observe a failure” becomes entangled with “how long did we observe the truck”. Component X in §3.4 has exactly this problem: short-observation trucks have 12% failure rate; long-observation trucks have 5%.
Following the epidemiological literature on incidence-density sampling (Prentice and Breslow 1978; Rothman-Greenland ch. 15), we replace §4.1’s sampler with a risk-set matched design: for each case with observed failure time T_f, controls are drawn from the risk set at T_f (the set of entities still under observation at that lifetime index) and their windows are aligned to T_f rather than to their own end-of-observation. Both case and control windows use the last K events with time_step < T_f. Under this sampling, the pooled 2 x 2 odds ratio of a mined pattern (case-in vs case-out; control-in vs control-out, Woolf-Haldane 0.5-continuity-corrected, 95% CI via log-OR variance) estimates the per-pattern hazard ratio rather than a prevalence lift. A mined pattern with MH-OR > 1 and 95% CI excluding 1 is a censoring-valid signal of elevated failure risk, not an artefact of the observation process.
The rest of the pipeline runs unchanged: FP-Growth on the risk-set windows, min-support 0.05, BH FDR correction on the p-values induced by the Fisher-exact null of the same 2 x 2 table. This is a drop-in generalisation of the matched-control design that lets the pipeline apply to right-censored traces without modifying the mining or significance stages.
Mining patterns from failure windows and then computing hypergeometric p-values on the same failure/control counts yields post-selection- invalid marginal p-values: the candidate set is chosen because it has high support in failure windows, so the fixed-hit-set null is not the null the pipeline actually operates under. BH or BY correction on those marginal p-values does not restore validity (Fithian, Sun and Taylor 2014; Loftus and Taylor 2015).
Our fix, applied to every trace: split the training set entity- disjoint into a 50% discovery half and a 50% inference half. FP-Growth / PrefixSpan run on the discovery half and produce a candidate pattern universe C. Every P in C is then scored on the inference half via exact hypergeometric p-value on that half’s case/control hit counts alone. Because C is chosen without touching the inference half, the resulting p-values are marginally valid and BH / BY correction on the family {p(P) : P in C} controls the inference-half FDR honestly.
Entity-disjoint splitting is used rather than random per-window splitting because otherwise the same entity’s windows could appear on both sides of the split, leaking information from discovery into inference.
The pooled 2x2 odds ratio previously used to score risk-set-matched
SCANIA patterns discards the matched-set structure that makes the
estimator censoring-valid. The standard estimator under incidence-
density (risk-set) sampling is a conditional logistic regression
stratified by matched set, equivalent to the sampled Cox partial
likelihood restricted to the pattern indicator (Prentice-Breslow 1978;
Langholz-Goldstein 1996). We estimate every SCANIA pattern’s coefficient
with statsmodels.ConditionalLogit stratified by
match_id; each case and its three risk-set-matched controls
form one stratum. Reported hazard ratios and 95% CIs are from that
matched fit.
Compared to the pooled 2x2 Woolf-Haldane analysis, the matched estimator is roughly 1.6× more conservative on this data: the same top pattern that scores MH-OR 2.72 [2.10, 3.51] under pooling scores HR 1.73 [1.53, 1.96] under proper matched conditional logistic.
The naive order_gain = sequence_lift − itemset_lift
compares a sequence like M → M → M against its itemset
counterpart {M}, which collapses three occurrences to one
presence. That conflates temporal order with event multiplicity. Our
count-preserving comparator (§4.7) shuffles the ordering within each
window while preserving the exact event multiset per window, then
rescores the sequence’s support on the shuffled corpus. The residual
order_effect = real_lift − mean(count-preserving-shuffle_lift)
isolates the pure ordering contribution.
For each mined pattern we compute an exact one-sided hypergeometric p-value on the observed failure-hit count against the label-permutation null with the pattern hit-set fixed. Under H0 the number of hits landing in the failure class is Hypergeom(N_F+N_C, hit_F+hit_C, N_F); the upper-tail probability of the observed hit count IS the label-permutation p-value, so we compute it in closed form. Benjamini-Hochberg FDR correction (Benjamini and Hochberg 1995) is applied per (horizon × pattern class) to give q-values.
Windows are split temporally by anchor timestamp. Cutoffs: Azure 2015-09-01, Alibaba 2018-01-07, BGL 2005-11-01, SCANIA 2020-01-01. Mining runs on training windows only; surviving patterns become binary presence features on both train and test. A logistic regression fit on train is evaluated on test for four feature sets:
For each configuration we report AUROC, AUPRC, F1 / precision / recall at threshold 0.5, and lead time (anchor − last_event_ts) on true-positive failure windows. Numbers are computed in a single pass per configuration and stored as one artifact, so a comparison across feature sets on the same trace cannot drift.
Time horizons of 1h and 6h leave 99.6% and 98% of failure windows
empty. Effectively no Azure PdM failure is preceded by an event in the
same hour. Useful horizons are 24h and the count-based
(last5, last10). Even at 24h, failure and
control windows separate cleanly by raw event count (failure mean 1.58
events, control mean 0.077).
Figure: azure_window_horizon_vs_events.png
A min_support sweep over {0.02, 0.05, 0.10, 0.15} preserves the headline ordering at every operating point:
| horizon | metric | 0.02 | 0.05 | 0.10 | 0.15 |
|---|---|---|---|---|---|
| 24h | combined | 0.996 | 0.996 | 0.996 | 0.996 |
| 24h | itemsets_only | 0.996 | 0.996 | 0.996 | 0.996 |
| last5 | combined | 0.815 | 0.810 | 0.803 | 0.774 |
| last5 | itemsets_only | 0.761 | 0.754 | 0.762 | 0.754 |
| last10 | combined | 0.664 | 0.696 | 0.751 | 0.741 |
| last10 | itemsets_only | 0.578 | 0.643 | 0.686 | 0.674 |
At every min_support tested, combined dominates itemsets_only by at least +4 AUROC points at last5 and at least +5 at last10.
Figure: azure_sensitivity_min_support.png
At Azure 24h,
{software_error:error2, software_error:error3} reaches lift
3.99 (present in 38.2% of failure windows, in 0.04% of controls;
P(failure | pattern) = 99.6%). All six 24h itemsets dominate the
random-label permutation null (permuted top 1.24).
At Azure last5/last10, the sequence
maintenance:comp4 → software_error:error2 → software_error:error3
reaches lift 3.73 as an ordered pattern but only 2.22 as the same items
unordered. Ordered patterns ending ... → error2 → error3
dominate the top-8 at both count-based horizons.
On Alibaba last5, the strongest ordered pattern is
task_success:M → task_success:R → task_success:M → task_success:M
with sequence lift 2.43 versus itemset lift 0.94 for the same event set
(order gain +1.49). At last3,
task_success:M → task_success:M → task_success:M reaches
sequence lift 3.06 vs itemset lift 1.37 (order gain +1.69): three
consecutive Map completions predict a subsequent failure much more
strongly than the mere presence of Map events would suggest.
Figure: azure_itemset_vs_sequence_lift.png
Head-to-head on temporally-held-out test sets:
| trace | horizon | event_count | itemsets_only | sequences_only | combined |
|---|---|---|---|---|---|
| Azure | 24h | 0.97 / 0.91 | 0.996 / 0.99 | n/a | 0.996 / 0.99 |
| Azure | last5 | 0.50 / 0.34 | 0.75 / 0.56 | 0.66 / 0.56 | 0.81 / 0.72 |
| Azure | last10 | 0.50 / 0.34 | 0.64 / 0.50 | 0.67 / 0.53 | 0.70 / 0.58 |
| Alibaba | last3 | 0.69 / 0.50 | 0.75 / 0.44 | 0.50 / 0.20 | 0.81 / 0.63 |
| Alibaba | last5 | 0.60 / 0.50 | 0.67 / 0.34 | 0.51 / 0.21 | 0.74 / 0.57 |
| Alibaba | last10 | 0.59 / 0.50 | 0.68 / 0.36 | 0.52 / 0.23 | 0.74 / 0.59 |
| BGL | last5 | 0.50 / 0.25 | 0.49 / 0.25 | n/a | 0.49 / 0.25 |
| BGL | last10 | 0.50 / 0.25 | 0.49 / 0.25 | 0.50 / 0.25 | 0.50 / 0.25 |
| BGL | last20 | 0.50 / 0.25 | 0.48 / 0.25 | 0.50 / 0.25 | 0.51 / 0.26 |
| SCANIA | last5 | 0.50 / 0.09 | 0.52 / 0.11 | n/a | 0.52 / 0.11 |
| SCANIA | last10 | 0.50 / 0.09 | 0.60 / 0.14 | 0.55 / 0.11 | 0.60 / 0.15 |
| SCANIA | last20 | 0.50 / 0.09 | 0.57 / 0.14 | 0.53 / 0.10 | 0.57 / 0.13 |
(AUROC / AUPRC on the temporally-held-out test set. SCANIA uses per-vehicle 90th-percentile-delta binning; BGL uses episode-anchored per-rack windows with alerts removed from the pre-alert stream. Both boundary traces sit near chance for every feature set at every horizon.)
Figure: four_dataset_predictive_comparison.png
The naive sequence_lift − itemset_lift metric confounds
ordering with event multiplicity: comparing M → M → M to
its itemset {M} compares “three occurrences” to “presence
of one M”. Our count- preserving comparator (§4.7) shuffles the ordering
within each window while preserving its exact event multiset, then
rescores the sequence’s support on the shuffled corpora. The residual
order effect = real_lift − mean(count-preserving-shuffle_lift)
isolates the pure ordering contribution.
Top-20 sequences per horizon:
| trace | horizon | real lift | count-preserving null lift | order effect |
|---|---|---|---|---|
| Azure | last5 | 2.85 | 2.33 | +0.52 |
| Azure | last10 | 3.01 | 1.92 | +1.09 |
| Alibaba | last3 | 1.91 | 1.93 | −0.02 (null) |
| Alibaba | last5 | 1.73 | 1.71 | +0.02 (null) |
| Alibaba | last10 | 2.09 | 1.85 | +0.24 |
Reading these against the naive order_gain values (up to
+1.69 on Alibaba last3), the count-preserving comparator
shows that essentially all of the reported Alibaba “order gain” was a
multiplicity effect: the same event multiset in ANY order carries
approximately the same lift as the specific ordered sequence. Azure
error2 → error3-terminating sequences retain a genuine
ordering effect of +0.5 to +1.1 lift units above the count-preserving
null. Order is a real signal on Azure and essentially not a signal on
Alibaba once multiplicity is controlled for.
The paper’s central object is the fraction of mined frequent patterns that pass the per-pattern statistical significance test on an inference sample disjoint from the discovery sample used for mining. Frequency alone is a weak proxy for predictiveness; the post- selection-valid test tells us which mined patterns carry elevated failure signal against matched controls under an inference regime not contaminated by the label-dependent candidate selection.
Every trace is split 50/50 entity-disjoint into a discovery half (for FP-Growth candidate selection) and an inference half (for exact hypergeometric p-values and BH / BY correction).
| trace | horizon | disc entities | patterns mined on disc | sig BH q<0.05 | sig BY q<0.05 |
|---|---|---|---|---|---|
| Azure | 24h | 50 | 8 | 8 (100%) | 8 (100%) |
| Azure | last5 | 50 | 79 | 52 (66%) | 46 (58%) |
| Azure | last10 | 50 | 815 | 379 (46%) | 241 (30%) |
| Alibaba | last3 | ~1.12M jobs | 11 | 6 (55%) | 5 (45%) |
| Alibaba | last5 | ~1.12M jobs | 11 | 5 (45%) | 5 (45%) |
| Alibaba | last10 | ~1.12M jobs | 11 | 5 (45%) | 5 (45%) |
| BGL | last5 | 32 | 2 | 1 (50%) | 1 (50%) |
| BGL | last10 | 32 | 3 | 1 (33%) | 1 (33%) |
| BGL | last20 | 32 | 3 | 1 (33%) | 1 (33%) |
| SCANIA | last5 | 11,775 | 66 | 9 (14%) | 8 (12%) |
| SCANIA | last10 | 11,775 | 597 | 41 (7%) | 6 (1%) |
| SCANIA | last20 | 11,775 | 37,797 | 0 (0%) | 0 (0%) |
Two consequences of post-selection-valid inference. First, the Azure
last10 fraction drops from 86% (naive) to 30% (BY-corrected
on inference half); the extra patterns were selection artefacts. Second,
the SCANIA last20 fraction drops from 6.0% (naive) to 0%
(post- selection valid): the 42,000-pattern mining run at min-support
0.05 does not survive an honest inference test. Both direction and
magnitude of these shifts match the pre-registered concern about
mining-and-testing on the same sample.
The Azure / Alibaba wins persist in weaker but still substantive form (46-100% at rich horizons); SCANIA under post-selection-valid inference no longer supports an aggregate “some fraction is predictive” claim on this mining threshold and requires the matched conditional-logistic analysis in §6.6 instead. On the two winning traces the majority of frequent patterns are also predictive; on the two boundary traces the majority are frequent-but-noise, and the paper’s concrete predictive-pattern list is short.
At BH q < 0.05: every Azure 24h itemset (6/6) and every Azure 24h
sequence (7/7) is significant; 53/77 Azure last5 itemsets
and 55/67 Azure last5 sequences; 562/657 Azure
last10 sequences. Both 1h and 6h Azure horizons flag zero
patterns as expected (0/3 sequences at 1h, 0/5 at 6h). On Alibaba: 6/10
last3 itemsets, 9/16 last3 sequences, 59/109
last10 sequences.
Applying the risk-set matched-sampling extension from §4.4 to SCANIA
Component X (2,272 cases x 3 controls each drawn from the risk set at
each case’s failure lifetime), FP-Growth at min-support 0.05 mines
42,453 candidate itemsets from the counter_surprise event
stream.
We estimate the per-pattern hazard ratio via conditional
logistic regression stratified by matched risk set (§4.6),
fitted with statsmodels.ConditionalLogit. On the top-200
patterns by case-hit count, 121 (60.5%) pass the joint criterion
HR CI excludes 1 AND p < 0.05. Top hazard ratios cluster in
feature-397 bin combinations, consistent with the concentration of
predictive signal in a small number of underlying histograms.
Top 5 predictive Component X signatures (matched HR, 95% CI, p):
| HR | 95% CI | p | n_case | n_control | pattern |
|---|---|---|---|---|---|
| 1.73 | [1.53, 1.96] | 1.0e-17 | 456 | (matched) | counter_surprise:397_{27, 28, 29} |
| 1.69 | [1.50, 1.91] | 4.1e-17 | 479 | (matched) | counter_surprise:397_{29, 34, 35} |
| 1.69 | [1.49, 1.91] | 3.7e-17 | 495 | (matched) | counter_surprise:397_{28, 29, 34} |
| 1.67 | [1.50, 1.87] | 2.1e-19 | 611 | (matched) | counter_surprise:397_{29, 34} |
| 1.66 | [1.48, 1.85] | 2.1e-19 | 630 | (matched) | counter_surprise:397_{28, 29} |
The pooled 2 x 2 analysis previously reported inflated these effects by roughly 1.6x (top pooled MH-OR 2.72 [2.10, 3.51] for the same pattern), because pooling discards the matched-set structure that the incidence-density sampling design creates. The matched HR of 1.73 is the correct estimator under Prentice-Breslow, and its tighter CI reflects that the matched design conditions out inter-vehicle heterogeneity the pooled analysis conflated with the pattern effect.
Top hazard ratios are concentrated in histogram-397 bin combinations. Grouping matched-significant patterns by their dominant histogram feature: feature 397 supplies the majority of significant patterns; features 158 and 167 contribute a smaller number of independent signatures. The “N distinct predictive patterns” reading of the matched output must therefore be interpreted with awareness of this underlying feature-set concentration.
Top 5 predictive Component X signatures (MH-OR, 95% CI):
| MH-OR | 95% CI | n_case | n_control | pattern (event_subtype_seq) |
|---|---|---|---|---|
| 2.72 | [2.10, 3.51] | 114 | 130 | counter_surprise:397_{10, 27, 28, 29} |
| 2.65 | [2.07, 3.39] | 122 | 143 | counter_surprise:{158_3, 397_28} |
| 2.63 | [2.05, 3.38] | 118 | 139 | counter_surprise:397_{10, 27, 29, 34} |
| 2.44 | [1.98, 3.01] | 165 | 212 | counter_surprise:397_{10, 28, 29, 34} |
| 2.37 | [1.94, 2.89] | 183 | 243 | counter_surprise:{158_9, 309_0} |
The top signatures are concentrated in bin combinations of the same
histogram feature (397), consistent with the §3.4 note that Component X
features encode 6 histograms. The 158_9 + 309_0
cross-feature signature is an example of a two-feature interaction that
pattern mining surfaces without needing a black-box classifier. Despite
these interpretable hazard-ratio-scored patterns, none of them lift a
temporally-held-out logistic regression beyond AUROC 0.60 (per §6.2):
the patterns are per-truck static discriminators, not temporal
precursors that a next-K-event alarm can act on. The distinction is
important operationally: hazard-ratio-scored patterns support
cohort-level fleet triage (which trucks warrant closer inspection), not
next-event alerting.
Lead-time detail: results/tables/{azure,alibaba}_leadtime.md.
Given a mined sequence S with lift L(S), we ask: does any proper subsequence S’ ⊂ S in the mined set already reach lift(S)? If so, the predictor lives in the subpart and S is redundant; if no proper subsequence matches, the FULL ordered sequence is the minimal predictor. Formally, S is “full-sequence-dominant” iff lift(S) > lift(S’) + 0.05 for every proper subsequence S’ the miner also produced.
Top-200 sequences per horizon:
| trace | horizon | full-seq dominant | subpart dominant | fraction full |
|---|---|---|---|---|
| Azure | last5 | 41 | 11 | 79% |
| Azure | last10 | 191 | 9 | 96% |
| Alibaba | last3 | 2 | 10 | 17% |
| Alibaba | last5 | 5 | 20 | 20% |
| Alibaba | last10 | 31 | 74 | 30% |
The finding is trace-dependent:
last10, 96% of top-200 predictive sequences are
full-dominant, meaning no proper subsequence they contain reaches their
lift. Concretely,
maintenance:comp4 → software_error:error2 → software_error:error3
reaches lift 3.73; the best proper subseq
software_error:error2 → software_error:error3 has lift 2.55
(delta +1.18). The full ordered chain adds real signal beyond any of its
parts. Operationally, an alarm should be keyed on the full ordered
sequence, not on any two-event fragment of it.task_waiting:R alone (lift ~3.98). Once a Waiting task
appears in a job, the failure risk is set; adding subsequent Success
events to the pattern does not lift it further. Operationally, the
useful alarm is short.This resolves an ambiguity the raw order-gain distribution left open. Order helps on both traces (§6.3), but for different reasons: on Azure the full ordering contributes signal beyond every subpart, and on Alibaba the ordering just distinguishes one privileged short prefix from bag-of-items noise.
On Azure PdM,
software_error:error2 → software_error:error3 at
last5 reaches sequence lift 3.73 vs itemset lift 2.22 for
the same items. The order-specific reading is that error2 and error3 are
not interchangeable noise: a machine reporting error2 first and then
error3 is materially more likely to reach a
terminal_failure than one that reports them in the other
order. In practical monitoring, an alarm keyed on the pair-in-order is
preferable to the same alarm keyed on the pair-as-set.
On Alibaba,
task_success:M → task_success:M → task_success:M at
last3 reaches sequence lift 3.06 vs itemset lift 1.37 for
the same items. Three consecutive Map completions predict a subsequent
Failed task more strongly than “the job contains Map completions” alone.
The operational reading is that the position of the failure inside the
DAG matters: jobs that make it through a Map-heavy prefix are the jobs
whose downstream Reduce or Join phases can fail, whereas jobs that fail
early do so in a different distribution of task types.
Four heterogeneous traces are useful empirical evidence, and they inform hypotheses about when pattern mining recovers meaningful predictive structure and when it does not. They are not a sufficient basis to infer a general “regime of validity” for the method: four case studies differ along too many axes (domain, event vocabulary, synthetic vs real, entity definition, target construction, observation cadence, class prevalence, control sampling) to isolate the causal factors that separate the two positive traces (Azure PdM, Alibaba v2018) from the two boundary traces (BGL, SCANIA Component X). We describe the pattern rather than claim it as a rule.
The four-trace survey resolves an obvious follow-up question: does the sequences+itemsets combined-feature-set advantage transfer to any operational event log? It does not.
Concretely: on BGL the best combined AUROC across horizons is 0.51 (chance) even when INFO-level messages and component granularity are included in the non-alert stream; on SCANIA the best combined AUROC across horizons is 0.60, unchanged when the fleet-wide 90th-percentile delta binning is replaced by per-vehicle-normalised 90th-percentile binning.
To determine whether the SCANIA gap is representation-loss (tokens destroy signal that is actually in the trace) or signal-absence (no representation could recover 0.75), we run a diagnostic ceiling test: LightGBM on a compact set of histogram-aware distributional descriptors (Wasserstein-1 distance to a per-vehicle baseline, signed centroid shift, entropy shift, tail-mass shift, and each descriptor’s slope over the last 20 readouts) computed on the same temporal split. The ceiling model reaches AUROC 0.60 / AUPRC 0.04, essentially identical to the pattern-mining pipeline. Logistic regression on the same 113 structured features reaches 0.58 / 0.05. The GBM-versus-LR gap is 0.02 AUROC, so classifier capacity is not the constraint either. The gap between SCANIA (~0.60) and the Azure / Alibaba wins (~0.80-1.00) reflects a limit of the readout cadence and feature vocabulary, not of the pattern-mining pipeline against a richer alternative representation.
Positive control on same-manufacturer data. To rule out the alternative explanation that SCANIA-family telemetry itself lacks predictive signal, we apply the same LightGBM ceiling test to SCANIA APS Failure at Scania Trucks (Costa and Nascimento 2016) (UCI 421, IDA 2016 industrial challenge). APS Failure uses the same anonymised histogram-encoded schema as Component X (7 histogram groups of 10 bins each + 100 single counters = 170 feature columns) but delivers one per-truck cross-sectional readout instead of a longitudinal readout stream, with a binary APS-system-failure label at a 1.67% positive rate. On the same LightGBM configuration our Component X ceiling used, APS Failure reaches AUROC 0.994 / AUPRC 0.934 on the canonical held-out test split (16,000 trucks, 375 positives); LR alone reaches 0.979 / 0.800. Same manufacturer, same anonymisation schema, different readout format, near-perfect predictability. This excludes “SCANIA-family data is inherently weak” as an explanation for the Component X boundary.
Root-cause diagnosis: trajectory-signal absence, not signal
absence. A stratified 5-fold cross-validation on the same 420
aggregated Component X features (mean / max / std / last of each of the
105 columns per vehicle), with each column linearly residualised against
length_of_study_time_step inside every fold, reaches AUROC
0.826 ± 0.005, comparable to Alibaba’s last3 combined score
(0.81) and BGL’s chance (0.51). The same LightGBM configuration under
the temporal split used elsewhere in the paper only reaches 0.67, and
the pattern-mining pipeline reaches 0.60. Component X features therefore
carry substantial per-truck failure signal, but the signal is a static
per-vehicle profile (aggregate usage, cumulative counter shape) rather
than a temporal degradation trajectory. Last-K-events windows and the
ordered-pattern mining built on them cannot see it, because there is no
pre-failure event ordering to catch; the discriminative information is
spread across the truck’s entire operating history.
This gives a sharper three-way regime-of-validity: (i) two wins (Azure PdM, Alibaba v2018), where target failure is preceded by a discriminable ordered event trajectory; (ii) BGL, where the target class is self-triggering with no discriminable non-alert precursor; (iii) Component X, where the target has strong per-truck signal in aggregate features but no last-K-events trajectory signal, so pattern mining on windows attains the temporal-split ceiling but not the transductive per-vehicle ceiling. The APS positive control confirms that (iii) is a target-shape distinction, not a manufacturer or schema deficiency.
The method’s regime of validity is therefore “trace has a rich native discrete event vocabulary AND failure class is not self-triggering AND readout-cadence signal capacity exceeds the target AUROC bar”. BGL fails the second condition; SCANIA fails the third; Azure PdM and Alibaba v2018 satisfy all three.
The two lead-time regimes in §6.7 speak to deployment. Azure inherits a structural 24h clock from the synthetic generator and should not be read as a real-world warning interval; Alibaba’s median 0-second lead time is the honest one, and a per-job classifier there must be paired with sub-second scheduling infrastructure to act on the signal at all.
batch_task
alone; the batch_instance table (21 GB compressed) that
would enable per-machine failure trajectories on the same trace is left
to future work.All code, mined-pattern parquets, discovery/inference splits, matched
hazard-ratio outputs, and end-to-end reproduction scripts are released
at the paper’s GitHub repository
(https://github.com/ApartsinProjects/FailurePatterns) with the rendered
manuscript hosted at
https://apartsinprojects.github.io/FailurePatterns/. The repository
carries: (i) per-dataset ingest scripts referencing the exact public
sources for Azure PdM (Kaggle mirror), Alibaba cluster-trace-v2018
(Alibaba OSS batch_task.tar.gz, ~130 MB), BGL (Loghub
Zenodo 8196385/BGL.zip, ~55 MB), and SCANIA Component X
(Swedish National Data Service DOI 10.5878/jvb5-d390,
CC-BY-4.0); (ii) window construction, mining (mlxtend FP-Growth 0.25.0
for itemsets, SPMF 2.64 via subprocess for PrefixSpan sequences),
scoring, discovery/ inference splitting, and matched
conditional-logistic scripts; (iii) all reported numbers
programmatically verified by scripts/audit_paper_numbers.py
against the artefact JSON/parquet files (numbers audit: 100% of claims
pass at last release); (iv) scripts/publish_paper.sh
regenerates the manuscript HTML and DOCX from Markdown source with
pandoc + citeproc + the bibliography, and copies the HTML into
docs/index.html for GitHub Pages.
Python 3.14.3 pinned via requirements.txt; Java 21
required for SPMF invocation. Random seeds fixed at 20260828 for every
stochastic step (discovery/inference split, control sampling,
permutation tests, LightGBM). Wall-clock: the full pipeline (ingest,
windows, mining, significance, matched hazard, predictive eval) runs in
under one hour on a single CPU-only workstation for Azure PdM, Alibaba
batch_task, BGL, and SCANIA combined, excluding the initial trace
downloads.
Frequent-pattern mining of discrete operational events surfaces interpretable pre-failure signatures on two of four traces studied. On both winning traces, sequences add real predictive information beyond itemsets when window definitions are rich enough for order to be a real degree of freedom; at those horizons, combining itemset and sequence features improves failure prediction by 5-10 AUROC points over either alone. The result replicates across a synthetic per-machine trace (Azure PdM) and a real per-job production trace (Alibaba v2018), and its regime of validity is mapped by two additional traces (BGL, SCANIA) where the pipeline does not find signal, with a mechanistic explanation for each.
batch_instance.paper-build skill once prose is
settled.bibtest. Expansion
pending scout return.