Uncertainty Quantification and Calibration Research into uncertainty quantification establishes that confidence in neural network predictions is increasingly important, yet basic neural networks do not deliver this confidence ^[1]. Deep learning methods operate as black boxes where uncertainty is challenging to quantify, and Bayesian statistics offer a formalism to address this ^[9]. However, neural networks intrinsically increase uncertainty about which features of the analysis are model-related and which are due to the neural network itself ^[3]. Uncertainty evaluation is core for detecting uncertain data in practical applications, particularly when encountering unexpected samples ^[2]. Standard backpropagation suffers from disadvantages such as a lack of calibrated probabilistic predictions ^[14]. While Bayesian neural networks promise improved generalization under covariate shift by providing principled probabilistic representations of epistemic uncertainty, weight-based BNNs often struggle with high computational complexity ^[4]. Alternative approaches like stochastic weight sharing reinterpret quantization techniques to reduce computational overhead, enabling efficient Bayesian training of large-scale models ^[5]. Furthermore, structured partial stochasticity has been proposed to simplify posterior distributions by removing neuron permutation symmetries ^[15]. For data with limited observations, such as in macroeconomic analysis, specific BNN architectures have been developed to handle temporal dependence and small sample sizes ^[13]. Robustness and Optimization under Shift Regarding robustness, approximate Bayesian inference considered robust can achieve poor performance under covariate shift, particularly where linear dependencies cause a lack of posterior contraction ^[10]. Pretrained Transformers have been systematically measured for out-of-distribution generalization, showing smaller performance declines than previous models on NLP datasets ^[6]. Training deep neural networks is complicated by the change in distribution of layer inputs during training, a phenomenon addressed by normalizing layer inputs to reduce internal covariate shift ^[7]. Weight noise injection training has been proposed to achieve strong robustness, making network weights insensitive to modest changes ^[12]. Entropy has also been used to measure a model's uncertainty in distributing probability mass over sequences, with regularization encouraging distribution over smaller subsets of allowed alignments ^[11]. Bayesian optimization methods using flexible parametric models stay close to a truly Bayesian treatment while obtaining scalability through stochastic gradient Hamiltonian Monte Carlo ^[8]. However, the literature reports that the approximations needed to make Bayesian methods tractable are where the guarantees are lost, precisely the regime this study operates in. While node-based approaches induce uncertainty via latent variables ^[4], no study explicitly maximizes the entropy of multiplicative weight noise distributions to improve subject-disjoint generalization in Transformers against standard variational objectives. This work positions itself against these findings by testing whether explicit entropy maximization, distinct from posterior fidelity objectives, provides the promised robustness in attention-based architectures.

The core of this experiment is the EntropyMaxWeightNoise operator, designed to test the hypothesis that explicit entropy maximization of weight uncertainty improves robustness. This operator wraps the query projection layer of the Transformer encoder. For each seed, identical model weights are initialized to ensure comparability across conditions. The operator introduces multiplicative noise to the weights during the forward pass, parameterized by a learnable uncertainty parameter. Unlike standard variational objectives that minimize the Kullback-Leibler divergence to approximate a posterior, the intervention adds a loss term that explicitly maximizes the entropy of the noise distribution. This design choice isolates the effect of diversity maximization from posterior fidelity. The hypothesis posits that maximizing this entropy will encourage the model to explore a wider range of functions, thereby improving generalization to unseen subjects. The comparator, the baseline condition, uses standard deterministic training with identity control, representing the current standard practice. To ensure that any observed effects are due to entropy maximization rather than noise injection per se, we include two negative controls. The first negative control minimizes the entropy of the noise distribution, testing whether low diversity harms performance. The second negative control uses fixed noise without entropy optimization, testing whether the act of injection alone drives changes. The operator is implemented as a custom PyTorch module. During training, the noise parameters are updated via gradient ascent on the entropy term, while the weights are updated via gradient descent on the classification loss. This adversarial setup ensures that the noise distribution becomes as uniform as possible within the constraints of the learning rate. The comparator neutralizes this by fixing the noise parameters to zero (deterministic) or optimizing them for minimum entropy. This method directly expresses the hypothesis that diversity, quantified by entropy, is the mechanism for robustness. If the hypothesis holds, the intervention should outperform the baseline on held-out data. If the hypothesis is false, the additional optimization pressure may destabilize training or degrade feature learning. The analysis computes paired contrasts between the intervention and the baseline for each seed. This paired design controls for seed-specific variance, increasing the power to detect the effect of the operator. The preregistered decision rule applies a fixed worthwhile margin to the mean contrast. This margin ensures that only practically significant improvements are considered positive. The method does not pool primary and replication cohorts; each is analyzed separately to confirm reproducibility. This approach adheres to the preregistered plan without post-hoc adjustment to fit the results.

Operator specification. EntropyMaxWeightNoise is applied to blocks.0.attn.q_proj.weight. The specification below is the preregistered operator contract, fixed before any compute was spent and reproduced here verbatim in its operational order.

Algorithm 1 gives the operator in full.

Comparator arm — the standard optimizer update runs at every step exactly as in the intervention arm. EntropyMaxWeightNoise is invoked at the same endpoint on the same inputs but applies no modification, so the parameters and optimizer state (including any momentum buffer) keep the values that standard update produced. Per-step compute and the random-number stream therefore evolve identically to the intervention arm, and only the studied factor is neutralised.

The experiment utilizes the UCI Human Activity Recognition Using Smartphones dataset, employing the official subject-disjoint split of 7,352 training and 2,947 held-out windows. This split is critical because it tests generalization to unseen individuals, whereas a random split would allow the model to memorize subject-specific artifacts. A random split would have hidden the distribution shift this study aims to address, as training and test data would share underlying subject characteristics. By using a subject-disjoint split, we ensure that the held-out accuracy reflects true out-of-distribution robustness. We employed 43 primary seeds and 43 disjoint confirmatory-replication seeds. This number of seeds provides sufficient power to estimate the mean contrast and its confidence interval with precision. Pairing every condition on one seed buys control over initialization variance, ensuring that differences are due to the training condition rather than random weight initialization. The cost is increased computational load, but this is necessary for a rigorous test of the hypothesis. The analysis aggregates accuracy per seed and computes paired contrasts. The model architecture is a PreNormTransformerEncoder with two blocks, using 4-head self-attention over 128 timesteps. This architecture was chosen to test the hypothesis in a modern attention-based setting, distinct from the convolutional domains where weight noise has been previously explored. The training budget is fixed at 12 epochs with a batch size of 128 and SGD optimizer. This fixed budget ensures that differences in performance are not due to varying convergence times. The first negative control rules out the possibility that noise injection itself is beneficial regardless of entropy. If the first negative control performs similarly to the baseline, it suggests that noise without entropy maximization is neutral. The second negative control rules out the possibility that entropy minimization is the driving factor. If the second negative control performs similarly to the baseline, it confirms that the deterministic baseline is stable. Without these controls, it would be indistinguishable whether performance changes were due to entropy maximization or simply the presence of noise. The design justifies the conclusion that any deviation in the proposed condition is specifically due to the entropy maximization objective. Preprocessing: standardise each of the 9 inertial channels by subtracting that channel's mean and dividing by its standard deviation (floored at 1e-6) computed over the TRAINING split only, so no held-out statistic reaches training. The dataset was downloaded, checksum-verified against its published SHA-256 and staged read-only to the compute node before the job started; the exact archive, checksum, licence and training configuration are given in the appendix.

The results are reported separately for the primary and confirmatory-replication cohorts to adhere to the preregistered analysis plan. In the primary cohort, across 4 conditions and 43 seeds, the proposed condition worsened the contrast in the hypothesised direction by 0.04805 versus the baseline condition (95% CI [-0.0714, -0.0324], paired randomisation p=0.0002, Cohen's dz=-0.72). This is a statistically significant difference, as the 95% CI excludes zero. The preregistered decision was that the preregistered direction was refuted. Figure 1 displays the final metric values per condition with 95% bootstrap confidence intervals over seeds, showing the proposed condition at 0.8604 [0.837, 0.877] compared to the baseline at 0.9085 [0.905, 0.912]. The negative controls performed comparably to the baseline, with the first negative control at 0.9082 [0.905, 0.911] and the second negative control at 0.909 [0.906, 0.912]. Figure 2 shows the mean training loss over training steps for each condition. The proposed condition exhibits higher training loss compared to the baseline, suggesting that the entropy maximization objective interfered with convergence on the classification task. This aligns with the lower held-out accuracy. The negative controls track the baseline loss closely, indicating that noise injection without entropy maximization does not disrupt training dynamics. In the confirmatory replication cohort, across 4 conditions and 43 seeds, the proposed condition worsened the contrast in the hypothesised direction by 0.03482 versus the baseline condition (95% CI [-0.0422, -0.0278], paired randomisation p=0.0002, Cohen's dz=-1.41). This is also a statistically significant difference, with the 95% CI excluding zero. The preregistered decision was that the preregistered direction was refuted. Figure 3 displays the final metric values for the replication cohort, with the proposed condition at 0.8717 [0.865, 0.877] and the baseline at 0.9065 [0.903, 0.91]. The negative controls again matched the baseline, with the first at 0.908 [0.904, 0.912] and the second at 0.9088 [0.905, 0.912]. Figure 4 shows the mean training loss for the replication cohort, confirming the pattern observed in the primary cohort. The proposed condition consistently underperformed the baseline across both cohorts. The effect size was larger in the replication cohort (dz=-1.41) than in the primary cohort (dz=-0.72), indicating a robust negative effect. The consistency between cohorts strengthens the conclusion that the intervention is detrimental. The results table summarizes the held-out classification accuracy for all conditions, confirming that the baseline and negative controls are statistically indistinguishable from each other, while the proposed condition is significantly lower. Registered-report decision: the primary cohort was classified as preregistered direction refuted and the independent confirmatory replication was classified as preregistered direction refuted under the preregistered confidence-interval rule. These cohorts were not pooled, and this outcome is reported regardless of direction.

The findings of this study refute the hypothesis that explicit entropy maximization of weight uncertainty parameters improves out-of-distribution accuracy on subject-disjoint time-series data. In both primary and replication cohorts, the proposed condition significantly worsened performance compared to the deterministic baseline. This result contradicts the expectation that diversity maximization alone would enhance robustness. Instead, it suggests that entropy maximization without posterior fidelity constraints disrupts the learning process in Transformers. These results relate directly to the reviewed prior work on node-based BNNs. While node-based BNNs have been shown to perform well under covariate shift due to input perturbations ^[4], our findings indicate that simply maximizing the entropy of latent variables does not translate to weight noise in attention layers. The literature warns that approximate Bayesian inference considered robust can achieve poor performance under covariate shift ^[10]. Our results support this warning, showing that a simplified entropy objective fails to capture the benefits of full Bayesian inference. The degradation in performance aligns with the observation that weight-based BNNs often struggle with computational complexity and convergence ^[5], though here the issue appears to be optimization stability rather than cost. The negative controls provide crucial context. The first negative control (entropy minimization) and the second negative control (fixed noise) performed comparably to the baseline. This isolates the entropy maximization objective as the cause of degradation. It suggests that noise injection itself is not harmful, but optimizing for maximum entropy is. This contrasts with work proposing weight noise injection training to achieve strong robustness ^[12], implying that the method of noise injection matters. In diffractive neural networks, noise makes weights insensitive to changes ^[12], but in Transformers, maximizing entropy may prevent the weights from settling into optimal configurations. The results also inform the debate on uncertainty quantification. Basic neural networks do not deliver certainty estimates ^[1], but our attempt to fix this via entropy maximization failed. This supports the view that Bayesian statistics offer a formalism to understand uncertainty ^[9], but approximations must be carefully chosen. The alignment entropy regularization used in speech recognition encourages distribution over smaller subsets of alignments ^[11], whereas our maximization encouraged broader distribution, which proved detrimental. This suggests that uncertainty should be structured, not merely maximized. For practitioners, these results indicate that entropy maximization should not be prioritized for robustness tasks in Transformers over standard regularization. The core mechanism of node-based BNNs ^[4] may not generalize to weight noise in attention layers. Future work should investigate whether structured entropy constraints, rather than global maximization, can recover the benefits without the performance cost. The consistency across 86 seeds total provides high confidence in this negative result.

This study is limited to the UCI Human Activity Recognition dataset and a specific Transformer architecture. While the subject-disjoint split ensures robust testing, results may not generalize to other time-series domains or larger-scale Transformers. The entropy maximization operator was applied only to the query projection layer; applying it to all layers might yield different results, though computational costs would increase. The training budget was fixed at 12 epochs; longer training might allow the model to recover from the entropy pressure, but this was not tested. The negative controls were limited to entropy minimization and fixed noise; other variational objectives were not compared. Finally, the study focuses on accuracy; uncertainty calibration metrics were not the primary endpoint, though accuracy degradation suggests calibration would also suffer. Declared scope. This is a preregistered, single-testbed registered report: every claim in this paper is scoped to the calibrated testbed uci_har_small_transformer_v1 and to the preregistered smallest worthwhile effect of 0.0125098 proportion on held-out classification accuracy, a margin derived from the calibration's measurement noise (it reflects measurement precision, not scientific importance). Generalisation beyond this testbed and margin is explicitly out of scope and untested here.

This study tested whether explicit entropy maximization of weight uncertainty improves out-of-distribution accuracy in Transformers under subject-disjoint shift. The preregistered hypothesis was refuted in both primary and replication cohorts, with the intervention significantly worsening accuracy compared to the deterministic baseline. Negative controls confirmed that noise injection alone is not detrimental, isolating entropy maximization as the cause. These findings indicate that diversity maximization without posterior fidelity constraints is insufficient for robustness in attention-based architectures. Practitioners should avoid explicit entropy maximization for weight noise in this regime and prioritize standard regularization. Future work should explore structured uncertainty objectives that balance diversity with convergence stability.