Keywords: central-bank communication; monetary policy; large language models; text as data; pre-registration; training-data contamination; Southeast Asia. JEL: E52, E58, C45, C55, G14.

Disclaimer. This is an independent open-source project. It uses only documents published on central banks’ public websites (or public mirrors of them), represents the views of no institution, and is not investment advice. The author is employed by a central bank; nothing here draws on non-public information, and the text makes no comment on any bank’s policy choices, only on measurable properties of published text and on statistical results. The Bank of Thailand adapter is opt-in and excluded from all headline results.

1 Introduction

After every policy meeting a central bank publishes a decision statement and, in most cases, minutes and a press conference. Reducing that text to a scalar, a tone running from dovish to hawkish, has become a standard operation in academic research and in market practice. The first generation of measures counted words [1], [2]; the second trained supervised classifiers on hand-labelled sentences [3]; the third, now spreading rapidly, simply asks a large language model (LLM) to read the document and report a stance [4], [5]. The third generation is attractive because it needs no labelled data, no lexicon and no language-specific engineering, and because models small enough to run on a desktop now read economic prose fluently. It also raises three questions that the literature, overwhelmingly concerned with the Federal Reserve and the ECB and almost entirely with English, has not answered together.

The first question is whether the measurement carries policy signal: does knowing the tone of today’s statement improve a forecast of the next decision beyond what the last decision already implies, and does the unexpected component of tone explain the market’s reaction on the day? The second is whether the measurement is stable: a tone index is only a measurement if a harmless rephrasing of the prompt, or a point release of the model, leaves it substantially unchanged. The third is whether the measurement is contaminated: a model whose training data extend to 2025 has read commentary on the very decisions a historical test asks it to predict, so an apparently predictive score may be recall rather than reading [6], [7].

This paper answers all three for seven central banks in three languages, with a design that was fixed before the data were examined. The headline set comprises the Federal Reserve (Fed), the European Central Bank (ECB), the Bank of England (BoE), the Reserve Bank of Australia (RBA), the Monetary Authority of Singapore (MAS), Bank Indonesia (BI) and Bank Negara Malaysia (BNM). Three of the seven publish parallel native-language and English versions of their statements, which affords a direct test of cross-lingual consistency that, to our knowledge, has not been attempted for central-bank text. The scorer at the centre of the study is deliberately modest: qwen3:4b, a four-billion-parameter open-weight model served locally through Ollama, chosen because it is what an analyst at an institution without a cloud budget, or with a prohibition on sending documents to an external service, can actually run.

We make three contributions. First, we release cbtone, an Apache-licensed Python library that fetches public central-bank communications politely, versions them as a content-hashed corpus, and scores them with four scorer families (a cited hawkish/dovish dictionary, a finance-tuned transformer, a local LLM and a small fine-tuned multilingual encoder) behind a single interface, so that the comparisons below can be repeated or extended to other banks in a few lines of configuration. Second, we report a pre-registered study (git tag prereg-v1) in which the five research questions, their hypotheses and thresholds, the minimum sample per cell, the multiple-testing families and the placebo designs were frozen before any scorer read the full corpus; a result that fails its threshold is reported as refuted, not re-specified. Third, we treat training-data contamination as a first-class threat: every LLM result is re-estimated on masked text, in which bank, committee and instrument names, countries, dates and rate levels are replaced by placeholders, and on a post-cutoff window of documents released after the model’s public release date, and a finding is called robust only if it survives both.

The results are uniformly negative, and we think that is the paper’s value. LLM tone does not improve out-of-sample prediction of the next decision over a persistence baseline at any of the four adequately powered banks; the tone surprise explains the announcement-window move beyond its placebo in a single bank–market cell and never beats the dictionary after correction; native-language and English versions of the same statement correlate only moderately; and five paraphrases of the same prompt produce indices whose agreement falls short of the pre-registered reliability floor. The negative findings are sharpened rather than weakened by the contamination protocol, because masking leaves the point estimates almost unchanged: there is no memorised signal to remove. Nor is the failure one of the instrument: a supervised encoder replicated from the FOMC literature, which reads human-labelled Fed sentences far more accurately than the LLM, does no better on the same tests. What the local LLM does do is read individual FOMC sentences better than a dictionary, which is consistent with the published literature and is precisely the kind of claim that does not generalise to policy signal without the tests reported here.

The remainder of the paper is organised as follows. Section 2 situates the study in the literature on tone measurement, LLM annotation and announcement effects. Section 3 describes the corpus, the decision and market data, and the labels. Section 4 sets out the pre-registered design. Sections 5 to 7 report the results on policy signal, on reliability and contamination, and on transfer to Southeast Asia. Section 8 discusses the findings, Section 9 the threats to validity, and Section 10 the reproducibility of every number; Section 11 concludes.

2 Related literature

2.1 Measuring the tone of central-bank text

The measurement of central-bank tone has moved from counting words to classifying them, and so far almost exclusively for the Federal Reserve. Lucca and Trebbi [1] and Apel and Blix Grimaldi [2] count hawkish and dovish terms in FOMC statements and Riksbank minutes respectively; Loughran and McDonald [8] show why generic sentiment lexicons mislead on financial text, and Picault and Renault [9] build a field-specific lexicon for the ECB. Tone dictionaries have also been applied to speeches to study the home bias of European central bankers [10]. Topic models exploit deliberation transcripts to separate information from forward guidance [11] and to study how transparency changes what committee members say [12], while voice-tone measures extend the lens beyond the written record [13]. Supervised transformers came next: FOMC-RoBERTa, trained on 2,480 hand-labelled FOMC sentences, reaches a weighted F1 of 0.72 on the held-out split against about 0.50 for a rule-based baseline and 0.59 for zero-shot ChatGPT-3.5 [3], and CentralBankRoBERTa classifies economic sentiment rather than policy stance [14]. The Fed is also the only bank with a public human-labelled hawkish/dovish set, so outside the Fed no scorer has been checked against human judgement.

2.2 LLMs as annotators, and the contamination problem

Asking an LLM to annotate text works, but how well depends on the setting. LLM annotation can match or beat crowd workers on cost and agreement for political text [15]; zero-shot reading of FOMC stance by an early GPT model is modest [4]; and the one study spanning many banks and languages finds a fine-tuned multilingual encoder far ahead of GPT-4o on stance classification [5]. Models can also be probed for the expectations they hold [16] and used to forecast returns from headlines [17]. Two reliability concerns follow directly. Prompt format alone can move accuracy by tens of points [18], and a recent application at the Reserve Bank of Australia stresses that “reasoning” about central-bank text must be made trustworthy before it is operationalised [19]. Ludwig, Mullainathan and Rambachan [20] give the econometric framing we adopt: an LLM output used as a regressor is a noisy, possibly non-classical measurement whose error structure must be characterised before downstream inference is interpreted.

Contamination is both real and measurable. LLMs recall macroeconomic outcomes almost perfectly before their training cutoff and at near-chance rates after it [6]; masking entity names changes GPT-based sentiment signals in-sample [21]; and formal tests of look-ahead bias in pretrained models are available [7]. We found no study that combines entity masking with a post-cutoff split on central-bank text, which is what our RQ4c does.

2.3 Announcement effects and the Southeast Asian evidence

The event-study template is well established. Kuttner [22] identifies the policy surprise from federal funds futures; Gürkaynak, Sack and Swanson [23] separate a target factor from a “path” factor that captures what the statement says; Bernanke and Kuttner [24] quantify the equity response; and Blinder et al. [25] survey why communication moves markets. Text-based surprise measures explain much less of daily moves than rate surprises do; changes in statement wording reach a pseudo-R² of at most 0.12 [26]. We therefore pre-registered small R² values as the relevant benchmark rather than as a disappointment. A recent Federal Reserve note comparing LLM and lexicon tone across many banks reports only moderate agreement between the two [27], which anticipates our RQ4 results.

Evidence for Southeast Asia is thin. For Malaysia there is a yield-curve event study of rate surprises [28]; for the region there are transparency surveys [29]; and the emerging-market communication literature more broadly covers Fed spillovers to emerging equity markets [30], the link between transparency and market expectations [31], and expectation formation in Brazil [32]. Singapore’s exchange-rate-centred regime is described by Parrado [33], but we found no study linking MAS statement text to the Singapore dollar, and none comparing tone between native-language and English versions of the same central-bank document.

3 Data

3.1 Corpus

Adapters fetch each bank’s public pages under its robots.txt, with a rate limit of at least two seconds per host and a descriptive User-Agent, caching every response with its SHA-256 digest. MAS serves a bot interstitial to automated clients and pbc.gov.cn disallows crawlers; with the author’s approval both are read from Internet Archive copies and never from the banks’ own sites. The snapshot used here (2026-10-14+ed5a49c28f7b41acaf4f21bda0d43570bd23049b363717c08d5efb3b9177bb8b) holds 2,935 documents. Each bank’s primary decision series was fixed in the pre-registration: FOMC statements; ECB monetary-policy decision press releases; BoE Monetary Policy Committee minutes (the Bank’s 2000–2006 decision releases are a single sentence and carry no tone); RBA statements from December 2007 (earlier releases exist only when the cash rate changed); MAS Monetary Policy Statements; and BI and BNM statements (English primary, with Indonesian and Malay twins used for RQ3). Text is committed to the repository only where the source’s terms permit redistribution (Fed, ECB decision releases, RBA); for the other banks the repository holds URLs and hashes, and make fetch rebuilds the text locally.

3.2 Policy decisions

Rate decisions are taken from the BIS central-bank policy-rate series and matched to meeting dates from the corpus; a meeting at which the rate is unchanged is a hold. MAS sets a path for the trade-weighted exchange rate rather than an interest rate, so its decisions (tighten, maintain, ease) are taken from the Authority’s own published decision table; a rule-based extractor exists for statements not yet in that table, but its accuracy is development accuracy only and it is used for two 2026 statements that are excluded from every test.

3.3 Market data

Daily exchange-rate closes come from Yahoo Finance, with bad ticks removed by a rule fixed before any test was run, and two-year government-bond yields from a free official daily source where one exists (United States, euro area, United Kingdom, Australia). Moves are measured between the closes of t − 1 and t + 1 around the release; a document without a published release time is timed at 00:00 local time on its release date, so that the window always contains the event. Exchange rates for Singapore, Indonesia and Malaysia are USD crosses, signed so that a positive move is a domestic appreciation.

3.4 Human and silver labels

Human labels exist only for the Fed: the hawkish/dovish test split of Shah, Paturi and Chava [3], which is used for evaluation only under its non-commercial licence and never for training. For Southeast Asia, 200 sentences (MAS 60; BI 35 English and 35 Indonesian; BNM 35 English and 35 Malay) carry silver labels produced by two LLM passes with different prompts and a tie-break; they are reported as silver throughout and are never treated as ground truth. Table 1 summarises the sample behind every test.

Table 1. Sample behind each test. Decisions are meetings with a matched policy decision; OOS, hikes and cuts are the out-of-sample decisions predicted by the expanding-window model in RQ1 and the hikes and cuts among them (for MAS, tightenings and easings of the exchange-rate path). A bank is powered for RQ1 when it has at least 40 OOS decisions with at least 8 hikes and 8 cuts. FX and two-year-yield events are announcements with a usable market window; “—” marks series for which no free daily source exists.

4 Empirical design

4.1 Scorers

Every scorer returns a probability vector over {hawkish, dovish, neutral} and an index τ = p_hawkish − p_dovish ∈ [−1, 1], computed on the same input: the decision, forward-guidance, outlook and risk paragraphs of a document selected by a rule-based tagger and capped at 6,000 characters. The dictionary (dict-v1) applies a 167-term lexicon assembled from the cited hawkish/dovish word lists with negation handling; it reads English only. The LLM is qwen3:4b run locally through Ollama at temperature 0 with a registered prompt (P1) and schema-constrained JSON output; every call is cached by content hash, so a re-run repeats no inference. The fine-tuned scorer is XLM-R base trained for at most 30 minutes on the LLM’s own sentence labels for corpus sentences, never on the FOMC data, so that it measures how much of the LLM’s reading a small multilingual encoder can absorb. FOMC-RoBERTa, the in-distribution reference, is our replication of the model of Shah, Paturi and Chava [3]. The authors’ checkpoint is gated and access was not granted. We therefore fine-tuned RoBERTa-large on the train split of their data with their recipe: a 20 % validation split, learning rate 10⁻⁵ and early stopping on validation macro-F1. Three seeds were trained and the seed with the best validation score was kept, so the test split played no part in model selection. The weights are released at https://huggingface.co/thanatchon36/cbtone-fomc-roberta-repl under CC-BY-NC-4.0, the licence of the training data, as a replication that carries none of the authors’ weights; readers with access to the original should prefer it. The replication reads English only and enters every analysis in which the pre-registration names the transformer scorer: the FOMC test split (H4b), the Southeast Asian silver sets (H5) and, as a descriptive comparison outside the Holm families, the next-decision and announcement-window analyses (RQ1, RQ2).

4.2 Pre-registration and hypotheses

The pre-registration (docs/RESEARCH_QUESTIONS.md at tag prereg-v1) fixed the five research questions, the hypotheses in Table 2, the sample minimum for each (bank, question) cell, the Holm families and the placebo designs. Deviations made after the tag are logged with their rationale in docs/DECISIONS.md (D17–D21) and concern compute allocation, document selection and the replacement of the gated reference model by a replication (D21), not thresholds or tests.

Table 2. Pre-registered hypotheses, their thresholds and the verdicts reported in Sections 5–7. Every verdict is read from results/summary.json.

4.3 Next-decision prediction (RQ1)

For each bank b with decisions y_t ∈ {cut, hold, hike} at meetings t = 1, …, T_b, an ordered-logit model Pr (y_t + 1 ≤ k ∣ x_t) = Λ(κ_k − x_t^′β),   x_t = (τ_t, Δτ_t, d_t), is refitted at every decision on an expanding window (initial 40 % of the sample, at least 30 decisions) and used to predict the next decision, where τ_t is the tone of the document released at t, Δτ_t its change from the previous document and d_t the direction of the last decision. The baseline is persistence, a smoothed estimate of Pr (y_t + 1 ∣ y_t) from the same window. Performance is the multiclass log-loss per decision, and the quantity of interest is the gain ΔLL_b = LL_b^reference − LL_b^model, positive when the tone model is better. Its sampling distribution is obtained by a moving-block bootstrap over out-of-sample decisions (block length 4); Holm’s correction is applied across the adequately powered banks; and a success additionally requires the real gain to exceed the 95th percentile of two placebo distributions, obtained by shuffling the dates of the tone series and by permuting the decision labels (1,000 permutations each). The same procedure is run with the dictionary as the reference, so that the LLM is compared both with a trivial forecast and with a simple text measure.

4.4 Announcement-window regressions (RQ2)

The tone surprise s_t = τ_t − τ̂_{t ∣ t − 1} is the residual from an AR(1) fitted to the tone series of that bank using documents released before t only. For each bank and market series the two-day move r_t around the release is regressed on the surprise, r_t = α + βs_t + ε_t, with Newey–West (HAC) standard errors. H2a requires the R² to exceed the 95th percentile of the same regression run on non-announcement days and a Holm-adjusted one-sided test on β in the hypothesised direction (a hawkish surprise appreciates the currency and raises the two-year yield); H2b compares the LLM’s R² with the dictionary’s by paired block bootstrap. Cells with fewer than 30 events are reported but excluded from the Holm family.

4.5 Cross-lingual consistency (RQ3)

For BI and BNM every statement is published in the national language and in English. The LLM scores both versions independently, and consistency is measured by the Spearman correlation of the two indices over parallel pairs and by the stance-flip rate, the share of pairs whose argmax labels are hawkish in one language and dovish in the other. H3 requires ρ ≥ 0.70 and a flip rate of at most 10 %. PBoC, which publishes Chinese and English, is exploratory because it has too few parallel pairs.

4.6 Reliability (RQ4a, RQ4b)

Stability is measured with Krippendorff’s α (interval) and ICC(2,1) across five registered paraphrases of the prompt, two temperatures (0 and 0.7), two model tags (qwen3:4b and qwen2.5:3b) and three strategies for aggregating sentence scores into a document score (mean, decision-weighted, maximum absolute index), each compared with the direct document score, on 120 documents and 1,000 sentences. Agreement with human labels is the three-class macro-F1 on the 496-sentence FOMC test split, with a bootstrap interval over sentences and a paired comparison with the dictionary.

4.7 Contamination (RQ4c)

Every LLM score is recomputed on masked text, in which a deterministic rule replaces the names of banks, committees, instruments and countries, all dates and all rate levels with typed placeholders, and every RQ1 and RQ2 estimate is recomputed on the post-cutoff window of documents released after the model’s public release date (2025-04-29). A finding from RQ1 or RQ2 is robust only if it keeps its sign and significance under both re-estimations. Because the model cannot have read commentary on post-cutoff decisions, and because masking removes the handles through which a memorised outcome could be retrieved, a signal that survives both is unlikely to be recall.

4.8 Transfer to Southeast Asia (RQ5)

Each Fed-calibrated scorer’s macro-F1 on the FOMC test split is compared with its macro-F1 on the silver sets per bank and language; H5 predicts a drop of at least 0.10 for MAS, whose exchange-rate framing is furthest from the Fed’s. The test was pre-registered with FOMC-RoBERTa as the reference scorer and a minimum of 50 sentences per cell. As a descriptive complement, we report the mean cosine distance between each bank’s sentences and Fed sentences in a multilingual sentence-embedding space (sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2).

4.9 Power, multiplicity and compute

Each (bank, question) cell has a pre-set minimum sample; underpowered cells are reported with their estimates but excluded from the Holm families and from majority counts, and a majority verdict requires at least three powered banks. All LLM scores come from one backend, the same qwen3:4b weights (digest 359d7dd4) served by Ollama on a desktop RTX 4070 Ti, on which the full pre-registered workload fits the two-hour LLM budget without invoking the pre-registered compute ladder: scoring the primary documents took 4 minutes, the masked re-scoring 3 minutes and the stability perturbations 71 minutes. The fine-tuned encoder trained in 6 minutes on an Apple M4 laptop. On that laptop the LLM processes about 220 prompt tokens per second, roughly 3.5 seconds per document at a 3,000-character cap, so a laptop-only replication requires the ladder described in Section 10.

5 Results: policy signal

5.1 The tone series

Figure 1 plots the LLM index of each bank’s primary document over the sample. The series are visibly plausible: the 2022–2023 tightening cycle appears as a sustained hawkish plateau at every bank that publishes a full statement, the 2008–2009 and 2020 easings as dovish troughs, and the exchange-rate-framed MAS statements as a smoother and more muted series. Two features matter for what follows. First, the ECB’s decision press releases before 2016 are short and formulaic, so the index alternates between neutral and the extremes rather than evolving gradually; second, for BI the series begins only in 2016, because the bank’s own index lists nothing older. Plausibility of this kind is the weakest test a tone index can pass, and the sections that follow ask whether anything more is true.

5.2 Does tone predict the next decision? (RQ1)

H1 is refuted. 4 banks meet the pre-registered power rule (Fed, ECB, RBA and BI). The LLM succeeds at none of them (Table 3, Figure 2).

Table 3. Next-decision prediction at the adequately powered banks. ΔLL = LL(reference) − LL(model) in multiclass log-loss per decision; positive favours the scorer. Intervals from a moving-block bootstrap. LLM rows: p is Holm-adjusted across the four banks and the last column reports the pre-registered refuting experiment. FOMC-RoBERTa (replication) rows are the pre-registered descriptive comparison: p is unadjusted and the placebos were not run.

The ECB is the only bank at which the LLM’s gain over persistence has a bootstrap interval that excludes zero and exceeds both placebo distributions (95th percentiles 0.014 for shuffled tone and 0.010 for permuted labels); the unadjusted p-value is 0.032 and the Holm-adjusted value 0.13, so the finding does not survive the pre-registered correction. The masked LLM gives an almost identical ECB estimate (0.059 [0.008, 0.111]), which is consistent with the gain being a property of the text rather than of memorised outcomes, but does not rescue it. At the RBA it is the dictionary, not the LLM, whose interval lies above zero (ΔLL 0.043 [0.002, 0.087]); the dictionary was not a registered hypothesis and we report it only as a reminder that the ranking of scorers is bank-specific. The BoE (only 5 out-of-sample hikes), MAS (29 predictions) and BNM (7 hikes) are underpowered and appear in Figure 2 with hollow markers.

Two further observations qualify the headline. Ranking metrics are high for every scorer, including the persistence baseline (Fed persistence AUROC 0.77, ECB 0.85), because most of the information about the next move is already in the last one: policy rates move in runs, and a model that knows the direction of the last decision ranks the next correctly most of the time. The LLM raises the ECB AUROC to 0.91, which is the same near-miss seen in log-loss. Second, BI’s masked-text AUROC of exactly 1.00 rests on 11 hikes and 9 cuts and is a small-sample artefact discussed in ANOMALIES.md (A9), not evidence of perfect ranking.

The pre-registration also asks that the transformer scorer be reported with the same comparisons, as a descriptive result. The FOMC-RoBERTa replication reads human-labelled Fed sentences with a macro-F1 of 0.71 against the LLM’s 0.48 (Section 6.2), and does not translate that advantage into prediction (Table 3, Figure 2). At the powered banks its gain over persistence has a bootstrap interval that excludes zero at 0 of 4. The closest are the RBA (0.041 [-0.003, 0.087], unadjusted p 0.07) and the ECB (0.038 [-0.005, 0.098], p 0.09); at the Fed, whose statements supplied its training sentences, the gain is 0.007 [-0.018, 0.026]. Its gain over the dictionary excludes zero at 0 of the four. A scorer that is far better calibrated to human judgements of individual sentences is therefore no better at the task the index is meant to serve, which locates the negative result in the information content of a scalar tone measure rather than in the LLM’s reading of it; Section 8 returns to this.

5.3 Does the tone surprise explain the market reaction? (RQ2)

H2a and H2b are refuted. Of 11 adequately powered bank–market cells, the LLM tone surprise explains the release-window move beyond its placebo in 1 (Table 4, Figure 3), and its R² exceeds the dictionary’s, after Holm correction, in 0.

Table 4. Announcement-window regressions of the market move on the LLM tone surprise, all powered cells. FX β is the signed two-day log move (positive = domestic appreciation) per unit surprise; yield β is in basis points per unit surprise. Newey–West standard errors; Holm correction across the eleven cells.

The single success is the Australian two-year yield: a one-unit hawkish surprise is associated with a 9.0 basis-point rise (R² 0.10 against a placebo 95th percentile of 0.029). Its R² gain over the dictionary has a bootstrap interval above zero ([0.001, 0.225]) but a Holm-adjusted p of 0.43, so H2b fails even there. The magnitude is economically sensible: the R² is of the order reported for wording-based surprises in the literature [26], and far below what rate surprises achieve [23]. For BI the LLM coefficient is statistically distinguishable from zero but carries the sign opposite to the hypothesis (β -0.0039, unadjusted p 0.007): more hawkish surprises coincide with rupiah depreciation over the day-level window. The pre-registered one-sided test does not count this as a success, and because BI publishes no release time the design cannot distinguish a reaction from the statement from a statement that responds to the day’s market move (ANOMALIES.md, A10). Pooled across banks with bank fixed effects, the FX coefficient is 0.0005 [-0.0010, 0.0020] (n = 1,179), indistinguishable from zero.

The pre-registered comparison of the LLM against the transformer scorer finds no difference. Across the 11 powered cells, the LLM’s R² gain over the FOMC-RoBERTa replication ranges from -0.014 (AUD/USD, RBA) to 0.035 (USD/IDR, BI), and its bootstrap interval excludes zero in 0 of them. The replication’s own announcement-window results mirror the LLM’s: its R² exceeds the non-announcement placebo in 2 of 11 cells, both at the RBA, where the two-year yield responds by 19.0 basis points per unit surprise (R² 0.07, placebo 95th percentile 0.029) and the AUD/USD R² of 0.025 clears its placebo of 0.023 by a hair. The better-calibrated instrument finds the same single market, and no more (Figure 3).

5.4 Do native-language and English versions agree? (RQ3)

H3 is refuted. On parallel statements the LLM’s native-language and English indices have a Spearman correlation of 0.61 for BI (Indonesian, n = 128) and 0.53 for BNM (Malay, n = 139), both below the pre-registered 0.70. The disagreement concerns intensity rather than direction: the stance flips between hawkish and dovish in only 2.3% and 2.2% of pairs, well within the 10 % ceiling, whereas the mean absolute gap between the two indices is 0.28 and 0.18 on a [−1, 1] scale (Figure 4). The fine-tuned multilingual encoder is no more consistent (ρ 0.44 and 0.54), which suggests that the inconsistency is not peculiar to generative decoding. The dictionary and the FOMC-RoBERTa replication are English-only and therefore not applicable, and PBoC, with 9 parallel pairs, is exploratory. Figure 4 makes the structure of the disagreement visible: both languages place most statements in a mildly dovish band, but the Indonesian versions are read as more dovish than the English ones in the lower-left quadrant, and the Malay versions cluster more tightly around neutral.

6 Results: reliability and contamination

6.1 Stability across prompts, temperatures, model versions and aggregation (H4a)

H4a is refuted. Table 5 and Figure 5 report Krippendorff’s α and ICC(2,1) for each perturbation. Across the five registered prompt paraphrases, document scores reach α = 0.65 and ICC(2,1) = 0.66 on 120 documents, short of the pre-registered 0.667 and 0.75; on 1,000 sentences α falls to 0.42. Temperature matters much less (α = 0.92 for documents and 0.81 for sentences at 0 against 0.7), which indicates that the instability is not sampling noise but a genuine dependence of the reading on the wording of the instruction. Replacing the model tag with qwen2.5:3b changes scores most of all: α = 0.46 for documents and -0.03 for sentences, the latter no better than chance agreement. Aggregating sentence scores into a document score yields indices that correlate with the direct document score at ρ between 0.62 and 0.72, so not every pair of strategies clears 0.70; the mean and decision-weighted aggregates are near-identical to each other (ρ = 0.98), as they must be by construction (ANOMALIES.md, A11).

Table 5. Agreement between scores of the same text under each perturbation, on 120 documents and 1,000 sentences; α is Krippendorff’s interval α and ICC is ICC(2,1). Pre-registered floors: α ≥ 0.667 and ICC(2,1) ≥ 0.75. 982 of the 1,000 sentences have a complete set of prompt scores; the remainder returned malformed JSON in at least one paraphrase and are treated as missing.

The practical reading is that a tone index from a small local model is, in part, a property of the prompt and of the model version. Two analysts using different but equally reasonable instructions will obtain series that agree only moderately, and a series computed before and after a model upgrade is not the same measurement. The low agreement between LLM and lexicon tone reported across many banks by Xu and Vilán [27] is of the same order as our prompt-paraphrase α, which suggests that the choice of prompt is as consequential as the choice between an LLM and a dictionary.

6.2 Agreement with human labels (H4b)

H4b is refuted. On the 496-sentence FOMC test split the LLM’s three-class macro-F1 is 0.48 [0.43, 0.53] (Table 6, Figure 6). It exceeds the dictionary’s by 0.21 [0.16, 0.26], a difference whose interval lies well away from zero, but it misses the pre-registered floor of 0.50, so the hypothesis fails on its first clause. The fine-tuned XLM-R, trained only on the LLM’s labels for corpus sentences and never on FOMC data, reaches 0.41 [0.36, 0.45], which shows that a 270-million-parameter encoder can absorb most of the LLM’s reading at a fraction of the inference cost.

Table 6. Three-class macro-F1 on the FOMC test split (496 sentences), with bootstrap intervals over sentences. Pre-registered floor 0.50. ᵃ Trained on the train split of the same data, so in-distribution (Section 4.1).

Our dictionary scores 0.27, well below the roughly 0.50 reported for a rule-based baseline by Shah, Paturi and Chava [3]. The two rule sets differ, and ours returns neutral whenever hawkish and dovish matches balance or neither appears, which is the case for almost every FOMC test sentence; the dictionary is therefore a floor rather than a competitive baseline on sentence-level data, even though it is competitive on document-level prediction (Section 5.2). The LLM also over-predicts the neutral class relative to the human labels, which depresses its macro-F1 relative to its accuracy.

Our FOMC-RoBERTa replication reaches 0.71 [0.67, 0.75], against the 0.72 weighted F1 the authors report [3]. Across the 3 training seeds its test macro-F1 is 0.70 ± 0.02, so the selected seed is typical and not a lucky draw. Removing the 17 test sentences that also occur in the train split changes its score only to 0.71. The model is trained on the same distribution as the test split, so it is a ceiling for this split and not a competitor. What it shows is the size of the gap: a supervised encoder of 355 million parameters, trained on about 1,600 labelled sentences, beats the zero-shot four-billion-parameter LLM on Fed text by a wide margin (0.71 against 0.48). Section 7 asks how much of that advantage survives outside the Fed.

6.3 Contamination (H4c)

No RQ1 or RQ2 finding is robust, for the simple reason that none is established in the first place. Table 7 reports the pooled estimates on original text, on masked text and on the post-cutoff window. Pooled across the powered banks, the LLM’s log-loss gain over the dictionary is 0.007 [-0.016, 0.031] on original text and 0.004 [-0.017, 0.028] on masked text: there is no effect to preserve, so the masked check cannot pass. The post-cutoff window (after 2025-04-29) holds 58 decisions and 78 FX events; its estimates overlap the full-sample intervals but are too wide to be informative on their own.

Table 7. Contamination protocol. The same pooled quantities re-estimated on entity-masked text and on documents released after the model’s public release date; n is given for the original, masked and post-cutoff estimates in turn.

The informative part of the exercise is what masking does to the point estimates: almost nothing. The ECB gain is 0.058 on original and 0.059 on masked text, and the pooled FX coefficient moves from 0.0005 to 0.0002. Had the LLM been retrieving memorised outcomes through the names and dates in the text, removing them would have degraded the original-text estimates; it does not. This is consistent with, though it does not prove, the absence of outcome memorisation in a four-billion-parameter model reading statements of this kind, and it implies that the negative results above are not an artefact of contamination either.

7 Transfer to Southeast Asia

H5 is untestable as pre-registered, and could not have been supported. The BI-English and BNM-English silver subsets have 35 sentences each, below the pre-registered minimum of 50, so the comparison of MAS with BI and BNM cannot be made. The other clause can be checked on the adequately powered MAS cell, and it fails: H5 needs both Fed-calibrated scorers to transfer poorly to MAS, and the dictionary does not.

The FOMC-RoBERTa replication does lose accuracy outside the Fed. On the 60 MAS silver sentences its macro-F1 is 0.47 against 0.71 on the FOMC test split, a drop of 0.25, beyond the pre-registered 0.10. The two underpowered English cells point in different directions: the drop is 0.27 for BNM and 0.02 for BI. So the MAS drop is not the largest, contrary to H5’s prediction. Part of each drop is error in the silver labels rather than loss of transfer. Those labels come from the LLM, which itself reaches only 0.48 on human FOMC labels, so a scorer can be penalised for disagreeing with a wrong silver label. The numbers are evidence that the supervised Fed model is less accurate on MAS text. They do not measure how much less.

Table 8 reports the full descriptive comparison. On the 60 MAS silver sentences the dictionary’s macro-F1 is 0.36 against 0.27 on FOMC sentences, so it does not lose performance on MAS (drop -0.09). The LLM and the fine-tuned encoder score higher on the silver sets than on the human FOMC labels (0.64 and 0.67 on MAS), but the silver labels were produced by the same model family, so these numbers measure self-agreement rather than accuracy and support no claim about transfer. The two silver passes agree on 71% of sentences, and the silver distribution is skewed towards dovish and neutral (80 dovish, 17 hawkish, 103 neutral of 200).

Table 8. Descriptive transfer comparison. Silver macro-F1 is against LLM-produced silver labels and, for the LLM and the fine-tuned encoder, measures self-agreement rather than accuracy. FOMC-RoBERTa reads English only. Drop = FOMC macro-F1 − silver macro-F1; the pre-registered hypothesis predicts a drop of at least 0.10 for MAS.

In the multilingual sentence-embedding space, MAS sentences are the farthest from Fed sentences (mean cosine distance 0.58, against 0.54 for the ECB and 0.43 within the Fed’s own sentences), consistent with the distinct vocabulary of an exchange-rate-centred regime [33]. The supervised Fed model loses accuracy on that text, and the dictionary does not. A human-labelled Southeast Asian set of adequate size is what would turn these descriptive drops into a test.

8 Discussion

Five findings deserve interpretation beyond the verdicts.

Persistence is the baseline to beat, and it is a strong one. Most of what can be predicted about the next decision is in the last decision, because policy rates move in runs. The ordered-logit model with tone, its change and the last direction never beats a smoothed persistence forecast after correction, and the one near-miss, the ECB, has an unadjusted p of 0.032 across 178 out-of-sample decisions. This is not a statement that statements are uninformative; it is a statement that the information a small LLM extracts from them, compressed to a scalar, adds little to a trivial forecast of the direction of the next move. Text may well inform the size and timing of moves, or the probability of a change at all, none of which the pre-registered test measures.

The announcement-window result is of the expected order. The Australian two-year yield responds to the tone surprise by about 9 basis points per unit of surprise, with an R² near 0.10. That is the magnitude the wording-based literature leads one to expect [26], an order of magnitude below the explanatory power of rate surprises [23]. The pre-registration anticipated small R² values and set the bar at beating a placebo rather than at a size; one cell in eleven cleared it, which after correction for eleven tests is what chance would deliver. We do not think the Australian result is spurious, but we also cannot claim it.

Cross-lingual disagreement is about intensity, not sign. The LLM reads the Indonesian and Malay versions of a statement as pointing the same way as the English version in roughly 98 % of cases, but assigns different strengths. For an analyst who needs a categorical stance this is reassuring; for one who needs an index it means that the native-language and English series are not interchangeable, and that the choice of language is a modelling decision to be reported. The fine-tuned encoder shows the same pattern, so the disagreement is not an artefact of generative decoding; it is more plausibly a property of how a small multilingual model represents evaluative language in the two tongues.

Prompt dependence is the binding reliability constraint. The agreement between five paraphrases of the same instruction (α = 0.65 at document level) is lower than the agreement between temperatures and comparable to the agreement between an LLM and a lexicon reported elsewhere [27]. In the framework of Ludwig, Mullainathan and Rambachan [20], the LLM index is a measurement with substantial and possibly non-classical error, and the magnitude of that error depends on a design choice that is rarely reported. The implication for practice is direct: a tone series is a measurement only when the prompt and the model digest are frozen and logged, and a change in either is a change of instrument.

The binding constraint is the signal, not the instrument. A natural reading of the negative results is that a four-billion-parameter model reads tone too noisily and that a better reader would find the signal. The FOMC-RoBERTa replication tests that reading. It agrees with human labels on Fed sentences at a macro-F1 of 0.71, against the LLM’s 0.48, yet it improves on persistence at none of the powered banks, the Fed included, and its announcement-window R² is indistinguishable from the LLM’s in every powered cell (Sections 5.2 and 5.3). Two instruments that differ greatly in how well they read individual sentences agree closely on how little a scalar tone index adds to a persistence forecast or to a two-day market move. The limiting factor is what a single hawkish–dovish number can carry about a decision that is mostly determined by the last one, not the model that produces the number. This does not rule out richer text measures, such as sentence-level or topic-specific tone, which this design does not test.

What, then, is a local LLM good for? It reads individual sentences better than a dictionary, with a macro-F1 that is close to the pre-registered floor and above the rule-based figure in the published literature; it produces series that track the broad policy cycle; and it does so in seconds per document, on hardware that costs less than a year of a commercial data subscription, with no text leaving the machine. These are the properties of a monitoring aid. They are not, on this evidence, the properties of a forecasting or trading signal, and the contamination protocol shows that the absence of signal is not because the model has been prevented from cheating.

9 Threats to validity

• Small samples for Southeast Asian banks. BI’s text begins in April 2016 because the bank’s own index lists nothing older; BNM changes its rate rarely; MAS met only twice a year until 2024. Several cells therefore fall below the pre-registered minimum and are reported as underpowered rather than tested, and the study is correspondingly less informative about the region than about the four large banks.
• Silver labels. The Southeast Asian evaluation set is labelled by the same model family as the LLM scorer. The LLM’s own silver score is reported but is used for no hypothesis, and the fine-tuned encoder’s silver score inherits the same circularity.
• A short post-cutoff window. The model’s public release date is late in the sample, so the post-cutoff check is wide and often inconclusive; an inconclusive check means a finding is not established as robust, which is the conservative reading.
• Document selection. One primary series per bank; the BoE uses minutes because its early decision releases carry no text; the RBA series starts in December 2007. A different choice of document could give different tone series, which the library makes easy to test but this paper does not.
• Timing. MAS and BI publish no release time, so their event windows span the whole release day and cannot separate the market’s reaction to the statement from the statement’s reaction to the market.
• MAS decisions. Taken from the Authority’s own table; the rule-based extractor’s accuracy is development accuracy only and affects two 2026 statements that are excluded from every test.
• Replicated reference model. The authors’ FOMC-RoBERTa is gated and access was not granted, so the reference scorer is our replication of their recipe (D21). It reaches the published level on the FOMC test split, but it is not the same set of weights, and it was trained without the authors’ grid over learning rate and batch size. Readers can check it directly: the weights are released under the data’s own non-commercial licence. Its RQ1 and RQ2 results are descriptive and outside the Holm families, as pre-registered. It was trained on Fed sentences only, so a weaker showing at other banks would be expected; that it does not beat persistence at the Fed either is the informative part.
• Compute and model scale. The four-billion-parameter model is far smaller than hosted frontier models. The results speak to what an analyst can run on local hardware, not to the capabilities of the largest models; a frontier model might pass tests this one fails, and might also be more contaminated.

10 Reproducibility

All code is released under the Apache-2.0 licence at https://github.com/thanatchon36/cbtone and on PyPI as cbtone. make reproduce rebuilds every table and figure from the committed corpus snapshot; LLM scores are cached by content hash, so a re-run repeats only the few calls that failed the first time, while the fine-tuned encoder is retrained, which on Apple MPS is not bit-for-bit deterministic (ANOMALIES.md, A13). The pre-registration is the git tag prereg-v1; every later deviation is listed in docs/DECISIONS.md (D17–D21) and every data anomaly in docs/ANOMALIES.md. LLM scores were computed on a desktop RTX 4070 Ti (D20); the fine-tuned encoder and the FOMC-RoBERTa replication were trained on an Apple M4 laptop (16 GB, MPS). The replication takes about two hours for three seeds, plus ten minutes to score the English corpus, and is not part of make reproduce; make fomc-roberta rebuilds both, and the weights are published separately (D21). Per-stage wall time is in results/timing.json and the hardware and software manifest in results/hardware.json. Every number in this paper, including those in the abstract, is injected from results/summary.json at build time (paper/build.py); none is typed by hand. A fresh-clone reproduction on 14 October 2026 completed in 23 minutes with no change to any verdict.

11 Conclusion

A small open-weight language model running on a consumer graphics card produces hawkish/dovish tone indices for seven central banks in three languages that look plausible over time and agree with human FOMC labels better than a dictionary, though far below a supervised encoder trained on those labels. On every pre-registered test of policy signal, next-decision prediction, announcement-window reaction, cross-lingual consistency and prompt stability, the evidence is negative or underpowered, and the entity-masking and post-cutoff checks show that the negative result is not an artefact of contamination, while a supervised encoder that reads Fed sentences far more accurately fails the same tests, so it is not an artefact of a weak instrument either. Three lessons follow for practitioners. First, persistence is a strong baseline for decision prediction, and gains from text are small, bank-specific and no larger for a better-calibrated scorer. Second, LLM tone depends on the prompt and on the model version, so a single-prompt index should not be treated as a measurement without a stability check, and the prompt and model digest should be reported as part of the measurement. Third, for Southeast Asian banks the binding constraint is sample size, which no improvement in the scorer can relax. The library, the masked and post-cutoff checks, and every number in this paper can be regenerated with a single command; we hope that the design, rather than the particular model, is what is reused.

References