Group-Level Training Data Attribution with Exact Shapley Analysis

Research
Shapley Attribution
Training Data
Machine Translation
Data Auditing
Robustness Analysis
LoRA / PEFT
Xiaojing Yang

Xiaojing Yang

@Entropyobserver

Group-Level Training Data Attribution with Exact Shapley Analysis

Tech Stack

Python
PyTorch
Transformers
Hugging Face
NLLB-200
LoRA
PEFT
SacreBLEU
chrF
Data Curation
Bootstrap
Statistics
SLIDE

Description

This project asks a data-centric question: Which types of training data contribute to shaping model behavior after fine-tuning?

We divide an English–Norwegian petroleum-domain corpus into four groups based on written-standard classifier scores and evaluate all 16 possible combinations using exact Shapley values.

Instead of using a single score for data value, we separately measure translation quality, terminology performance, and written-standard behavior.

As the first author, I led the research and experiments for this project. My contributions included:

  • Designed the group-level Shapley attribution framework, including the coalition setup, evaluation metrics, and attribution procedure.
  • Led the experiments across NLLB and decoder-only models, including LoRA fine-tuning, coalition training, evaluation, and size-matched random baselines.
  • Evaluated the robustness of the attribution results using bootstrap confidence intervals and group-level statistical tests.
  • Analyzed how different training groups contribute to translation quality, terminology accuracy, and written-standard behavior.

Research Questions

RQ1

How do training-data groups contribute to translation quality, terminology performance, and written-standard behavior?

RQ2

Can group size alone explain these contributions?

RQ3

How consistent are these attribution patterns across model architectures?

Training-Data Groups

We use SLIDE scores to divide the 13,935 training pairs into four mutually exclusive groups. These groups are used as the units for Shapley attribution.

GroupRulePairs
High-BokmålsNB ≥ 0.80 and sNN < 0.3010,113
BoundarysNB ≥ 0.50 and sNN ≥ 0.50880
Nynorsk-likesNN ≥ 0.50 and sNB < 0.502,645
Uncertain-otherAll remaining cases297

Here, sNB and sNN are independent SLIDE scores ranging from 0 to 1.

What is in each group?

High-Bokmål

Mostly complete Bokmål sentences about petroleum production, drilling, licences, fields, and geology.

Nynorsk-like

Mostly complete petroleum-domain sentences containing recognizable Nynorsk words and grammatical forms.

Boundary

A mixture of ordinary Bokmål sentences and short items such as contact lines, captions, dates, measurements, and headings.

Uncertain-other

Mostly short metadata-like items: dates, coordinates, company names, headings, links, labels, and text fragments.

These are operational units for Shapley attribution, not gold linguistic categories. The Boundary group mainly reflects classifier uncertainty rather than a clear written-standard category.

All 16 Training-Data Combinations

All sixteen coalitions formed from the four training-data groups, organized by coalition size

The empty coalition uses the base model; the other 15 are fine-tuned separately.

A coalition is one possible selection of the four groups. Each group is either included or excluded.

24 = 16 coalitions

Exact Shapley Value

φg(m) = ∑ w(S,g) Δgvm(S)
S ⊆ G ∖ {g}
Shapley weight
w(S,g) = |S|!(|G| − |S| − 1)!|G|!
Marginal contribution
Δgvm(S) =vm(S ∪ {g}) − vm(S)

For each group g, its exact Shapley value is the weighted average change in metric m when that group is added across every possible coalition.

G = all groups · S = a coalition without g · vm(S) = S’s score on metric m

Experimental Pipeline

Full group-level attribution protocol from corpus grouping and coalition construction through model training, evaluation, exact Shapley attribution, cross-branch comparison, and robustness checks

Complete group-level attribution pipeline.

  1. 1

    Group the corpus

    13,935 training pairs → 4 operational groups

    SLIDE-based grouping · 200-item manual audit

  2. 2

    Enumerate coalitions

    4 groups → 2⁴ = 16 coalitions

    including the empty coalition

  3. 3

    Train and evaluate

    15 non-empty coalitions · 3 seeds · 2 architectures

    NLLB-600M · NorMistral-7B-warm

  4. 4

    Compute exact Shapley values

    Use all 16 coalition scores

    → one contribution value per group and metric

  5. 5

    Stress-test the results

    Size-matched random groups · test subsets · bootstrap CIs

    training-schedule and grouping-threshold checks

RQ1. How do training-data groups contribute to translation quality, terminology performance, and written-standard behavior?

Exact Shapley attribution · NLLB-600M · all 16 coalitions · three-seed average

Training group
BLEU
chrF
TermF1
High-Bokmål output change (pp)
Nynorsk-like output change (pp)
High-Bokmål
+19.83
+13.56
+0.273
+38.1
-33.8
Boundary
+2.04
+1.08
+0.038
-1.6
-0.8
Nynorsk-like
+3.21
+2.60
-0.028
-45.9
+46.9
Uncertain-other
+0.610
+0.560
+0.028
+1.2
-0.7
Positive contributionNegative contributionpp = percentage points · Intensity normalized within each metric

Key Finding

High-Bokmål → broad gains

↑ Translation quality · ↑ terminology · ↑ High-Bokmål output

Nynorsk-like → mixed effects

↑ Translation quality · ↓ terminology · ↑ Nynorsk-like output

The same training data can improve one aspect of model behavior while harming another.

Why does Nynorsk-like data have positive value?

Overall contribution on the full test set: +3.21 BLEU

1. Group the test references

Use the same SLIDE classifier to identify:

  • High-Bokmål
  • Nynorsk-like

2. Measure source overlap

For each English test source, find the most similar English training source.

Overlap = maximum word- and 5-gram similarity score

Low: <0.30Mid: 0.30 ≤ score < 0.70High: ≥0.70

3. Recompute attribution for each subset

6 subsets = Reference standard × Overlap level

For each subset:

  • Calculate BLEU for all 16 coalitions
  • Recompute the Nynorsk-like Shapley value

Saved predictions · No retraining

Reference standard
Low overlap
Mid overlap
High overlap
High-Bokmål
-5.17 (n=373)
-2.92 (n=678)
-3.47 (n=262)
Nynorsk-like
+3.87 (n=32)
+22.52 (n=99)
+41.97 (n=159)

Key Finding

High-Bokmål references → negative at every overlap level

Nynorsk-like references → positive at every overlap level

Higher overlap → larger gains for Nynorsk-like references

RQ2. Can group size alone explain these contributions?

Why test this?

The High-Bokmål group has the strongest contribution, but it is also the largest group. Could this simply be a size effect?

Size-matched test

Compare the true High-Bokmål group with three random groups of the same size.

3 random groupsSame sizeDifferent examples

BLEU

True High-Bokmål19.83
Random same-size mean11.85

chrF

True High-Bokmål13.56
Random same-size mean7.88

TermF1

True High-Bokmål0.273
Random same-size mean0.088

Key Finding

Random groups also contribute to translation quality, but much less than the true High-Bokmål group.

Group size matters, but group size alone does not explain the observed contribution.

RQ3. How consistent are these attribution patterns across model architectures?

We compare the full group-level Shapley results from NLLB-600M (encoder–decoder) with NorMistral-7B-warm (decoder-only).

Training group
BLEU
chrF
TermF1
High-Bokmål output (pp)
Nynorsk-like output (pp)
High-Bokmål
N+19.83
M+15.31
N+13.56
M+10.95
N+0.273
M+0.148
N+38.1
M+39.1
N−33.8
M−35.1
Boundary
N+2.04
M+0.36
N+1.08
M−0.27
N+0.038
M−0.018
N−1.6
M−5.3
N−0.8
M+2.0
Nynorsk-like
N+3.21
M+1.56
N+2.60
M+1.20
N−0.028
M−0.092
N−45.9
M−45.9
N+46.9
M+47.8
Uncertain-other
N+0.61
M−1.30
N+0.56
M−1.45
N+0.028
M−0.005
N+1.2
M−0.3
N−0.7
M−0.4
N = NLLB-600MM = NorMistral-7B-warmBlue = positiveRed = negativeOutput changes are reported in percentage points (pp).

High-Bokmål

Strong positive quality and Bokmål-output effects in both models

Nynorsk-like

The same output shift in both models: less Bokmål, more Nynorsk-like

Smaller groups

Boundary and uncertain-other effects depend more on the architecture

Key FindingThe main attribution patterns are consistent, but smaller effects can depend on the model architecture.
All Projects