QWEN3.5-4B · LORA R16 · LIVE-Y GRPO

Inside one reinforcement-learning update

The base model is never rewritten. Six sampled policies face matched randomness; delayed local cost decides which order tokens become more or less likely; gradients update only low-rank LoRA branches.

FORWARD PASS

What the wholesaler sees—and what it does not

ROLE-LOCAL JSON OBSERVATION

Visible every week

inventory on handbacklog inventory positionunits on order incoming retailer ordersshipment received units filledlast 8 weeks

Hidden demand lawfactory capacityother roles’ private state and policy

POLICY FORWARD

Frozen knowledge + a trainable correction

Base W

frozen

LoRA BA

train

h′ = hW + h · (α/r)BA with r=16, α=16

The LoRA residual is injected into q/k/v/o attention and gate/up/down MLP projections.

SAMPLED ACTION

One strict JSON order

{"quantity":48}

temperature .7 · top-p .95
32-token cap · integer 0…128

MATCHED ROLLOUTS

What one decision is compared against

Shared across all six: scenario seeddemand drawscounterparty RNGsame week Sampled actions diverge, so later states differ.

member 1order 32G −217.5A −1.76

member 2order 0G −195.5A +0.74

member 3order 48G −194.0A +0.91

member 4order 48G −200.0A +0.23

member 5order 48G −195.0A +0.80

member 6order 0G −210.0A −0.91

Actual artifact · update 16 · training seed 8 · week 16 · same-timestep group mean μ = −202.0

DELAYED CREDIT

Check the six weeks this order can affect

MEMBER 3 · ORDER 48 AT WEEK 16

w1627

w1719

w1827

w1943

w2035

w2143

SAME-TIMESTEP NORMALIZATION

member return−194.0

group mean μ−202.0

group spread σ8.79

advantage (G−μ)/σ+0.91

Protocol failure inside the window bypasses normalization and receives constant A = −5.

TOKEN CHECK + OPTIMIZER

Only the part that carries the decision receives credit

DECISION-TOKEN MASK

JSON boilerplate gets zero loss weight

{0

quantity0

:0

481

}0

rollout token likelihood ℓoldvscurrent ℓθ

r = exp(ℓθ − ℓold) · PPO clip [0.8, 1.2] · dual clip 3

BACKPROPAGATION

Positive A raises this order’s probability; negative A lowers it

q_projk_projv_projo_projgate_projup_projdown_proj

Base 4B parameters: no update. LoRA A/B: AdamW update at 1e−5, global gradient norm clipped to 1.0.

Weights moved → clear stale prefix KV cache↺ next rollout

THREE LEVELS

Three comparisons, kept separate

CREDIT COMPARISON

Trajectory vs groupmates

matched on: seed + week
compare: 6-week return
produces: advantage A

POLICY COMPARISON

Current vs rollout likelihood

compare: token log-prob
stabilize: ratio clipping
updates: LoRA only

HELD-OUT EVALUATION

Final adapter vs untouched base + baselines

Qwen base6.73

blind constant-1819.82

LoRA + GRPO20.64

adaptive base-stock73.9

hindsight-perfect100

16updates

8seeds / update

6group members

768trajectories

1× A40 · 15.4 hours · ≈$7.50