Simulator + LLM. Does the shaft help, and why the LLM loses track

The shaft test

The question was simple. If the LLM gets the shaft’s estimated grades (partial) or every grade (full), does it get closer to the MILP?

For the test set I used the six mines where the shaft helped the MILP most: waste cover, decoy, three random mines and the 3D flat seam. On the six 27-block 3D mines from the earlier runs the shaft added almost nothing even for the MILP (gains of 0.01 to 11), so they could not show anything.

The shaft data reaches the LLM by coordinates. Each extraction action carries its block’s (x, y, z), and the system prompt lists the drilled blocks the same way, for example (2, 0, 1) ~0.85. In full, the prompt lists every block’s grade instead.

Totals over the six mines (% of full-information MILP)

	Greedy	MILP	Sim+LLM
Blind	52,840 (87.4%)	49,673 (82.1%)	50,290 (83.1%)
Partial	52,840 (87.4%)	57,697 (95.4%)	49,182 (81.3%)
Full	52,840 (87.4%)	60,487 (100%)	51,814 (85.7%)

Greedy is the same in every row because it only ever chooses among blocks it can mine now, and those are always revealed.

Per mine

Mine	Greedy	MILP blind / partial / full	LLM blind / partial / full
Waste cover	7,180	3,946 / 6,898 / 7,834	7,176 / 5,687 / 7,020
Decoy	6,899	8,416 / 10,306 / 10,500	6,899 / 6,899 / 6,748
Random seed 6	13,876	13,457 / 14,376 / 14,462	13,876 / 13,876 / 13,320
3D flat seam	−170	795 / 1,666 / 1,676	−2,654 / −1,648 / −170
Random seed 5	11,383	9,591 / 10,367 / 11,822	11,320 / 11,394 / 11,270
Random seed 1	13,672	13,467 / 14,083 / 14,192	13,672 / 12,975 / 13,625

The MILP turns the shaft into 13 points of NPV. The LLM stays at greedy’s level, and in partial it is even slightly worse.

Waste cover, step by step

Waste cover shows how information can hurt. The surface is all 0.1 grade, with richer ore under it (0.6 at the edges, up to 0.9 in the middle).

Period	Blind (7,176)	Partial (5,687)
1	surface x = 0, 1	surface x = 2, then x = 0
2	finishes x = 1, so the edge ore opens and it processes 33 t	more surface, processes nothing
3	more ore	the middle ore finally opens, processes 17 t

In partial the LLM followed the drill data toward the 0.85 ore in the middle. But a middle block needs three surface blocks removed where an edge block needs two, and it also stripped a block that did not help. Revenue started one period later, which costs more than the higher grade earns back. The blind run took the first block in the list, which happened to be the cheapest way in.

Why the paper got close to the MILP

My earlier paper (Sim2Schedule) reported the LLM at 94.5% of the MILP. Three things explain the difference.

Paper mine (27 blocks, 10 periods)	NPV	% of paper’s MILP
MILP (5% gap)	20,579	100%
LLM with context	19,456	94.5%
Greedy, paper	19,062	92.6%
Greedy, today’s simulator	20,277	98.5%

• The mine is easy for greedy. Almost every top-layer grade is rich, so following the immediate value already gets close to the optimum.
• The MILP was solved to a 5% gap, so the true optimum could be up to 5% higher.
• The simulator changed after the paper (the greedy rule and proportional processing). Greedy on the same mine went from 19,062 to 20,277.

Today’s mines were designed so that greedy fails. On those, an LLM that behaves like greedy is far from the MILP. That matches the paper’s own observation that without context the LLM falls back to greedy-like behaviour.

What the logs show

I read the waste-cover partial run call by call. The LLM clearly uses the drill data, since every call reasons about reaching the 0.85 block. Then it goes wrong in four ways.

1. It forgets what it has mined. In call 1 it mines block 3 fully. In call 3 it chooses block 3 again: “we should extract block 3 first”. The reply is rejected.
2. It misreads the precedence rule, differently each time. Call 2 gets it right. Call 7 decides “each block at z = 1 touches the block directly above at same x”. Call 8 looks for blocks above a surface block.
3. After a rejection it takes 1 t. “we want to minimize amount extracted to reduce cost… choose 1 ton. That is best.” Every extraction shows a loss per tonne, so the smallest amount looks best, even though it only postpones the cost and leaves blocks half-mined.
4. It does not know what ore is worth. “We don’t know processing revenue… maybe processing revenue per ton is high.”

How much it remembers

Each call sends the system prompt, the last 3 prompts with the LLM’s 3 replies, and the new prompt. Anything older is dropped. Retries take history slots too, so after two invalid replies only one real decision is left in view. The model’s own reasoning is never sent back. So “which blocks are done” survived at most three calls.

Two more things in the prompt add to the confusion. max_extractable_tons drops to the capacity left in the period (for example from 50 to 20 for every block), which the LLM reads as every block shrinking. And the original prompt line “a block is only available for processing after it has been fully extracted” is not true in the current simulator, which processes the mined share of a block straight away.

The fix. Live mine state

Three changes, as a new prompt version.

• Live mine state in every prompt, rebuilt from the simulator at each step. Only blocks whose grade is known are listed (in blind, unknown blocks are left out, since the mine size is already given). Each entry is (block, grade, tons left to extract, tons left to process). For example, in partial, (12, '~0.87', 50.0, 43).
• Precedence written out. In the coordinate version, “a block at (x, y, z) can only be extracted after every block at (x + dx, y + dy, z − 1), with dx and dy each −1, 0 or +1 and inside the mine, has been fully extracted.” I checked that this rule matches the simulator’s precedence for every block on four mines. The block-number version keeps the general sentence.
• One line on partial extraction. “Extracting part of a block costs the same per tonne as extracting all of it, but only a fully extracted block unlocks the blocks below it. A period ends only when its mining capacity is used, so extracting less now does not avoid the cost.”

To see whether naming blocks by number or by coordinates matters, I built both versions and ran them on decoy.

Decoy results (block-number version)

Decoy	Greedy	MILP	LLM before (coordinates, no state)	LLM now (block numbers + live state)
Blind	6,899	8,416	6,899, 61 actions	6,899, 21 actions
Partial	6,899	10,306	6,899, 24 actions	6,899, 17 actions
Full	6,899	10,500	6,748, 22 actions	9,793, 25 actions

The nibbling is gone. The same mine takes about a third of the actions, with almost no invalid replies.

Full reaches 93% of the MILP. This is the first time the LLM moved clearly beyond greedy. Its schedule:

Period	What it did
1	mines rich surface blocks 1 and 2, processes 1 (like greedy)
2	finishes 2 and starts stripping waste block 5
3	mines 3 and 8, processes 3
4	finishes waste blocks 4, 5 and 6, with no revenue this period
5	mines block 12 (0.9 ore) and processes it
6	more of 12, plus 8, 10 and 11

Block 12 sits under surface blocks 4, 5 and 6. With the true grade in front of it and the live state showing progress, the LLM paid for that waste in periods 2 to 4 to open 0.9 ore in periods 5 and 6. Greedy never makes that trade.

Partial tried the same and failed on geometry. It went for block 12 too, and even mined part of block 5 in period 1. Then:

• it worked out the layers upside down: “top layer is z = 1… block 2’s above block is block 9, so block 2 cannot be extracted until block 9 is fully extracted”,
• it thought only block 5 sits above block 12, when 4, 5 and 6 all have to go,
• it misread the state and decided block 12 was already fully extracted.

So in partial the drill data changed its intent, but without block positions it could not work out which blocks to remove. In full, with every block’s grade listed, it got the layout right.

What I take from today

• On these mines the gap to the MILP is about planning, not information. The MILP turns a shaft into 13% more NPV because it plans the whole stripping sequence.
• The LLM failed mainly because it lost track of the mine. Once it can see the live state, it stops nibbling and, with full information, makes a real multi-period trade-off.
• Partial information still needs the geometry to be clear. Five estimated blocks named only by number were not enough for it to plan the route.

Next

• Run the coordinate version on decoy (blind, partial, full). If partial with coordinates reaches block 12 in time, block naming is what held it back.
• Repeat the full run with other seeds before treating the 93% as solid.
• Then run the better of the two versions on the other five shaft mines.