Simulator + LLM. The value fix and five models

What changed in the prompt

Yesterday’s diagnosis was that the value shown for every extraction (npv_impact_per_unit) was the mining cost alone, −10 / 1.1^t per tonne, identical for ore and waste. Greedy ranks by a different number, npv_delta, which adds the discounted revenue of the ore that extraction makes processable this period (within processing capacity and the blend limits).

The new prompt keeps everything from the structure version and changes only that value. The extraction description gets one extra sentence:

Context: This action incurs a mining cost. Its npv_impact_per_unit also includes the revenue of the ore it makes ready to process in the current period.

The difference on the first decision of the decoy mine:

before: (1, 1, 0.8, 50, -9.09), (1, 2, 0.6, 50, -9.09), (1, 3, 0.0, 50, -9.09), ...
after:  (1, 1, 0.8, 50, 63.64), (1, 2, 0.6, 50, 45.45), (1, 3, 0.0, 50, -9.09), ...

Ore blocks now look worth mining, and waste still shows the bare cost.

Testing a bigger model first

Before changing the prompt, I checked whether a larger model would see past the misleading number on its own. I sent the exact waste-cover decision from yesterday (period 4, ore block 10 next to four waste blocks, all at −6.83) to gpt-oss-120b on Groq, three times at medium effort and once at low effort, and to gpt-oss-20b on Groq for comparison.

All five calls chose waste block 6, amount 1. The 120B model’s reasoning:

“block 10 has ore_concentration 0.6 vs 0.1 for others. However npv impact per unit is same −6.83 regardless of concentration… choose smallest amount: 1.”

So the problem was never the model’s size. The model trusts the number it is given. It also explains the 1 t nibbles seen yesterday. With every extraction showing a loss, “take the smallest amount” looks like the sensible choice.

Results (actions_value prompt, full information)

The missing Groq runs hit Groq’s free daily limit (see below).

• 20 of 24 runs are within 0.5% of greedy, most of them to the cent.
• 2 runs beat greedy. gpt-oss on central cone (+126%) and deepseek-r1 on waste cover (+2.5%). The central cone run took 80 actions, more than any other, so it needs repeating before I read much into it.
• 2 runs fall clearly below greedy, both on central cone. deepseek-r1 (334, −63%) and gpt-oss-120b (673, −25%).
• No model gets near the MILP where planning ahead matters. Central cone, central pipe and waste cover all stay at greedy’s level, well below the MILP.

For comparison, the same two mines across all of the prompts so far (gpt-oss 20B):

Speed

• Groq answers in under a second, but the free tier allows only 8,000 tokens per minute, so the scheduler has to pace itself. That is why Groq ends up at about the same seconds per action as my GPU.
• gpt-oss takes the most actions (66 on average, against 20 to 27 for the others), because it still splits blocks into small pieces. The other models more often take a whole block at once.
• qwen3 14B is the slowest at about 46 seconds per action, because of its long reasoning.

Groq’s daily limit

The free tier allows 200,000 tokens per model per day. The gpt-oss-120b run on central pipe used it up partway through, then got one call through roughly every 12.5 minutes. By step 112 it was still extracting 1 t at a time from the same block, where the local models finished the whole mine in 20 to 30 steps. I stopped it there. On the three mines it didn’t reach, every local model already matches greedy within 0.2%, so the missing runs are unlikely to change the picture.

Moving machines

My supervisor’s machine went offline today, so I moved to another Ubuntu machine. Everything was in GitHub apart from the run logs. Two small changes made the move painless.

• Finished runs are now recognised from results/, which is in git, instead of from the log files, which are not. The batch picked up exactly where it stopped.
• Gurobi is switched off for the LLM runs, since this machine has no academic licence. The LLM and greedy only use the simulator, and the MILP numbers come from my earlier results, so nothing is lost. Greedy reproduces the old numbers exactly, which confirms the simulator is unchanged.

What I take from today

• The right number matters more than the right model. One changed value fixed a failure that neither more prompt text nor a model six times larger could fix.
• Once the LLM is given greedy’s number, it behaves like greedy. That is a sensible floor, but it is also a ceiling for a one-step decision.
• The mines where the MILP wins (central cone, central pipe, waste cover) need several steps of planning, such as stripping waste now for ore two periods later. A value that only looks one step ahead can’t show that.

Next

• Let the simulator take the full amount, so the LLM only chooses the block. This removes the nibbling and should cut the calls, and the tokens, sharply.
• Plan once per period instead of once per step. The LLM ranks blocks or sets a target for the period, and the simulator carries it out. That gives it a chance to plan beyond one step, which is what greedy can’t do.
• Repeat the two runs that beat greedy with other seeds before treating them as real.