In case you have skilled a deep reinforcement studying agent in Python and tried to run it in opposition to a dwell MT5 terminal, you’ll have seen this: the agent converges cleanly, the backtest Sharpe seems good, after which on a paper account it behaves prefer it by no means skilled. No error, no warning. Simply quietly unhealthy choices.
In most of those instances the mannequin is ok. The issue is the bridge — the layer that computes the statement vector from uncooked market information. If that computation differs even barely between coaching and manufacturing, the agent receives inputs it by no means noticed throughout coaching, and a coverage is simply pretty much as good because the distribution of inputs it was skilled on.
The failure that produces no error message
One concrete instance. The coaching pipeline match Z-score normalization parameters over the total historic dataset. These parameters had been by no means saved. The manufacturing bridge as a substitute computed a rolling Z-score anchored to at any time when the dwell system began. Similar function names, completely different imply, completely different normal deviation. Each normalized worth shifted exterior the coaching distribution, and the coverage produced near-random actions with full confidence.
The explanation that is so arduous to catch is that nothing throws. The tensor shapes match. OnnxRun() returns a sound motion. The order goes out. The one symptom is efficiency that’s worse than random.
Three parity failures value checking first
- Normalization parameter drift. Freeze μ and σ throughout coaching, save them to a file, and cargo them within the EA. Recomputing them at runtime from dwell information defeats your complete function of normalization consistency.
- Place state encoding mismatch. Coaching encoded state as flat/lengthy/quick = {0, 1, 2}; the MQL5 bridge used {-1, 0, 1}. The dimension matches, so the mannequin accepts it — however “lengthy” in manufacturing now has the numeric signature of “flat” in coaching. Directionally inverted choices, full confidence.
- Session flags on the incorrect clock. The Python facet used datetime.now() with no timezone and defaulted to OS native time, whereas bar timestamps got here again in UTC. London/NY/Asian session flags landed six hours off, so session-conditional insurance policies fired within the incorrect periods all day.
The bar-boundary lure
This one is particular to how MQL5 indexes bars. Shut[0] is the present, still-forming bar; it modifications on each tick. Shut[1] is the final accomplished bar. Throughout coaching, each statement is constructed from accomplished bars solely — the agent by no means as soon as noticed a partially-formed bar. In case your dwell statement builder reads Shut[0] for any price-derived function, you might be feeding the coverage a worth no coaching pattern ever contained.
The distribution of Shut[0] over a bar is genuinely completely different from the distribution of a accomplished shut: greater intra-bar volatility, and in quiet regimes it sits biased towards the open. The repair is to gate the entire inference step on a brand new bar:
static datetime lastBarTime = 0;
datetime currentBarTime = iTime(Image(), Interval(), 0);
if(currentBarTime == lastBarTime)
return; // no new bar closed — do nothing
lastBarTime = currentBarTime;
// Construct statement from accomplished bars (index 1+) and question the mannequin right here
The identical precept applies to each indicator you feed the agent. Learn from the final accomplished bar, not the present one:
double rsi = iRSI(Image(), PERIOD_M15, 14, PRICE_CLOSE, 1); // index 1, not 0
Validate the bridge earlier than any cash strikes
Not one of the checks beneath require dwell buying and selling:
- Historic replay parity check. Take a 500-bar phase from the check interval, run it via the manufacturing bridge, and log the total statement vector at each bar. Diff it element-by-element in opposition to the coaching pipeline output for a similar bars. The utmost absolute distinction ought to be beneath 1e-6 . Something bigger is a parity failure — discover it function by function earlier than continuing.
- Random agent check. Earlier than loading the skilled coverage, run a coverage that picks purchase/promote/flat uniformly at random on a paper account for every week. Cumulative reward ought to sit close to zero, minus transaction prices. Whether it is badly detrimental, the atmosphere or reward has a bias the random agent is exposing — the coverage shouldn’t be the trigger.
- Canary account. Solely then deploy the true coverage on 5–10% of supposed capital at minimal lot measurement for 2 weeks, and examine Sharpe and drawdown in opposition to the paper baseline.
Maintain the guardrails out of the mannequin
The coverage maps observations to actions. It has no idea of “the Python course of simply crashed” or “that is an illiquid 3 AM session.” These belong within the EA: a tough stop-loss on each place, a tough position-size ceiling that overrides the agent’s lot requests, a each day loss circuit breaker, and a heartbeat watchdog that treats a silent Python course of as unavailable and stops opening positions. The agent will be incorrect concerning the market; it mustn’t ever be capable of go away an unmanaged place open throughout a connectivity failure.
Deploying a DRL agent to MetaTrader is an engineering drawback, not a machine studying drawback. Get the statement parity proper, show it in opposition to historic information, and implement threat within the execution layer independently of the mannequin. That covers the vast majority of dwell deployment failures.
