18
u/Bowaka 1d ago
Tree based models are particularly sensitive to distribution drift.
1
u/Luctom 1d ago
may I ask why?
2
u/addsubps 1d ago
Because they optimize thresholds, not weights.
1
u/Bowaka 23h ago
It's more that they average the existing within the training space but are not able to extrapolate outside of it, unlike a linear regression for example.
Trees don't learn a relationship within the data, it just learn its statistical behavior locally. The problem behind that on space problems, its easy to go in unexplored zones.
6
u/veskald 1d ago
One thing missing from this thread: how are you scoring the model? If its accuracy or AUC, thats not the same question as "does this make money". A model can be right about direction more often than not and still lose after spread and commission, because the moves you catch on 10 minute windows are small and the costs are fixed. Score the model by net PnL per trade after real costs from the start, not by prediction quality - a lot of "promising but not profitable" models die instantly under that metric, and you stop wasting months on them.
The other thing is you are now searching: 5, 10, 15 minute windows, different symbols, XGBoost then LGBM then HAR. Every combination you try raises the chance that the best one looks good by luck alone. Keep a log of everything you tested including what died, and when something finally works, look at that number first - one winner out of fifty attempts is a very different claim than one out of three.
0
u/Grand-Fly-6090 1d ago
On scoring, I’m using net pnl in agreement with your suggestion. My cost model is a simple 1 bp supported by trades executed by my bot.
2
u/veskald 1d ago
Good, thats the right metric. One thing to check on SOXL specifically: is that 1 bp covering spread and slippage or just commission? On a 3x ETF the spread alone can be wider than that at some times of day, and with 10 minute windows the move you are capturing is small enough that the difference decides whether the edge exists. Would be worth pulling actual fill prices from your bot trades against the signal price and seeing what the real number is like a execution log.
1
6
u/Manfred_der_Gorilla 1d ago
I tried Xgboost and random forests for 15 minute intervals, but it is really just noisy. You might get some decent in sample predictive power but out of sample my models always collapse
1
u/Grand-Fly-6090 1d ago
I've tried 5 and 15 minute windows, and they were worse than 10 minute windows. Could have been the SOXL symbol and when I was looking.
8
u/Chemical_Badger6227 1d ago
10min windows are very noisy. Try using 1hr windows. I’ve back tested over 100 strategies and found it zero edge using XGBoost. In fact the best edge came from the simplest of signals. In my case, it was volume ratio and the mvrv ratio.
3
u/Grand-Fly-6090 1d ago
Do you mean 1 hour windows of 1 minute bars, or are you suggesting to use 1 hour ohlc bars?
2
3
u/Suspicious_Meet_1220 1d ago
I have tried, it works on papers. What mean is the model is guessing with 70% accuracy. Missing at detecting pullbacks during major ones and reversal is not very smooth it goes from 100- 60/50 in a stepa manner so i am not able to design a strategy arround it.
2
u/IllustriousGrade7691 1d ago
Need larger window 10 minutes are random noise and it is not the modell its always the data modell differnce is negligible between LGBM XGBoost. Everyone is using lgbm since it trains faster. Also i would advise you to not manually code you modells but use Codex. Think is you will not create a good modell per hand heck even AI cannot there is just not that much inforamtino. People make moy arb tading between platforms or some shit.
Big quant firms maybe will manyge 30% return per year but you would not. Give codex or claude a try
1
u/Grand-Fly-6090 1d ago
Agreed. I am using codex for my XGBoost work. I'll take a look at LGBM and HAR mentioned below. Thank you.
2
u/IllustriousGrade7691 1d ago
What are you trying to predict Stocks or Crypto, might want to check https://numer.ai/ its an interesting albeit old project idk how relevant it is the ai agent can explain they do target returns prediction 20 or 60 days in the future use things like multiple targets groups and feature neutralisation. Check their notebooks or let codex check them and see if it helps the modell
1
2
2
u/jamesscheibel 1d ago edited 1d ago
the short answer is yes. but i use homegrown GBM. it is comparable to XGBoost though. there are a lot of asterisks next to that yes. the data miner tool/method you use is just 1 tiny piece. like trying to build a hot rod from scratch and deciding on what kind of engine you are going to use but having decided very little else. i do use 10 minute bars fwiw
1
1
u/Grand-Fly-6090 1d ago
The 10 minute bars are working out a little better. I guess they filter out some of the noise.
2
u/Linett-Chukwuemeka61 1d ago
XGBoost is solid for time series forecasting if you're doing feature engineering properly - just watch out for overfitting on noisy market data since it'll happily memorize noise patterns. What kind of prediction targets are you working with?
1
u/Grand-Fly-6090 1d ago
I’m trying to determine if my sell limit (target) will be hit before my sell stop. The target is 1.5R, so a P(target) > 0.5 should be worthwhile. The 1 minute bar data was way too noisy to do anything useful.
2
u/ZealousidealName5703 15h ago edited 15h ago
Depending on your boundary conditions, yes xgboost can be extremely effective. I got 0.88/1 AUC at highest but really low number of live trades like 50 per day. My current AUC is at 0.8541.
1
u/Grand-Fly-6090 14h ago
That’s pretty awesome! What flavor of bars are you using. I died on 1 minute bars. 10 minute bars are better.
2
u/mr-claesson 1d ago edited 1d ago
You need to engineer features that you believe have an edge, like if you believe in fair value gap, macd divergence etc. Pure OHLCV does not work.
Next challenge is to make the training data somewhat balanced to avoid lazy guessing. It is really easy to achieve high accuracy on a highly unbalanced/biased training set.
And after that you have years ahead of you to fall in other traps until your realize the foundation, then you can start hunting optimizations using different model architectures
To lower the risk of overfiting make sure results are stable over rolling folds splitts, not just an ordinary train/validate/test split
1
u/jamesscheibel 1d ago
yeah, getting the right features is crucial but then the whole things is a house of cards. you need it all to make it work.
1
u/stratcore 1d ago
One check before changing models: overlapping 10-minute windows can leak almost the same bars into train and test, even with a time-ordered split at row level. Purge at least the full feature and label horizon around each fold, fit every transform inside the fold, and keep one final period untouched. Also compare against a regularized logistic baseline. If XGBoost cannot beat that after actual costs and turnover, the extra complexity is not buying signal.
1
u/Outrageous_Band9708 1d ago
that will never happen
everything below 1H candles is pure noise
anyone with a strat there is just riding market beta without realizing it.
2
u/mkvalor 9h ago
What is missing are derived features from the base data. Not sufficient to just point it at OHLCV and call it a day. And each ticket/contract needs its own feature engineering and modeling.
For one thing, price is not predictive. So you have to think about how to transform price into something more stationary. And I don't mean just copy a pytorch line about standardizing price or normalizing it.
Here is a hint: consider how to aggregate all sorts of behaviors such as buckets representing counys of orders over the past 10ms, 100 ms, 500 ms... This becomes a new set of features that can inform the boosted tree model. If your data doesn't come in with sub-second precision, grab at least milliseconds from your own computer clock and add them on.
First you must consider what the market is doing, then observe it, then you can eventually get to the place of modeling it.
2
u/Livid_Parsnip_2816 5h ago
I’d stop asking XGBoost to predict every one-minute bar and use it as a meta-filter. Generate trades with a simpler base rule, then train the model on take/skip or position size. Ten raw bars carry very little context; add volatility state, spread, time of day and distance from the higher-timeframe trend, then use purged walk-forward validation with an embargo. Normalized features help, but leakage and target choice are usually the bigger problem.
55
u/roztok_potok 1d ago
No ml method will work if you don't have a profitable model as a base. Ml is addition only.