I am trying understand how tree-based regression model handle the dependencies of the target variables on the interaction of explanatory variables.
However my experiment revealed that my understanding about the fitting process of a lgbm is not correct. And I don’t know why.
My experiment is quite simple: a target (for sake of simplicity only in [0, 1]) and two explanatory variables with two values such that the mean of the target is the same for each of the values of the explanatory variables. Then there is a third variable that models the interaction of the explanatory variables by a simple count.
So in code:
>>>
import polars as pl
df = pl.Dataframe(
{
„y“: [0, 0, 1, 1, 0, 0, 1, 1], # mean across „A“ values the same; mean across „B“ values the same
„A“: [1, 1, 1, 1, 0, 0, 0, 0],
„B“: [1, 1, 0, 0, 1, 1, 0, 0],
„AB“ [1, 1, 2, 2, 3, 3, 4, 4] # just some IDs for the interaction
}
)
<<<
I then fitted a lgbm just with „A“ and „B“ and got the expected constant 0.5 forecast
>>>
from lightgbm import LGBMRegressor
lgbm = LGBMRegressor(min_child_samples=1)
lgbm.fit(df[[„A“, „B“]].to_numpy(), df[„y“].to_numpy())
lgbm.predict(df[[„A“, „B“]].to_numpy()).round(0)
array([0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5, 0.5])
<<<
Then I did the same but with „AB“ and expected a perfect fit. But I was disappointed, it fitted to constant zero
>>>
lgbm = LGBMRegressor(min_child_samples=1)
lgbm.fit(df[[„AB“]].to_numpy(), df[„y“].to_numpy())
lgbm.predict(df[[„AB“]].to_numpy()).round(0)
array([0, 0, 0, 0, 0, 0, 0, 0,])
<<<
I tried to code „AB“ as category. But still no perfect fit:
>>>
lgbm = LGBMRegressor(min_child_samples=1)
lgbm.fit(df[[„AB“]].to_numpy(), df[„y“].to_numpy())
lgbm.predict(df[[„AB“]].to_numpy()).round(0)
array([0, 0, 1, 1, 0, 0, 0, 0,])
<<<
Super confusing!
I then turned to catboost and found even without „AB“ it fit the data perfectly:
>>>
from catboost import CatBoostRegressor
cbm = LGBMRegressor(min_data_in_leaf=1)
cbm.fit(df[[„A“, „B“]].to_numpy(), df[„y“].to_numpy())
cbm.predict(df[[„A“, „B“]].to_numpy()).round(0)
array([0, 0, 1, 1, 1, 1, 0, 0])
<<<
I thought that lgbm should be able to fit the data with „AB“. The variable allows for perfect splits since the gain for each split is super clear. But somehow it cannot go „down“ the tree to fit the values for AB=3.
What is the difference of catboost that allows for a perfect fit even without an explicit modeling of the interaction? Does it split less lazy and explores split of splits, while building the trees?
