Gradient Boosting Methods

Gradient Boosting Methods: XGBoost, LightGBM, and CatBoost

Introduction

You’ve learned that random forests build many independent trees in parallel and average their predictions. But what if, instead of building trees independently, each new tree could learn from the mistakes of previous trees? This is the key insight behind gradient boosting - arguably the most powerful machine learning technique for structured/tabular data.

In Kaggle competitions and real-world applications, gradient boosting methods (especially XGBoost, LightGBM, and CatBoost) consistently win or place highly. Let’s understand why.

The Core Idea: Learning from Mistakes

Random Forests vs Gradient Boosting

Random Forest: “Let’s build 100 trees independently, each on different random samples, and average their predictions.”

Gradient Boosting: “Let’s build trees sequentially. Each new tree focuses on correcting the errors made by all previous trees.”

Intuitive Example: Estimating House Prices

Imagine you’re estimating house prices with a team of experts:

Random Forest Approach: - 100 experts independently look at the data - Each gives their estimate - You average all 100 estimates - Experts don’t learn from each other

Gradient Boosting Approach: - Expert 1 makes predictions (simple rules) - Prediction: “All houses cost $300K” - Error: Off by a lot!

Expert 2 looks at Expert 1’s mistakes
- “Okay, Expert 1 overestimated small houses and underestimated large ones”
- Expert 2 adds corrections: “-$100K for houses < 1500 sqft, +$200K for houses > 2500 sqft”
Expert 3 looks at remaining mistakes
- “Still underestimating houses in downtown”
- Expert 3 adds: “+$50K for downtown location”
Continue for 100 experts…
Final prediction: Expert 1 + Expert 2 + Expert 3 + … + Expert 100

Each new expert (tree) specializes in correcting the mistakes of all previous experts!

How Gradient Boosting Works: The Mathematics

The Algorithm (Simplified)

Initialize: Start with a simple prediction (often the mean of training targets) \[F_0(x) = \bar{y}\]
For each tree m = 1 to M:
1. Calculate residuals (errors from current model): \[r_i = y_i - F_{m-1}(x_i)\]
2. Fit a new tree $h_m(x)$ to predict these residuals
3. Add this tree to the ensemble (with a learning rate $\eta$): \[F_m(x) = F_{m-1}(x) + \eta \cdot h_m(x)\]
Final prediction: $F_M(x)$ = sum of all trees

Key Insight: Each tree is trained to predict the residuals (mistakes) of the previous model, not the original targets!

Example 1: Simple Gradient Boosting by Hand

Let’s predict house prices with just 3 trees:

Training Data: | House | Features | True Price | |——-|———-|————| | 1 | Small | $200K | | 2 | Medium | $300K | | 3 | Large | $400K |

Step 1: Initialize - $F_0$ = mean price = $(200 + 300 + 400)/3 = 300K$ - Predictions: [300K, 300K, 300K] - Residuals: [-100K, 0K, +100K]

Step 2: Tree 1 (predict residuals) - Tree 1 learns: “Small houses → -100K, Large houses → +100K” - After Tree 1: $F_1$ = 300K + Tree1 - House 1: 300K + (-100K) = 200K ✓ - House 2: 300K + 0K = 300K ✓ - House 3: 300K + 100K = 400K ✓ - New residuals: [0, 0, 0] - Perfect!

In practice, trees are limited (max_depth, learning_rate), so we need many trees to gradually reduce errors.

Example 2: With Learning Rate

Let’s add a learning rate (η = 0.5) to prevent overfitting:

Step 1: Initialize - $F_0$ = 300K - Residuals: [-100K, 0K, +100K]

Step 2: Tree 1 - Tree 1 predicts: [-100K, 0K, +100K] - But we only add 50% of this (η = 0.5) - $F_1$ = 300K + 0.5 × Tree1 - House 1: 300K + 0.5×(-100K) = 250K - House 2: 300K + 0.5×(0K) = 300K - House 3: 300K + 0.5×(100K) = 350K - New residuals: [-50K, 0K, +50K]

Step 3: Tree 2 - Tree 2 predicts these new residuals: [-50K, 0K, +50K] - $F_2$ = 250K, 300K, 350K + 0.5 × Tree2 - House 1: 250K + 0.5×(-50K) = 225K - House 2: 300K + 0.5×(0K) = 300K - House 3: 350K + 0.5×(50K) = 375K - New residuals: [-25K, 0K, +25K]

Step 4: Tree 3… - Continue, each time cutting the error in half

Why a learning rate? - Slower learning (smaller steps) often generalizes better - More trees needed, but each tree is “weaker” - Prevents overfitting to training data

Example 3: Visualizing Sequential Learning

Code

import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import GradientBoostingRegressor

# Simple non-linear relationship
np.random.seed(42)
X = np.linspace(0, 10, 50).reshape(-1, 1)
y = np.sin(X).ravel() + np.random.normal(0, 0.2, X.shape[0])

# Train with just 3 trees to see progression
gb = GradientBoostingRegressor(
    n_estimators=3,
    max_depth=2,
    learning_rate=1.0,
    random_state=42
)

# Visualize each stage
fig, axes = plt.subplots(1, 4, figsize=(16, 4))
X_plot = np.linspace(0, 10, 200).reshape(-1, 1)

# Initial prediction (mean)
axes[0].scatter(X, y, alpha=0.5)
axes[0].axhline(y.mean(), color='r', linewidth=2)
axes[0].set_title('Step 0: Initialize with mean')
axes[0].set_ylim(-1.5, 1.5)

# After each tree
for i in range(1, 4):
    gb_temp = GradientBoostingRegressor(
        n_estimators=i,
        max_depth=2,
        learning_rate=1.0,
        random_state=42
    )
    gb_temp.fit(X, y)
    y_pred = gb_temp.predict(X_plot)
    
    axes[i].scatter(X, y, alpha=0.5)
    axes[i].plot(X_plot, y_pred, 'r-', linewidth=2)
    axes[i].plot(X_plot, np.sin(X_plot), 'g--', alpha=0.5)
    axes[i].set_title(f'After Tree {i}')
    axes[i].set_ylim(-1.5, 1.5)

plt.tight_layout()
plt.show()

What you’ll observe: - Step 0: Flat line (mean) - After Tree 1: Rough approximation of the pattern - After Tree 2: Better fit, capturing more details - After Tree 3: Even closer to the true function

Each tree adds a “correction layer” to improve the fit!

The Three Kings: XGBoost, LightGBM, and CatBoost

All three implement gradient boosting but with different optimizations and features.

XGBoost (eXtreme Gradient Boosting)

History: Released 2014, dominated Kaggle competitions for years

Key Features: - Regularization: Built-in L1/L2 penalties prevent overfitting - Handling missing values: Learns best direction for missing data - Parallel processing: Fast training through parallelized tree construction - Tree pruning: Grows deep then prunes back (not greedy) - Cache awareness: Optimized memory access patterns

When to use: - ✓ You have structured/tabular data - ✓ You need high accuracy - ✓ You want automatic handling of missing values - ✓ Your dataset is medium-sized (10K-1M rows)

LightGBM (Light Gradient Boosting Machine)

History: Released by Microsoft in 2016

Key Innovation: Leaf-wise (best-first) tree growth instead of level-wise

Traditional (level-wise):

        Root
       /    \
      A      B     <- Grow entire level
     / \    / \
    C   D  E   F   <- Then next level

LightGBM (leaf-wise):

        Root
       /    \
      A      B
     / \
    C   D            <- Split the leaf with highest gain
       / \
      E   F          <- Then next best leaf

Key Features: - Faster training: Especially on large datasets - Lower memory usage: Through histogram-based binning - Better accuracy: On deep trees - Handles categorical features: Native support without encoding

When to use: - ✓ You have large datasets (>100K rows) - ✓ Training speed matters - ✓ You have categorical features - ✓ You need lower memory footprint

CatBoost (Categorical Boosting)

History: Released by Yandex in 2017

Key Innovation: Superior handling of categorical variables

Key Features: - Ordered boosting: Reduces overfitting through clever target encoding - Categorical features: No need for one-hot or label encoding - Symmetric trees: Faster prediction, better regularization - Minimal tuning: Works well with default parameters - GPU acceleration: Excellent GPU support

When to use: - ✓ You have many categorical features - ✓ You want minimal hyperparameter tuning - ✓ You need fast predictions (deployment) - ✓ You want to avoid target leakage from categorical encoding

Example 4: Comparing All Methods

Code

import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor
import xgboost as xgb
import lightgbm as lgb
from catboost import CatBoostRegressor
import time

# Load California housing dataset
from sklearn.datasets import fetch_california_housing
housing = fetch_california_housing()
X, y = housing.data, housing.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Dictionary to store results
results = {}

# 1. Random Forest (baseline)
print("Training Random Forest...")
start = time.time()
rf = RandomForestRegressor(n_estimators=100, random_state=42, n_jobs=-1)
rf.fit(X_train, y_train)
rf_time = time.time() - start
rf_pred = rf.predict(X_test)
results['Random Forest'] = {
    'R2': r2_score(y_test, rf_pred),
    'RMSE': np.sqrt(mean_squared_error(y_test, rf_pred)),
    'Time': rf_time
}

# 2. Sklearn Gradient Boosting
print("Training Sklearn GradientBoosting...")
start = time.time()
gb = GradientBoostingRegressor(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    random_state=42
)
gb.fit(X_train, y_train)
gb_time = time.time() - start
gb_pred = gb.predict(X_test)
results['GradientBoosting'] = {
    'R2': r2_score(y_test, gb_pred),
    'RMSE': np.sqrt(mean_squared_error(y_test, gb_pred)),
    'Time': gb_time
}

# 3. XGBoost
print("Training XGBoost...")
start = time.time()
xgb_model = xgb.XGBRegressor(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    random_state=42
)
xgb_model.fit(X_train, y_train)
xgb_time = time.time() - start
xgb_pred = xgb_model.predict(X_test)
results['XGBoost'] = {
    'R2': r2_score(y_test, xgb_pred),
    'RMSE': np.sqrt(mean_squared_error(y_test, xgb_pred)),
    'Time': xgb_time
}

# 4. LightGBM
print("Training LightGBM...")
start = time.time()
lgb_model = lgb.LGBMRegressor(
    n_estimators=100,
    learning_rate=0.1,
    max_depth=3,
    random_state=42,
    verbose=-1
)
lgb_model.fit(X_train, y_train)
lgb_time = time.time() - start
lgb_pred = lgb_model.predict(X_test)
results['LightGBM'] = {
    'R2': r2_score(y_test, lgb_pred),
    'RMSE': np.sqrt(mean_squared_error(y_test, lgb_pred)),
    'Time': lgb_time
}

# 5. CatBoost
print("Training CatBoost...")
start = time.time()
cat_model = CatBoostRegressor(
    iterations=100,
    learning_rate=0.1,
    depth=3,
    random_state=42,
    verbose=0
)
cat_model.fit(X_train, y_train)
cat_time = time.time() - start
cat_pred = cat_model.predict(X_test)
results['CatBoost'] = {
    'R2': r2_score(y_test, cat_pred),
    'RMSE': np.sqrt(mean_squared_error(y_test, cat_pred)),
    'Time': cat_time
}

# Display results
results_df = pd.DataFrame(results).T
results_df = results_df.sort_values('R2', ascending=False)
print("\n" + "="*60)
print("RESULTS COMPARISON")
print("="*60)
print(results_df.to_string())
print("="*60)

# Typical output:
#                     R2      RMSE    Time
# XGBoost          0.847     0.476   0.123
# LightGBM         0.846     0.477   0.089
# CatBoost         0.845     0.479   0.234
# GradientBoosting 0.842     0.483   0.543
# Random Forest    0.812     0.527   0.178

Training Random Forest...
Training Sklearn GradientBoosting...
Training XGBoost...
Training LightGBM...
Training CatBoost...

============================================================
RESULTS COMPARISON
============================================================
                        R2      RMSE      Time
Random Forest     0.804412  0.506261  0.442597
LightGBM          0.779154  0.537958  0.162263
XGBoost           0.778669  0.538548  0.118204
GradientBoosting  0.775645  0.542215  1.868890
CatBoost          0.751702  0.570414  0.119798
============================================================

Typical Observations: - Accuracy: Gradient boosting methods beat random forest - Speed: LightGBM is fastest, sklearn GB is slowest - Consistency: All three (XGB, LGB, CB) perform similarly well

Key Hyperparameters

Common to All Boosting Methods:

n_estimators / iterations: Number of trees
- More trees → better fit, but risk overfitting and slower
- Start: 100, increase to 500-1000 with proper early stopping
- Example: n_estimators=500
learning_rate / eta: How much each tree contributes
- Lower → better generalization, need more trees
- Higher → faster training, risk overfitting
- Typical: 0.01 to 0.3
- Rule: learning_rate × n_estimators ≈ constant
- Example: learning_rate=0.05
max_depth / depth: Maximum depth of each tree
- Deeper → can capture complex patterns, risk overfitting
- Shallower → faster, better regularization
- Typical: 3-8
- Example: max_depth=5
subsample: Fraction of samples used for each tree
- < 1.0 → adds randomness, prevents overfitting
- Typical: 0.8
- Example: subsample=0.8
colsample_bytree: Fraction of features per tree
- < 1.0 → adds randomness, like random forest
- Typical: 0.8
- Example: colsample_bytree=0.8

XGBoost Specific:

reg_alpha: L1 regularization
- Higher → more regularization
- Example: reg_alpha=0.1
reg_lambda: L2 regularization
- Higher → more regularization
- Example: reg_lambda=1.0

LightGBM Specific:

num_leaves: Number of leaves (instead of max_depth)
- Controls model complexity
- Rule: num_leaves < 2^max_depth
- Example: num_leaves=31
min_child_samples: Minimum samples in a leaf
- Prevents overfitting
- Example: min_child_samples=20

CatBoost Specific:

cat_features: List of categorical feature indices
- Automatic handling
- Example: cat_features=[0, 3, 5]

Example 5: Hyperparameter Tuning with Cross-Validation

Code

from sklearn.model_selection import GridSearchCV
import xgboost as xgb

# Define parameter grid
param_grid = {
    'n_estimators': [100, 300, 500],
    'learning_rate': [0.01, 0.05, 0.1],
    'max_depth': [3, 5, 7],
    'subsample': [0.8, 1.0],
    'colsample_bytree': [0.8, 1.0]
}

# XGBoost model
xgb_model = xgb.XGBRegressor(random_state=42)

# Grid search with cross-validation
grid_search = GridSearchCV(
    xgb_model,
    param_grid,
    cv=5,
    scoring='neg_mean_squared_error',
    n_jobs=-1,
    verbose=1
)

# Fit
grid_search.fit(X_train, y_train)

# Best parameters
print(f"Best parameters: {grid_search.best_params_}")
print(f"Best CV score: {np.sqrt(-grid_search.best_score_):.4f}")

# Test performance
best_model = grid_search.best_estimator_
y_pred = best_model.predict(X_test)
print(f"Test RMSE: {np.sqrt(mean_squared_error(y_test, y_pred)):.4f}")
print(f"Test R²: {r2_score(y_test, y_pred):.4f}")

Fitting 5 folds for each of 108 candidates, totalling 540 fits
Best parameters: {'colsample_bytree': 0.8, 'learning_rate': 0.05, 'max_depth': 7, 'n_estimators': 500, 'subsample': 0.8}
Best CV score: 0.4511
Test RMSE: 0.4395
Test R²: 0.8526

Example 6: Early Stopping (Crucial!)

Prevent overfitting by stopping when validation performance stops improving:

Code

from sklearn.model_selection import train_test_split
import xgboost as xgb

# Split into train and validation
X_train_sub, X_val, y_train_sub, y_val = train_test_split(
    X_train, y_train, test_size=0.2, random_state=42
)

# XGBoost with early stopping
xgb_model = xgb.XGBRegressor(
    n_estimators=1000,  # Set high
    learning_rate=0.05,
    max_depth=5,
    random_state=42,
    early_stopping_rounds=50
)

# Fit with early stopping
xgb_model.fit(
    X_train_sub, y_train_sub,
    eval_set=[(X_val, y_val)],
    verbose=10
)

print(f"Best iteration: {xgb_model.best_iteration}")
print(f"Best score: {xgb_model.best_score}")

# LightGBM version
import lightgbm as lgb

lgb_model = lgb.LGBMRegressor(
    n_estimators=1000,
    learning_rate=0.05,
    max_depth=5,
    random_state=42
)

lgb_model.fit(
    X_train_sub, y_train_sub,
    eval_set=[(X_val, y_val)],
    callbacks=[lgb.early_stopping(stopping_rounds=50)]
)

# CatBoost version
from catboost import CatBoostRegressor, Pool

cat_model = CatBoostRegressor(
    iterations=1000,
    learning_rate=0.05,
    depth=5,
    random_state=42,
    verbose=0
)

cat_model.fit(
    X_train_sub, y_train_sub,
    eval_set=(X_val, y_val),
    early_stopping_rounds=50
)

[0] validation_0-rmse:1.13896
[10]    validation_0-rmse:0.88264
[20]    validation_0-rmse:0.74563
[30]    validation_0-rmse:0.66844
[40]    validation_0-rmse:0.62387
[50]    validation_0-rmse:0.59209
[60]    validation_0-rmse:0.56613
[70]    validation_0-rmse:0.55122
[80]    validation_0-rmse:0.54012
[90]    validation_0-rmse:0.53316
[100]   validation_0-rmse:0.52811
[110]   validation_0-rmse:0.52326
[120]   validation_0-rmse:0.52000
[130]   validation_0-rmse:0.51716
[140]   validation_0-rmse:0.51433
[150]   validation_0-rmse:0.51187
[160]   validation_0-rmse:0.50988
[170]   validation_0-rmse:0.50737
[180]   validation_0-rmse:0.50490
[190]   validation_0-rmse:0.50360
[200]   validation_0-rmse:0.50163
[210]   validation_0-rmse:0.50050
[220]   validation_0-rmse:0.49979
[230]   validation_0-rmse:0.49897
[240]   validation_0-rmse:0.49833
[250]   validation_0-rmse:0.49723
[260]   validation_0-rmse:0.49696
[270]   validation_0-rmse:0.49614
[280]   validation_0-rmse:0.49487
[290]   validation_0-rmse:0.49445
[300]   validation_0-rmse:0.49393
[310]   validation_0-rmse:0.49286
[320]   validation_0-rmse:0.49218
[330]   validation_0-rmse:0.49159
[340]   validation_0-rmse:0.49066
[350]   validation_0-rmse:0.49002
[360]   validation_0-rmse:0.48991
[370]   validation_0-rmse:0.48915
[380]   validation_0-rmse:0.48830
[390]   validation_0-rmse:0.48773
[400]   validation_0-rmse:0.48721
[410]   validation_0-rmse:0.48684
[420]   validation_0-rmse:0.48693
[430]   validation_0-rmse:0.48672
[440]   validation_0-rmse:0.48612
[450]   validation_0-rmse:0.48541
[460]   validation_0-rmse:0.48500
[470]   validation_0-rmse:0.48468
[480]   validation_0-rmse:0.48422
[490]   validation_0-rmse:0.48373
[500]   validation_0-rmse:0.48312
[510]   validation_0-rmse:0.48303
[520]   validation_0-rmse:0.48228
[530]   validation_0-rmse:0.48204
[540]   validation_0-rmse:0.48154
[550]   validation_0-rmse:0.48127
[560]   validation_0-rmse:0.48094
[570]   validation_0-rmse:0.48059
[580]   validation_0-rmse:0.48050
[590]   validation_0-rmse:0.48021
[600]   validation_0-rmse:0.48010
[610]   validation_0-rmse:0.47997
[620]   validation_0-rmse:0.47983
[630]   validation_0-rmse:0.47982
[640]   validation_0-rmse:0.47979
[650]   validation_0-rmse:0.47936
[660]   validation_0-rmse:0.47903
[670]   validation_0-rmse:0.47859
[680]   validation_0-rmse:0.47827
[690]   validation_0-rmse:0.47811
[700]   validation_0-rmse:0.47802
[710]   validation_0-rmse:0.47769
[720]   validation_0-rmse:0.47763
[730]   validation_0-rmse:0.47749
[740]   validation_0-rmse:0.47763
[750]   validation_0-rmse:0.47759
[760]   validation_0-rmse:0.47752
[770]   validation_0-rmse:0.47725
[780]   validation_0-rmse:0.47715
[790]   validation_0-rmse:0.47736
[800]   validation_0-rmse:0.47709
[810]   validation_0-rmse:0.47723
[820]   validation_0-rmse:0.47726
[830]   validation_0-rmse:0.47728
[840]   validation_0-rmse:0.47724
[850]   validation_0-rmse:0.47696
[860]   validation_0-rmse:0.47680
[870]   validation_0-rmse:0.47644
[880]   validation_0-rmse:0.47632
[890]   validation_0-rmse:0.47624
[900]   validation_0-rmse:0.47627
[910]   validation_0-rmse:0.47604
[920]   validation_0-rmse:0.47592
[930]   validation_0-rmse:0.47603
[940]   validation_0-rmse:0.47588
[950]   validation_0-rmse:0.47583
[960]   validation_0-rmse:0.47562
[970]   validation_0-rmse:0.47542
[980]   validation_0-rmse:0.47533
[990]   validation_0-rmse:0.47517
[999]   validation_0-rmse:0.47519
Best iteration: 987
Best score: 0.47513507080438516
Training until validation scores don't improve for 50 rounds

/Users/tgardos/Source/courses/ds701/DS701-Course-Notes-FA26/.venv/lib/python3.12/site-packages/lightgbm/sklearn.py:1106: LGBMDeprecationWarning: The argument 'eval_set' is deprecated, use 'eval_X' and 'eval_y' instead.
  eval_set = _validate_eval_set_Xy(eval_set=eval_set, eval_X=eval_X, eval_y=eval_y)

Did not meet early stopping. Best iteration is:
[1000]  valid_0's l2: 0.225795

CatBoostRegressor(depth=5, iterations=1000, learning_rate=0.05, loss_function='RMSE', random_state=42, verbose=0)

Best Practice: Always use early stopping with a validation set!

Example 7: Feature Importance

Code

import matplotlib.pyplot as plt

# Train XGBoost
xgb_model = xgb.XGBRegressor(n_estimators=100, random_state=42)
xgb_model.fit(X_train, y_train)

# Get feature importance
importance = xgb_model.feature_importances_
feature_names = housing.feature_names

# Create DataFrame for easy sorting
importance_df = pd.DataFrame({
    'Feature': feature_names,
    'Importance': importance
}).sort_values('Importance', ascending=False)

# Plot
plt.figure(figsize=(10, 6))
plt.barh(importance_df['Feature'], importance_df['Importance'])
plt.xlabel('Importance')
plt.title('XGBoost Feature Importance')
plt.tight_layout()
plt.show()

print(importance_df)

# Multiple importance types in XGBoost
xgb.plot_importance(xgb_model, importance_type='gain')  # Average gain
xgb.plot_importance(xgb_model, importance_type='weight')  # Number of splits
xgb.plot_importance(xgb_model, importance_type='cover')  # Coverage
plt.show()

      Feature  Importance
0      MedInc    0.479923
5    AveOccup    0.158646
7   Longitude    0.108794
6    Latitude    0.096155
1    HouseAge    0.065551
2    AveRooms    0.042610
3   AveBedrms    0.025066
4  Population    0.023254

Example 8: Handling Categorical Features with CatBoost

Code

from catboost import CatBoostRegressor
import pandas as pd

# Create sample data with categorical features
data = pd.DataFrame({
    'size': [1200, 1500, 1800, 2000, 2200, 2500],
    'neighborhood': ['A', 'B', 'A', 'C', 'B', 'C'],
    'style': ['Modern', 'Victorian', 'Modern', 'Ranch', 'Victorian', 'Ranch'],
    'price': [200, 250, 280, 320, 300, 380]
})

X = data[['size', 'neighborhood', 'style']]
y = data['price']

# Identify categorical columns
cat_features = ['neighborhood', 'style']  # or indices [1, 2]

# CatBoost handles categorical features natively!
model = CatBoostRegressor(
    iterations=100,
    learning_rate=0.1,
    depth=3,
    cat_features=cat_features,
    random_state=42,
    verbose=0
)

model.fit(X, y)

# Make predictions (can handle categorical inputs directly)
new_house = pd.DataFrame({
    'size': [1600],
    'neighborhood': ['A'],
    'style': ['Modern']
})
prediction = model.predict(new_house)
print(f"Predicted price: ${prediction[0]:.0f}K")

Predicted price: $214K

Why this matters: No need for label encoding or one-hot encoding, which can: - Create sparse high-dimensional data - Introduce order where none exists - Cause target leakage

Comparison Table: Which Method to Choose?

Criterion	Random Forest	XGBoost	LightGBM	CatBoost
Accuracy	Good	Excellent	Excellent	Excellent
Training Speed	Medium	Medium	Fast	Slow
Prediction Speed	Slow	Medium	Medium	Fast
Memory Usage	High	Medium	Low	Medium
Large Datasets (>1M)	✗	✓	✓✓	✓
Categorical Features	✗	△	✓	✓✓
Handles Missing Data	✓	✓✓	✓	✓
Overfitting Risk	Low	Medium	High	Low
Hyperparameter Tuning	Easy	Medium	Hard	Easy
GPU Support	✗	✓	✓	✓✓
Interpretability	Medium	Medium	Medium	Medium

Decision Guide:

Use XGBoost if: - You want excellent out-of-box performance - You have missing data - You need a balance of speed and accuracy - You’re used to it (most tutorials use XGBoost)

Use LightGBM if: - You have large datasets (>100K rows) - Training speed is critical - You need low memory footprint - You have some categorical features

Use CatBoost if: - You have many categorical features - You want minimal hyperparameter tuning - Prediction speed (deployment) matters - You want to avoid target leakage

Use Random Forest if: - You need an easy baseline - Interpretability is important - You have small data (<10K rows) - You want parallel training with zero dependencies between models

Common Pitfalls and Best Practices

Pitfall 1: Not Using Early Stopping

Code

# ❌ BAD: Fixed number of trees, might overfit
model = xgb.XGBRegressor(n_estimators=1000)
model.fit(X_train, y_train)

# ✅ GOOD: Early stopping prevents overfitting
model = xgb.XGBRegressor(n_estimators=1000, early_stopping_rounds=50)
model.fit(
    X_train, y_train,
    eval_set=[(X_val, y_val)]
)

[0] validation_0-rmse:0.95177
[1] validation_0-rmse:0.80297
[2] validation_0-rmse:0.70462
[3] validation_0-rmse:0.64292
[4] validation_0-rmse:0.60112
[5] validation_0-rmse:0.56877
[6] validation_0-rmse:0.54492
[7] validation_0-rmse:0.52394
[8] validation_0-rmse:0.51287
[9] validation_0-rmse:0.50102
[10]    validation_0-rmse:0.49031
[11]    validation_0-rmse:0.48151
[12]    validation_0-rmse:0.47407
[13]    validation_0-rmse:0.46265
[14]    validation_0-rmse:0.45749
[15]    validation_0-rmse:0.45096
[16]    validation_0-rmse:0.44656
[17]    validation_0-rmse:0.44195
[18]    validation_0-rmse:0.43807
[19]    validation_0-rmse:0.43346
[20]    validation_0-rmse:0.42855
[21]    validation_0-rmse:0.42571
[22]    validation_0-rmse:0.42075
[23]    validation_0-rmse:0.41870
[24]    validation_0-rmse:0.41484
[25]    validation_0-rmse:0.41050
[26]    validation_0-rmse:0.40721
[27]    validation_0-rmse:0.40342
[28]    validation_0-rmse:0.40062
[29]    validation_0-rmse:0.39870
[30]    validation_0-rmse:0.39655
[31]    validation_0-rmse:0.39485
[32]    validation_0-rmse:0.39123
[33]    validation_0-rmse:0.38855
[34]    validation_0-rmse:0.38416
[35]    validation_0-rmse:0.38039
[36]    validation_0-rmse:0.37846
[37]    validation_0-rmse:0.37666
[38]    validation_0-rmse:0.37461
[39]    validation_0-rmse:0.37242
[40]    validation_0-rmse:0.37054
[41]    validation_0-rmse:0.36732
[42]    validation_0-rmse:0.36498
[43]    validation_0-rmse:0.36343
[44]    validation_0-rmse:0.36215
[45]    validation_0-rmse:0.36107
[46]    validation_0-rmse:0.35935
[47]    validation_0-rmse:0.35616
[48]    validation_0-rmse:0.35346
[49]    validation_0-rmse:0.35216
[50]    validation_0-rmse:0.35092
[51]    validation_0-rmse:0.34854
[52]    validation_0-rmse:0.34507
[53]    validation_0-rmse:0.34270
[54]    validation_0-rmse:0.33935
[55]    validation_0-rmse:0.33707
[56]    validation_0-rmse:0.33656
[57]    validation_0-rmse:0.33473
[58]    validation_0-rmse:0.33351
[59]    validation_0-rmse:0.33253
[60]    validation_0-rmse:0.33153
[61]    validation_0-rmse:0.33034
[62]    validation_0-rmse:0.32942
[63]    validation_0-rmse:0.32805
[64]    validation_0-rmse:0.32705
[65]    validation_0-rmse:0.32483
[66]    validation_0-rmse:0.32358
[67]    validation_0-rmse:0.32146
[68]    validation_0-rmse:0.31919
[69]    validation_0-rmse:0.31719
[70]    validation_0-rmse:0.31526
[71]    validation_0-rmse:0.31455
[72]    validation_0-rmse:0.31388
[73]    validation_0-rmse:0.31162
[74]    validation_0-rmse:0.30966
[75]    validation_0-rmse:0.30883
[76]    validation_0-rmse:0.30754
[77]    validation_0-rmse:0.30632
[78]    validation_0-rmse:0.30559
[79]    validation_0-rmse:0.30376
[80]    validation_0-rmse:0.30188
[81]    validation_0-rmse:0.30032
[82]    validation_0-rmse:0.29968
[83]    validation_0-rmse:0.29900
[84]    validation_0-rmse:0.29795
[85]    validation_0-rmse:0.29650
[86]    validation_0-rmse:0.29556
[87]    validation_0-rmse:0.29315
[88]    validation_0-rmse:0.29258
[89]    validation_0-rmse:0.29135
[90]    validation_0-rmse:0.29050
[91]    validation_0-rmse:0.28951
[92]    validation_0-rmse:0.28871
[93]    validation_0-rmse:0.28753
[94]    validation_0-rmse:0.28704
[95]    validation_0-rmse:0.28565
[96]    validation_0-rmse:0.28405
[97]    validation_0-rmse:0.28282
[98]    validation_0-rmse:0.28216
[99]    validation_0-rmse:0.28114
[100]   validation_0-rmse:0.27989
[101]   validation_0-rmse:0.27907
[102]   validation_0-rmse:0.27854
[103]   validation_0-rmse:0.27731
[104]   validation_0-rmse:0.27608
[105]   validation_0-rmse:0.27546
[106]   validation_0-rmse:0.27338
[107]   validation_0-rmse:0.27248
[108]   validation_0-rmse:0.27161
[109]   validation_0-rmse:0.27125
[110]   validation_0-rmse:0.27074
[111]   validation_0-rmse:0.26951
[112]   validation_0-rmse:0.26831
[113]   validation_0-rmse:0.26661
[114]   validation_0-rmse:0.26433
[115]   validation_0-rmse:0.26349
[116]   validation_0-rmse:0.26287
[117]   validation_0-rmse:0.26163
[118]   validation_0-rmse:0.26055
[119]   validation_0-rmse:0.26003
[120]   validation_0-rmse:0.25897
[121]   validation_0-rmse:0.25763
[122]   validation_0-rmse:0.25612
[123]   validation_0-rmse:0.25551
[124]   validation_0-rmse:0.25521
[125]   validation_0-rmse:0.25481
[126]   validation_0-rmse:0.25402
[127]   validation_0-rmse:0.25273
[128]   validation_0-rmse:0.25114
[129]   validation_0-rmse:0.24966
[130]   validation_0-rmse:0.24854
[131]   validation_0-rmse:0.24768
[132]   validation_0-rmse:0.24658
[133]   validation_0-rmse:0.24598
[134]   validation_0-rmse:0.24476
[135]   validation_0-rmse:0.24374
[136]   validation_0-rmse:0.24295
[137]   validation_0-rmse:0.24235
[138]   validation_0-rmse:0.24127
[139]   validation_0-rmse:0.24010
[140]   validation_0-rmse:0.23918
[141]   validation_0-rmse:0.23852
[142]   validation_0-rmse:0.23757
[143]   validation_0-rmse:0.23712
[144]   validation_0-rmse:0.23565
[145]   validation_0-rmse:0.23451
[146]   validation_0-rmse:0.23372
[147]   validation_0-rmse:0.23321
[148]   validation_0-rmse:0.23286
[149]   validation_0-rmse:0.23193
[150]   validation_0-rmse:0.23150
[151]   validation_0-rmse:0.23117
[152]   validation_0-rmse:0.23004
[153]   validation_0-rmse:0.22889
[154]   validation_0-rmse:0.22811
[155]   validation_0-rmse:0.22719
[156]   validation_0-rmse:0.22616
[157]   validation_0-rmse:0.22561
[158]   validation_0-rmse:0.22523
[159]   validation_0-rmse:0.22474
[160]   validation_0-rmse:0.22419
[161]   validation_0-rmse:0.22354
[162]   validation_0-rmse:0.22266
[163]   validation_0-rmse:0.22190
[164]   validation_0-rmse:0.22112
[165]   validation_0-rmse:0.22072
[166]   validation_0-rmse:0.21991
[167]   validation_0-rmse:0.21932
[168]   validation_0-rmse:0.21835
[169]   validation_0-rmse:0.21769
[170]   validation_0-rmse:0.21728
[171]   validation_0-rmse:0.21688
[172]   validation_0-rmse:0.21610
[173]   validation_0-rmse:0.21554
[174]   validation_0-rmse:0.21517
[175]   validation_0-rmse:0.21418
[176]   validation_0-rmse:0.21366
[177]   validation_0-rmse:0.21294
[178]   validation_0-rmse:0.21268
[179]   validation_0-rmse:0.21208
[180]   validation_0-rmse:0.21160
[181]   validation_0-rmse:0.21103
[182]   validation_0-rmse:0.21035
[183]   validation_0-rmse:0.20917
[184]   validation_0-rmse:0.20854
[185]   validation_0-rmse:0.20781
[186]   validation_0-rmse:0.20729
[187]   validation_0-rmse:0.20668
[188]   validation_0-rmse:0.20566
[189]   validation_0-rmse:0.20538
[190]   validation_0-rmse:0.20426
[191]   validation_0-rmse:0.20371
[192]   validation_0-rmse:0.20327
[193]   validation_0-rmse:0.20301
[194]   validation_0-rmse:0.20238
[195]   validation_0-rmse:0.20150
[196]   validation_0-rmse:0.20100
[197]   validation_0-rmse:0.20066
[198]   validation_0-rmse:0.20050
[199]   validation_0-rmse:0.19959
[200]   validation_0-rmse:0.19883
[201]   validation_0-rmse:0.19844
[202]   validation_0-rmse:0.19758
[203]   validation_0-rmse:0.19688
[204]   validation_0-rmse:0.19623
[205]   validation_0-rmse:0.19596
[206]   validation_0-rmse:0.19564
[207]   validation_0-rmse:0.19505
[208]   validation_0-rmse:0.19465
[209]   validation_0-rmse:0.19423
[210]   validation_0-rmse:0.19343
[211]   validation_0-rmse:0.19266
[212]   validation_0-rmse:0.19211
[213]   validation_0-rmse:0.19141
[214]   validation_0-rmse:0.19123
[215]   validation_0-rmse:0.19088
[216]   validation_0-rmse:0.18986
[217]   validation_0-rmse:0.18935
[218]   validation_0-rmse:0.18859
[219]   validation_0-rmse:0.18783
[220]   validation_0-rmse:0.18715
[221]   validation_0-rmse:0.18652
[222]   validation_0-rmse:0.18598
[223]   validation_0-rmse:0.18566
[224]   validation_0-rmse:0.18554
[225]   validation_0-rmse:0.18522
[226]   validation_0-rmse:0.18480
[227]   validation_0-rmse:0.18431
[228]   validation_0-rmse:0.18364
[229]   validation_0-rmse:0.18293
[230]   validation_0-rmse:0.18204
[231]   validation_0-rmse:0.18198
[232]   validation_0-rmse:0.18156
[233]   validation_0-rmse:0.18062
[234]   validation_0-rmse:0.18035
[235]   validation_0-rmse:0.17941
[236]   validation_0-rmse:0.17891
[237]   validation_0-rmse:0.17829
[238]   validation_0-rmse:0.17788
[239]   validation_0-rmse:0.17740
[240]   validation_0-rmse:0.17707
[241]   validation_0-rmse:0.17590
[242]   validation_0-rmse:0.17488
[243]   validation_0-rmse:0.17451
[244]   validation_0-rmse:0.17409
[245]   validation_0-rmse:0.17373
[246]   validation_0-rmse:0.17303
[247]   validation_0-rmse:0.17261
[248]   validation_0-rmse:0.17217
[249]   validation_0-rmse:0.17175
[250]   validation_0-rmse:0.17138
[251]   validation_0-rmse:0.17077
[252]   validation_0-rmse:0.16993
[253]   validation_0-rmse:0.16960
[254]   validation_0-rmse:0.16913
[255]   validation_0-rmse:0.16852
[256]   validation_0-rmse:0.16827
[257]   validation_0-rmse:0.16769
[258]   validation_0-rmse:0.16753
[259]   validation_0-rmse:0.16724
[260]   validation_0-rmse:0.16654
[261]   validation_0-rmse:0.16587
[262]   validation_0-rmse:0.16583
[263]   validation_0-rmse:0.16560
[264]   validation_0-rmse:0.16524
[265]   validation_0-rmse:0.16523
[266]   validation_0-rmse:0.16436
[267]   validation_0-rmse:0.16368
[268]   validation_0-rmse:0.16305
[269]   validation_0-rmse:0.16283
[270]   validation_0-rmse:0.16272
[271]   validation_0-rmse:0.16214
[272]   validation_0-rmse:0.16195
[273]   validation_0-rmse:0.16123
[274]   validation_0-rmse:0.16079
[275]   validation_0-rmse:0.16024
[276]   validation_0-rmse:0.16011
[277]   validation_0-rmse:0.15934
[278]   validation_0-rmse:0.15898
[279]   validation_0-rmse:0.15839
[280]   validation_0-rmse:0.15774
[281]   validation_0-rmse:0.15685
[282]   validation_0-rmse:0.15604
[283]   validation_0-rmse:0.15563
[284]   validation_0-rmse:0.15531
[285]   validation_0-rmse:0.15482
[286]   validation_0-rmse:0.15450
[287]   validation_0-rmse:0.15416
[288]   validation_0-rmse:0.15344
[289]   validation_0-rmse:0.15284
[290]   validation_0-rmse:0.15232
[291]   validation_0-rmse:0.15197
[292]   validation_0-rmse:0.15178
[293]   validation_0-rmse:0.15141
[294]   validation_0-rmse:0.15108
[295]   validation_0-rmse:0.15030
[296]   validation_0-rmse:0.14985
[297]   validation_0-rmse:0.14942
[298]   validation_0-rmse:0.14929
[299]   validation_0-rmse:0.14868
[300]   validation_0-rmse:0.14820
[301]   validation_0-rmse:0.14781
[302]   validation_0-rmse:0.14750
[303]   validation_0-rmse:0.14724
[304]   validation_0-rmse:0.14702
[305]   validation_0-rmse:0.14639
[306]   validation_0-rmse:0.14568
[307]   validation_0-rmse:0.14546
[308]   validation_0-rmse:0.14512
[309]   validation_0-rmse:0.14470
[310]   validation_0-rmse:0.14397
[311]   validation_0-rmse:0.14348
[312]   validation_0-rmse:0.14292
[313]   validation_0-rmse:0.14253
[314]   validation_0-rmse:0.14185
[315]   validation_0-rmse:0.14134
[316]   validation_0-rmse:0.14105
[317]   validation_0-rmse:0.14060
[318]   validation_0-rmse:0.14023
[319]   validation_0-rmse:0.13993
[320]   validation_0-rmse:0.13923
[321]   validation_0-rmse:0.13907
[322]   validation_0-rmse:0.13900
[323]   validation_0-rmse:0.13866
[324]   validation_0-rmse:0.13823
[325]   validation_0-rmse:0.13780
[326]   validation_0-rmse:0.13756
[327]   validation_0-rmse:0.13696
[328]   validation_0-rmse:0.13639
[329]   validation_0-rmse:0.13590
[330]   validation_0-rmse:0.13534
[331]   validation_0-rmse:0.13512
[332]   validation_0-rmse:0.13480
[333]   validation_0-rmse:0.13429
[334]   validation_0-rmse:0.13341
[335]   validation_0-rmse:0.13314
[336]   validation_0-rmse:0.13294
[337]   validation_0-rmse:0.13247
[338]   validation_0-rmse:0.13225
[339]   validation_0-rmse:0.13195
[340]   validation_0-rmse:0.13170
[341]   validation_0-rmse:0.13100
[342]   validation_0-rmse:0.13070
[343]   validation_0-rmse:0.13011
[344]   validation_0-rmse:0.12963
[345]   validation_0-rmse:0.12928
[346]   validation_0-rmse:0.12891
[347]   validation_0-rmse:0.12883
[348]   validation_0-rmse:0.12873
[349]   validation_0-rmse:0.12796
[350]   validation_0-rmse:0.12754
[351]   validation_0-rmse:0.12745
[352]   validation_0-rmse:0.12715
[353]   validation_0-rmse:0.12662
[354]   validation_0-rmse:0.12596
[355]   validation_0-rmse:0.12570
[356]   validation_0-rmse:0.12506
[357]   validation_0-rmse:0.12492
[358]   validation_0-rmse:0.12446
[359]   validation_0-rmse:0.12407
[360]   validation_0-rmse:0.12362
[361]   validation_0-rmse:0.12341
[362]   validation_0-rmse:0.12306
[363]   validation_0-rmse:0.12297
[364]   validation_0-rmse:0.12276
[365]   validation_0-rmse:0.12242
[366]   validation_0-rmse:0.12200
[367]   validation_0-rmse:0.12185
[368]   validation_0-rmse:0.12154
[369]   validation_0-rmse:0.12118
[370]   validation_0-rmse:0.12104
[371]   validation_0-rmse:0.12036
[372]   validation_0-rmse:0.12010
[373]   validation_0-rmse:0.11987
[374]   validation_0-rmse:0.11931
[375]   validation_0-rmse:0.11901
[376]   validation_0-rmse:0.11893
[377]   validation_0-rmse:0.11881
[378]   validation_0-rmse:0.11845
[379]   validation_0-rmse:0.11819
[380]   validation_0-rmse:0.11780
[381]   validation_0-rmse:0.11722
[382]   validation_0-rmse:0.11687
[383]   validation_0-rmse:0.11654
[384]   validation_0-rmse:0.11608
[385]   validation_0-rmse:0.11567
[386]   validation_0-rmse:0.11548
[387]   validation_0-rmse:0.11516
[388]   validation_0-rmse:0.11486
[389]   validation_0-rmse:0.11459
[390]   validation_0-rmse:0.11417
[391]   validation_0-rmse:0.11369
[392]   validation_0-rmse:0.11344
[393]   validation_0-rmse:0.11329
[394]   validation_0-rmse:0.11299
[395]   validation_0-rmse:0.11287
[396]   validation_0-rmse:0.11262
[397]   validation_0-rmse:0.11248
[398]   validation_0-rmse:0.11221
[399]   validation_0-rmse:0.11205
[400]   validation_0-rmse:0.11179
[401]   validation_0-rmse:0.11144
[402]   validation_0-rmse:0.11133
[403]   validation_0-rmse:0.11116
[404]   validation_0-rmse:0.11058
[405]   validation_0-rmse:0.11018
[406]   validation_0-rmse:0.10995
[407]   validation_0-rmse:0.10958
[408]   validation_0-rmse:0.10912
[409]   validation_0-rmse:0.10860
[410]   validation_0-rmse:0.10820
[411]   validation_0-rmse:0.10796
[412]   validation_0-rmse:0.10783
[413]   validation_0-rmse:0.10761
[414]   validation_0-rmse:0.10733
[415]   validation_0-rmse:0.10720
[416]   validation_0-rmse:0.10703
[417]   validation_0-rmse:0.10691
[418]   validation_0-rmse:0.10673
[419]   validation_0-rmse:0.10640
[420]   validation_0-rmse:0.10594
[421]   validation_0-rmse:0.10566
[422]   validation_0-rmse:0.10536
[423]   validation_0-rmse:0.10506
[424]   validation_0-rmse:0.10471
[425]   validation_0-rmse:0.10440
[426]   validation_0-rmse:0.10410
[427]   validation_0-rmse:0.10388
[428]   validation_0-rmse:0.10333
[429]   validation_0-rmse:0.10306
[430]   validation_0-rmse:0.10290
[431]   validation_0-rmse:0.10263
[432]   validation_0-rmse:0.10237
[433]   validation_0-rmse:0.10211
[434]   validation_0-rmse:0.10201
[435]   validation_0-rmse:0.10168
[436]   validation_0-rmse:0.10144
[437]   validation_0-rmse:0.10103
[438]   validation_0-rmse:0.10094
[439]   validation_0-rmse:0.10078
[440]   validation_0-rmse:0.10036
[441]   validation_0-rmse:0.10023
[442]   validation_0-rmse:0.09998
[443]   validation_0-rmse:0.09951
[444]   validation_0-rmse:0.09925
[445]   validation_0-rmse:0.09909
[446]   validation_0-rmse:0.09890
[447]   validation_0-rmse:0.09854
[448]   validation_0-rmse:0.09839
[449]   validation_0-rmse:0.09802
[450]   validation_0-rmse:0.09786
[451]   validation_0-rmse:0.09756
[452]   validation_0-rmse:0.09740
[453]   validation_0-rmse:0.09721
[454]   validation_0-rmse:0.09690
[455]   validation_0-rmse:0.09673
[456]   validation_0-rmse:0.09650
[457]   validation_0-rmse:0.09598
[458]   validation_0-rmse:0.09556
[459]   validation_0-rmse:0.09527
[460]   validation_0-rmse:0.09501
[461]   validation_0-rmse:0.09494
[462]   validation_0-rmse:0.09451
[463]   validation_0-rmse:0.09434
[464]   validation_0-rmse:0.09394
[465]   validation_0-rmse:0.09385
[466]   validation_0-rmse:0.09356
[467]   validation_0-rmse:0.09329
[468]   validation_0-rmse:0.09313
[469]   validation_0-rmse:0.09261
[470]   validation_0-rmse:0.09223
[471]   validation_0-rmse:0.09199
[472]   validation_0-rmse:0.09183
[473]   validation_0-rmse:0.09151
[474]   validation_0-rmse:0.09129
[475]   validation_0-rmse:0.09095
[476]   validation_0-rmse:0.09081
[477]   validation_0-rmse:0.09042
[478]   validation_0-rmse:0.09017
[479]   validation_0-rmse:0.08997
[480]   validation_0-rmse:0.08966
[481]   validation_0-rmse:0.08955
[482]   validation_0-rmse:0.08925
[483]   validation_0-rmse:0.08907
[484]   validation_0-rmse:0.08875
[485]   validation_0-rmse:0.08858
[486]   validation_0-rmse:0.08833
[487]   validation_0-rmse:0.08809
[488]   validation_0-rmse:0.08799
[489]   validation_0-rmse:0.08787
[490]   validation_0-rmse:0.08767
[491]   validation_0-rmse:0.08737
[492]   validation_0-rmse:0.08721
[493]   validation_0-rmse:0.08693
[494]   validation_0-rmse:0.08672
[495]   validation_0-rmse:0.08649
[496]   validation_0-rmse:0.08632
[497]   validation_0-rmse:0.08622
[498]   validation_0-rmse:0.08616
[499]   validation_0-rmse:0.08597
[500]   validation_0-rmse:0.08572
[501]   validation_0-rmse:0.08540
[502]   validation_0-rmse:0.08509
[503]   validation_0-rmse:0.08476
[504]   validation_0-rmse:0.08457
[505]   validation_0-rmse:0.08436
[506]   validation_0-rmse:0.08425
[507]   validation_0-rmse:0.08399
[508]   validation_0-rmse:0.08368
[509]   validation_0-rmse:0.08352
[510]   validation_0-rmse:0.08314
[511]   validation_0-rmse:0.08285
[512]   validation_0-rmse:0.08268
[513]   validation_0-rmse:0.08245
[514]   validation_0-rmse:0.08232
[515]   validation_0-rmse:0.08217
[516]   validation_0-rmse:0.08189
[517]   validation_0-rmse:0.08170
[518]   validation_0-rmse:0.08165
[519]   validation_0-rmse:0.08150
[520]   validation_0-rmse:0.08133
[521]   validation_0-rmse:0.08125
[522]   validation_0-rmse:0.08104
[523]   validation_0-rmse:0.08097
[524]   validation_0-rmse:0.08072
[525]   validation_0-rmse:0.08066
[526]   validation_0-rmse:0.08042
[527]   validation_0-rmse:0.08027
[528]   validation_0-rmse:0.08015
[529]   validation_0-rmse:0.07996
[530]   validation_0-rmse:0.07971
[531]   validation_0-rmse:0.07943
[532]   validation_0-rmse:0.07921
[533]   validation_0-rmse:0.07888
[534]   validation_0-rmse:0.07882
[535]   validation_0-rmse:0.07855
[536]   validation_0-rmse:0.07835
[537]   validation_0-rmse:0.07802
[538]   validation_0-rmse:0.07795
[539]   validation_0-rmse:0.07762
[540]   validation_0-rmse:0.07757
[541]   validation_0-rmse:0.07737
[542]   validation_0-rmse:0.07705
[543]   validation_0-rmse:0.07688
[544]   validation_0-rmse:0.07650
[545]   validation_0-rmse:0.07635
[546]   validation_0-rmse:0.07619
[547]   validation_0-rmse:0.07608
[548]   validation_0-rmse:0.07587
[549]   validation_0-rmse:0.07564
[550]   validation_0-rmse:0.07539
[551]   validation_0-rmse:0.07508
[552]   validation_0-rmse:0.07478
[553]   validation_0-rmse:0.07466
[554]   validation_0-rmse:0.07445
[555]   validation_0-rmse:0.07437
[556]   validation_0-rmse:0.07409
[557]   validation_0-rmse:0.07407
[558]   validation_0-rmse:0.07387
[559]   validation_0-rmse:0.07355
[560]   validation_0-rmse:0.07352
[561]   validation_0-rmse:0.07347
[562]   validation_0-rmse:0.07317
[563]   validation_0-rmse:0.07306
[564]   validation_0-rmse:0.07277
[565]   validation_0-rmse:0.07251
[566]   validation_0-rmse:0.07220
[567]   validation_0-rmse:0.07195
[568]   validation_0-rmse:0.07189
[569]   validation_0-rmse:0.07184
[570]   validation_0-rmse:0.07163
[571]   validation_0-rmse:0.07148
[572]   validation_0-rmse:0.07134
[573]   validation_0-rmse:0.07114
[574]   validation_0-rmse:0.07107
[575]   validation_0-rmse:0.07104
[576]   validation_0-rmse:0.07093
[577]   validation_0-rmse:0.07082
[578]   validation_0-rmse:0.07075
[579]   validation_0-rmse:0.07053
[580]   validation_0-rmse:0.07041
[581]   validation_0-rmse:0.07025
[582]   validation_0-rmse:0.07019
[583]   validation_0-rmse:0.06996
[584]   validation_0-rmse:0.06981
[585]   validation_0-rmse:0.06949
[586]   validation_0-rmse:0.06924
[587]   validation_0-rmse:0.06898
[588]   validation_0-rmse:0.06845
[589]   validation_0-rmse:0.06816
[590]   validation_0-rmse:0.06801
[591]   validation_0-rmse:0.06788
[592]   validation_0-rmse:0.06768
[593]   validation_0-rmse:0.06757
[594]   validation_0-rmse:0.06742
[595]   validation_0-rmse:0.06733
[596]   validation_0-rmse:0.06720
[597]   validation_0-rmse:0.06708
[598]   validation_0-rmse:0.06686
[599]   validation_0-rmse:0.06680
[600]   validation_0-rmse:0.06667
[601]   validation_0-rmse:0.06640
[602]   validation_0-rmse:0.06611
[603]   validation_0-rmse:0.06575
[604]   validation_0-rmse:0.06547
[605]   validation_0-rmse:0.06538
[606]   validation_0-rmse:0.06505
[607]   validation_0-rmse:0.06499
[608]   validation_0-rmse:0.06485
[609]   validation_0-rmse:0.06451
[610]   validation_0-rmse:0.06438
[611]   validation_0-rmse:0.06432
[612]   validation_0-rmse:0.06416
[613]   validation_0-rmse:0.06389
[614]   validation_0-rmse:0.06388
[615]   validation_0-rmse:0.06371
[616]   validation_0-rmse:0.06363
[617]   validation_0-rmse:0.06338
[618]   validation_0-rmse:0.06329
[619]   validation_0-rmse:0.06315
[620]   validation_0-rmse:0.06300
[621]   validation_0-rmse:0.06298
[622]   validation_0-rmse:0.06289
[623]   validation_0-rmse:0.06252
[624]   validation_0-rmse:0.06230
[625]   validation_0-rmse:0.06210
[626]   validation_0-rmse:0.06193
[627]   validation_0-rmse:0.06180
[628]   validation_0-rmse:0.06157
[629]   validation_0-rmse:0.06149
[630]   validation_0-rmse:0.06135
[631]   validation_0-rmse:0.06117
[632]   validation_0-rmse:0.06112
[633]   validation_0-rmse:0.06106
[634]   validation_0-rmse:0.06090
[635]   validation_0-rmse:0.06070
[636]   validation_0-rmse:0.06059
[637]   validation_0-rmse:0.06045
[638]   validation_0-rmse:0.06020
[639]   validation_0-rmse:0.05991
[640]   validation_0-rmse:0.05978
[641]   validation_0-rmse:0.05964
[642]   validation_0-rmse:0.05955
[643]   validation_0-rmse:0.05927
[644]   validation_0-rmse:0.05924
[645]   validation_0-rmse:0.05916
[646]   validation_0-rmse:0.05895
[647]   validation_0-rmse:0.05884
[648]   validation_0-rmse:0.05868
[649]   validation_0-rmse:0.05854
[650]   validation_0-rmse:0.05838
[651]   validation_0-rmse:0.05820
[652]   validation_0-rmse:0.05793
[653]   validation_0-rmse:0.05761
[654]   validation_0-rmse:0.05745
[655]   validation_0-rmse:0.05716
[656]   validation_0-rmse:0.05712
[657]   validation_0-rmse:0.05698
[658]   validation_0-rmse:0.05691
[659]   validation_0-rmse:0.05681
[660]   validation_0-rmse:0.05676
[661]   validation_0-rmse:0.05668
[662]   validation_0-rmse:0.05640
[663]   validation_0-rmse:0.05634
[664]   validation_0-rmse:0.05615
[665]   validation_0-rmse:0.05603
[666]   validation_0-rmse:0.05589
[667]   validation_0-rmse:0.05566
[668]   validation_0-rmse:0.05554
[669]   validation_0-rmse:0.05547
[670]   validation_0-rmse:0.05533
[671]   validation_0-rmse:0.05520
[672]   validation_0-rmse:0.05501
[673]   validation_0-rmse:0.05493
[674]   validation_0-rmse:0.05489
[675]   validation_0-rmse:0.05484
[676]   validation_0-rmse:0.05480
[677]   validation_0-rmse:0.05468
[678]   validation_0-rmse:0.05454
[679]   validation_0-rmse:0.05440
[680]   validation_0-rmse:0.05436
[681]   validation_0-rmse:0.05416
[682]   validation_0-rmse:0.05408
[683]   validation_0-rmse:0.05396
[684]   validation_0-rmse:0.05387
[685]   validation_0-rmse:0.05375
[686]   validation_0-rmse:0.05358
[687]   validation_0-rmse:0.05349
[688]   validation_0-rmse:0.05328
[689]   validation_0-rmse:0.05314
[690]   validation_0-rmse:0.05311
[691]   validation_0-rmse:0.05302
[692]   validation_0-rmse:0.05295
[693]   validation_0-rmse:0.05275
[694]   validation_0-rmse:0.05258
[695]   validation_0-rmse:0.05250
[696]   validation_0-rmse:0.05242
[697]   validation_0-rmse:0.05237
[698]   validation_0-rmse:0.05222
[699]   validation_0-rmse:0.05207
[700]   validation_0-rmse:0.05198
[701]   validation_0-rmse:0.05196
[702]   validation_0-rmse:0.05189
[703]   validation_0-rmse:0.05175
[704]   validation_0-rmse:0.05167
[705]   validation_0-rmse:0.05162
[706]   validation_0-rmse:0.05155
[707]   validation_0-rmse:0.05148
[708]   validation_0-rmse:0.05142
[709]   validation_0-rmse:0.05132
[710]   validation_0-rmse:0.05117
[711]   validation_0-rmse:0.05098
[712]   validation_0-rmse:0.05090
[713]   validation_0-rmse:0.05072
[714]   validation_0-rmse:0.05054
[715]   validation_0-rmse:0.05045
[716]   validation_0-rmse:0.05036
[717]   validation_0-rmse:0.05014
[718]   validation_0-rmse:0.05009
[719]   validation_0-rmse:0.04986
[720]   validation_0-rmse:0.04971
[721]   validation_0-rmse:0.04953
[722]   validation_0-rmse:0.04948
[723]   validation_0-rmse:0.04930
[724]   validation_0-rmse:0.04923
[725]   validation_0-rmse:0.04914
[726]   validation_0-rmse:0.04905
[727]   validation_0-rmse:0.04880
[728]   validation_0-rmse:0.04862
[729]   validation_0-rmse:0.04854
[730]   validation_0-rmse:0.04846
[731]   validation_0-rmse:0.04844
[732]   validation_0-rmse:0.04822
[733]   validation_0-rmse:0.04817
[734]   validation_0-rmse:0.04806
[735]   validation_0-rmse:0.04792
[736]   validation_0-rmse:0.04781
[737]   validation_0-rmse:0.04756
[738]   validation_0-rmse:0.04746
[739]   validation_0-rmse:0.04732
[740]   validation_0-rmse:0.04715
[741]   validation_0-rmse:0.04708
[742]   validation_0-rmse:0.04691
[743]   validation_0-rmse:0.04678
[744]   validation_0-rmse:0.04673
[745]   validation_0-rmse:0.04669
[746]   validation_0-rmse:0.04646
[747]   validation_0-rmse:0.04633
[748]   validation_0-rmse:0.04623
[749]   validation_0-rmse:0.04615
[750]   validation_0-rmse:0.04613
[751]   validation_0-rmse:0.04600
[752]   validation_0-rmse:0.04585
[753]   validation_0-rmse:0.04583
[754]   validation_0-rmse:0.04576
[755]   validation_0-rmse:0.04562
[756]   validation_0-rmse:0.04559
[757]   validation_0-rmse:0.04544
[758]   validation_0-rmse:0.04540
[759]   validation_0-rmse:0.04533
[760]   validation_0-rmse:0.04511
[761]   validation_0-rmse:0.04493
[762]   validation_0-rmse:0.04483
[763]   validation_0-rmse:0.04467
[764]   validation_0-rmse:0.04460
[765]   validation_0-rmse:0.04458
[766]   validation_0-rmse:0.04441
[767]   validation_0-rmse:0.04436
[768]   validation_0-rmse:0.04423
[769]   validation_0-rmse:0.04414
[770]   validation_0-rmse:0.04411
[771]   validation_0-rmse:0.04398
[772]   validation_0-rmse:0.04373
[773]   validation_0-rmse:0.04355
[774]   validation_0-rmse:0.04334
[775]   validation_0-rmse:0.04326
[776]   validation_0-rmse:0.04322
[777]   validation_0-rmse:0.04311
[778]   validation_0-rmse:0.04300
[779]   validation_0-rmse:0.04286
[780]   validation_0-rmse:0.04275
[781]   validation_0-rmse:0.04264
[782]   validation_0-rmse:0.04253
[783]   validation_0-rmse:0.04231
[784]   validation_0-rmse:0.04224
[785]   validation_0-rmse:0.04213
[786]   validation_0-rmse:0.04204
[787]   validation_0-rmse:0.04199
[788]   validation_0-rmse:0.04190
[789]   validation_0-rmse:0.04178
[790]   validation_0-rmse:0.04162
[791]   validation_0-rmse:0.04148
[792]   validation_0-rmse:0.04139
[793]   validation_0-rmse:0.04127
[794]   validation_0-rmse:0.04120
[795]   validation_0-rmse:0.04104
[796]   validation_0-rmse:0.04090
[797]   validation_0-rmse:0.04074
[798]   validation_0-rmse:0.04069
[799]   validation_0-rmse:0.04058
[800]   validation_0-rmse:0.04049
[801]   validation_0-rmse:0.04044
[802]   validation_0-rmse:0.04043
[803]   validation_0-rmse:0.04034
[804]   validation_0-rmse:0.04019
[805]   validation_0-rmse:0.04007
[806]   validation_0-rmse:0.03986
[807]   validation_0-rmse:0.03976
[808]   validation_0-rmse:0.03966
[809]   validation_0-rmse:0.03963
[810]   validation_0-rmse:0.03944
[811]   validation_0-rmse:0.03940
[812]   validation_0-rmse:0.03931
[813]   validation_0-rmse:0.03925
[814]   validation_0-rmse:0.03922
[815]   validation_0-rmse:0.03917
[816]   validation_0-rmse:0.03904
[817]   validation_0-rmse:0.03892
[818]   validation_0-rmse:0.03886
[819]   validation_0-rmse:0.03872
[820]   validation_0-rmse:0.03865
[821]   validation_0-rmse:0.03856
[822]   validation_0-rmse:0.03845
[823]   validation_0-rmse:0.03843
[824]   validation_0-rmse:0.03836
[825]   validation_0-rmse:0.03830
[826]   validation_0-rmse:0.03825
[827]   validation_0-rmse:0.03817
[828]   validation_0-rmse:0.03812
[829]   validation_0-rmse:0.03808
[830]   validation_0-rmse:0.03806
[831]   validation_0-rmse:0.03805
[832]   validation_0-rmse:0.03803
[833]   validation_0-rmse:0.03785
[834]   validation_0-rmse:0.03775
[835]   validation_0-rmse:0.03764
[836]   validation_0-rmse:0.03754
[837]   validation_0-rmse:0.03745
[838]   validation_0-rmse:0.03734
[839]   validation_0-rmse:0.03720
[840]   validation_0-rmse:0.03715
[841]   validation_0-rmse:0.03706
[842]   validation_0-rmse:0.03692
[843]   validation_0-rmse:0.03680
[844]   validation_0-rmse:0.03660
[845]   validation_0-rmse:0.03651
[846]   validation_0-rmse:0.03634
[847]   validation_0-rmse:0.03631
[848]   validation_0-rmse:0.03622
[849]   validation_0-rmse:0.03608
[850]   validation_0-rmse:0.03598
[851]   validation_0-rmse:0.03588
[852]   validation_0-rmse:0.03582
[853]   validation_0-rmse:0.03569
[854]   validation_0-rmse:0.03562
[855]   validation_0-rmse:0.03552
[856]   validation_0-rmse:0.03536
[857]   validation_0-rmse:0.03533
[858]   validation_0-rmse:0.03518
[859]   validation_0-rmse:0.03515
[860]   validation_0-rmse:0.03505
[861]   validation_0-rmse:0.03496
[862]   validation_0-rmse:0.03486
[863]   validation_0-rmse:0.03483
[864]   validation_0-rmse:0.03477
[865]   validation_0-rmse:0.03474
[866]   validation_0-rmse:0.03465
[867]   validation_0-rmse:0.03454
[868]   validation_0-rmse:0.03447
[869]   validation_0-rmse:0.03440
[870]   validation_0-rmse:0.03438
[871]   validation_0-rmse:0.03437
[872]   validation_0-rmse:0.03433
[873]   validation_0-rmse:0.03424
[874]   validation_0-rmse:0.03418
[875]   validation_0-rmse:0.03408
[876]   validation_0-rmse:0.03402
[877]   validation_0-rmse:0.03394
[878]   validation_0-rmse:0.03391
[879]   validation_0-rmse:0.03385
[880]   validation_0-rmse:0.03373
[881]   validation_0-rmse:0.03366
[882]   validation_0-rmse:0.03360
[883]   validation_0-rmse:0.03346
[884]   validation_0-rmse:0.03340
[885]   validation_0-rmse:0.03328
[886]   validation_0-rmse:0.03322
[887]   validation_0-rmse:0.03311
[888]   validation_0-rmse:0.03303
[889]   validation_0-rmse:0.03296
[890]   validation_0-rmse:0.03291
[891]   validation_0-rmse:0.03278
[892]   validation_0-rmse:0.03272
[893]   validation_0-rmse:0.03265
[894]   validation_0-rmse:0.03260
[895]   validation_0-rmse:0.03256
[896]   validation_0-rmse:0.03252
[897]   validation_0-rmse:0.03247
[898]   validation_0-rmse:0.03244
[899]   validation_0-rmse:0.03237
[900]   validation_0-rmse:0.03234
[901]   validation_0-rmse:0.03231
[902]   validation_0-rmse:0.03219
[903]   validation_0-rmse:0.03213
[904]   validation_0-rmse:0.03203
[905]   validation_0-rmse:0.03200
[906]   validation_0-rmse:0.03195
[907]   validation_0-rmse:0.03185
[908]   validation_0-rmse:0.03177
[909]   validation_0-rmse:0.03171
[910]   validation_0-rmse:0.03163
[911]   validation_0-rmse:0.03148
[912]   validation_0-rmse:0.03140
[913]   validation_0-rmse:0.03130
[914]   validation_0-rmse:0.03123
[915]   validation_0-rmse:0.03113
[916]   validation_0-rmse:0.03104
[917]   validation_0-rmse:0.03092
[918]   validation_0-rmse:0.03089
[919]   validation_0-rmse:0.03082
[920]   validation_0-rmse:0.03077
[921]   validation_0-rmse:0.03065
[922]   validation_0-rmse:0.03053
[923]   validation_0-rmse:0.03047
[924]   validation_0-rmse:0.03041
[925]   validation_0-rmse:0.03033
[926]   validation_0-rmse:0.03027
[927]   validation_0-rmse:0.03022
[928]   validation_0-rmse:0.03017
[929]   validation_0-rmse:0.03011
[930]   validation_0-rmse:0.03001
[931]   validation_0-rmse:0.02995
[932]   validation_0-rmse:0.02990
[933]   validation_0-rmse:0.02977
[934]   validation_0-rmse:0.02968
[935]   validation_0-rmse:0.02955
[936]   validation_0-rmse:0.02950
[937]   validation_0-rmse:0.02940
[938]   validation_0-rmse:0.02930
[939]   validation_0-rmse:0.02927
[940]   validation_0-rmse:0.02921
[941]   validation_0-rmse:0.02908
[942]   validation_0-rmse:0.02897
[943]   validation_0-rmse:0.02886
[944]   validation_0-rmse:0.02879
[945]   validation_0-rmse:0.02871
[946]   validation_0-rmse:0.02862
[947]   validation_0-rmse:0.02852
[948]   validation_0-rmse:0.02847
[949]   validation_0-rmse:0.02843
[950]   validation_0-rmse:0.02832
[951]   validation_0-rmse:0.02819
[952]   validation_0-rmse:0.02810
[953]   validation_0-rmse:0.02801
[954]   validation_0-rmse:0.02794
[955]   validation_0-rmse:0.02787
[956]   validation_0-rmse:0.02784
[957]   validation_0-rmse:0.02780
[958]   validation_0-rmse:0.02778
[959]   validation_0-rmse:0.02767
[960]   validation_0-rmse:0.02765
[961]   validation_0-rmse:0.02761
[962]   validation_0-rmse:0.02754
[963]   validation_0-rmse:0.02745
[964]   validation_0-rmse:0.02735
[965]   validation_0-rmse:0.02727
[966]   validation_0-rmse:0.02720
[967]   validation_0-rmse:0.02715
[968]   validation_0-rmse:0.02707
[969]   validation_0-rmse:0.02706
[970]   validation_0-rmse:0.02699
[971]   validation_0-rmse:0.02691
[972]   validation_0-rmse:0.02686
[973]   validation_0-rmse:0.02684
[974]   validation_0-rmse:0.02678
[975]   validation_0-rmse:0.02674
[976]   validation_0-rmse:0.02666
[977]   validation_0-rmse:0.02661
[978]   validation_0-rmse:0.02652
[979]   validation_0-rmse:0.02647
[980]   validation_0-rmse:0.02641
[981]   validation_0-rmse:0.02633
[982]   validation_0-rmse:0.02624
[983]   validation_0-rmse:0.02620
[984]   validation_0-rmse:0.02617
[985]   validation_0-rmse:0.02605
[986]   validation_0-rmse:0.02599
[987]   validation_0-rmse:0.02590
[988]   validation_0-rmse:0.02589
[989]   validation_0-rmse:0.02582
[990]   validation_0-rmse:0.02574
[991]   validation_0-rmse:0.02568
[992]   validation_0-rmse:0.02558
[993]   validation_0-rmse:0.02556
[994]   validation_0-rmse:0.02549
[995]   validation_0-rmse:0.02537
[996]   validation_0-rmse:0.02533
[997]   validation_0-rmse:0.02524
[998]   validation_0-rmse:0.02523
[999]   validation_0-rmse:0.02518

XGBRegressor(base_score=None, booster=None, callbacks=None,
             colsample_bylevel=None, colsample_bynode=None,
             colsample_bytree=None, device=None, early_stopping_rounds=50,
             enable_categorical=True, eval_metric=None, feature_types=None,
             feature_weights=None, gamma=None, grow_policy=None,
             importance_type=None, interaction_constraints=None,
             learning_rate=None, max_bin=None, max_cat_threshold=None,
             max_cat_to_onehot=None, max_delta_step=None, max_depth=None,
             max_leaves=None, min_child_weight=None, missing=nan,
             monotone_constraints=None, multi_strategy=None, n_estimators=1000,
             n_jobs=None, num_parallel_tree=None, ...)

In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.

XGBRegressor

?Documentation for XGBRegressoriFitted

Parameters

	base_score base_score: typing.Union[float, typing.List[float], NoneType] The initial prediction score of all instances, global bias.	None
	booster	None
	callbacks callbacks: typing.Optional[typing.List[xgboost.callback.TrainingCallback]] List of callback functions that are applied at end of each iteration. It is possible to use predefined callbacks by using :ref:`Callback API <callback_api>`. .. note:: States in callback are not preserved during training, which means callback objects can not be reused for multiple training sessions without reinitialization or deepcopy. .. code-block:: python for params in parameters_grid: # be sure to (re)initialize the callbacks before each run callbacks = [xgb.callback.LearningRateScheduler(custom_rates)] reg = xgboost.XGBRegressor(**params, callbacks=callbacks) reg.fit(X, y)	None
	colsample_bylevel colsample_bylevel: typing.Optional[float] Subsample ratio of columns for each level.	None
	colsample_bynode colsample_bynode: typing.Optional[float] Subsample ratio of columns for each split.	None
	colsample_bytree colsample_bytree: typing.Optional[float] Subsample ratio of columns when constructing each tree.	None
	device device: typing.Optional[str] .. versionadded:: 2.0.0 Device ordinal, available options are `cpu`, `cuda`, and `gpu`.	None
	early_stopping_rounds early_stopping_rounds: typing.Optional[int] .. versionadded:: 1.6.0 - Activates early stopping. Validation metric needs to improve at least once in every early_stopping_rounds round(s) to continue training. Requires at least one item in eval_set in :py:meth:`fit`. - If early stopping occurs, the model will have two additional attributes: :py:attr:`best_score` and :py:attr:`best_iteration`. These are used by the :py:meth:`predict` and :py:meth:`apply` methods to determine the optimal number of trees during inference. If users want to access the full model (including trees built after early stopping), they can specify the `iteration_range` in these inference methods. In addition, other utilities like model plotting can also use the entire model. - If you prefer to discard the trees after `best_iteration`, consider using the callback function :py:class:`xgboost.callback.EarlyStopping`. - If there's more than one item in eval_set, the last entry will be used for early stopping. If there's more than one metric in eval_metric, the last metric will be used for early stopping.	50
	enable_categorical enable_categorical: bool See the same parameter of :py:class:`DMatrix` for details.	True
	eval_metric eval_metric: typing.Union[str, typing.List[typing.Union[str, typing.Callable]], typing.Callable, NoneType] .. versionadded:: 1.6.0 Metric used for monitoring the training result and early stopping. It can be a string or list of strings as names of predefined metric in XGBoost (See :doc:`/parameter`), one of the metrics in :py:mod:`sklearn.metrics`, or any other user defined metric that looks like `sklearn.metrics`. If custom objective is also provided, then custom metric should implement the corresponding reverse link function. Unlike the `scoring` parameter commonly used in scikit-learn, when a callable object is provided, it's assumed to be a cost function and by default XGBoost will minimize the result during early stopping. For advanced usage on Early stopping like directly choosing to maximize instead of minimize, see :py:obj:`xgboost.callback.EarlyStopping`. See :doc:`/tutorials/custom_metric_obj` and :ref:`custom-obj-metric` for more information. .. code-block:: python from sklearn.datasets import load_diabetes from sklearn.metrics import mean_absolute_error X, y = load_diabetes(return_X_y=True) reg = xgb.XGBRegressor( tree_method="hist", eval_metric=mean_absolute_error, ) reg.fit(X, y, eval_set=[(X, y)])	None
	feature_types feature_types: typing.Optional[typing.Sequence[str]] .. versionadded:: 1.7.0 Used for specifying feature types without constructing a dataframe. See the :py:class:`DMatrix` for details.	None
	feature_weights feature_weights: Optional[ArrayLike] Weight for each feature, defines the probability of each feature being selected when colsample is being used. All values must be greater than 0, otherwise a `ValueError` is thrown.	None
	gamma gamma: typing.Optional[float] (min_split_loss) Minimum loss reduction required to make a further partition on a leaf node of the tree.	None
	grow_policy grow_policy: typing.Optional[str] Tree growing policy. - depthwise: Favors splitting at nodes closest to the node, - lossguide: Favors splitting at nodes with highest loss change.	None
	importance_type	None
	interaction_constraints interaction_constraints: typing.Union[str, typing.List[typing.Tuple[str]], NoneType] Constraints for interaction representing permitted interactions. The constraints must be specified in the form of a nested list, e.g. ``[[0, 1], [2, 3, 4]]``, where each inner list is a group of indices of features that are allowed to interact with each other. See :doc:`tutorial </tutorials/feature_interaction_constraint>` for more information	None
	learning_rate learning_rate: typing.Optional[float] Boosting learning rate (xgb's "eta")	None
	max_bin max_bin: typing.Optional[int] If using histogram-based algorithm, maximum number of bins per feature	None
	max_cat_threshold max_cat_threshold: typing.Optional[int] .. versionadded:: 1.7.0 .. note:: This parameter is experimental Maximum number of categories considered for each split. Used only by partition-based splits for preventing over-fitting. Also, `enable_categorical` needs to be set to have categorical feature support. See :doc:`Categorical Data </tutorials/categorical>` and :ref:`cat-param` for details.	None
	max_cat_to_onehot max_cat_to_onehot: Optional[int] .. versionadded:: 1.6.0 .. note:: This parameter is experimental A threshold for deciding whether XGBoost should use one-hot encoding based split for categorical data. When number of categories is lesser than the threshold then one-hot encoding is chosen, otherwise the categories will be partitioned into children nodes. Also, `enable_categorical` needs to be set to have categorical feature support. See :doc:`Categorical Data </tutorials/categorical>` and :ref:`cat-param` for details.	None
	max_delta_step max_delta_step: typing.Optional[float] Maximum delta step we allow each tree's weight estimation to be.	None
	max_depth max_depth: typing.Optional[int] Maximum tree depth for base learners.	None
	max_leaves max_leaves: typing.Optional[int] Maximum number of leaves; 0 indicates no limit.	None
	min_child_weight min_child_weight: typing.Optional[float] Minimum sum of instance weight(hessian) needed in a child.	None
	missing missing: float Value in the data which needs to be present as a missing value. Default to :py:data:`numpy.nan`.	nan
	monotone_constraints monotone_constraints: typing.Union[typing.Dict[str, int], str, NoneType] Constraint of variable monotonicity. See :doc:`tutorial </tutorials/monotonic>` for more information.	None
	multi_strategy multi_strategy: typing.Optional[str] .. versionadded:: 2.0.0 .. note:: This parameter is working-in-progress. The strategy used for training multi-target models, including multi-target regression and multi-class classification. See :doc:`/tutorials/multioutput` for more information. - ``one_output_per_tree``: One model for each target. - ``multi_output_tree``: Use multi-target trees.	None
	n_estimators n_estimators: typing.Optional[int] Number of gradient boosted trees. Equivalent to number of boosting rounds.	1000
	n_jobs n_jobs: typing.Optional[int] Number of parallel threads used to run xgboost. When used with other Scikit-Learn algorithms like grid search, you may choose which algorithm to parallelize and balance the threads. Creating thread contention will significantly slow down both algorithms.	None
	num_parallel_tree	None
	random_state random_state: typing.Union[numpy.random.mtrand.RandomState, numpy.random._generator.Generator, int, NoneType] Random number seed. .. note:: Using gblinear booster with shotgun updater is nondeterministic as it uses Hogwild algorithm.	None
	reg_alpha reg_alpha: typing.Optional[float] L1 regularization term on weights (xgb's alpha).	None
	reg_lambda reg_lambda: typing.Optional[float] L2 regularization term on weights (xgb's lambda).	None
	sampling_method sampling_method: typing.Optional[str] Sampling method. Used only by the GPU version of ``hist`` tree method. - ``uniform``: Select random training instances uniformly. - ``gradient_based``: Select random training instances with higher probability when the gradient and hessian are larger. (cf. CatBoost)	None
	scale_pos_weight scale_pos_weight: typing.Optional[float] Balancing of positive and negative weights.	None
	subsample subsample: typing.Optional[float] Subsample ratio of the training instance.	None
	tree_method tree_method: typing.Optional[str] Specify which tree method to use. Default to auto. If this parameter is set to default, XGBoost will choose the most conservative option available. It's recommended to study this option from the parameters document :doc:`tree method </treemethod>`	None
	validate_parameters validate_parameters: typing.Optional[bool] Give warnings for unknown parameter.	None
	verbosity verbosity: typing.Optional[int] The degree of verbosity. Valid values are 0 (silent) - 3 (debug).	None
	objective objective: typing.Union[str, xgboost.objective.Objective, xgboost.sklearn._SklObjWProto, typing.Callable[[typing.Any, typing.Any], typing.Tuple[numpy.ndarray, numpy.ndarray]], NoneType] Specify the learning task and the corresponding learning objective or a custom objective to be used. For custom objective, see :doc:`/tutorials/custom_metric_obj` and :ref:`custom-obj-metric` for more information, along with the end note for function signatures.	'reg:squarederror'

Fitted attributes

Name	Type	Value
evals_result_	dict	{'va..._0': OrderedDict({...94182151791]})}
feature_importances_	ndarray[float32](8,)	[0.44,0.07,0.04,...,0.14,0.12,0.14]
intercept_	ndarray[float32](1,)	[2.07]
n_features_in_	int	8

Pitfall 2: High Learning Rate with Few Trees

Code

# ❌ BAD: Fast learning, few trees → underfitting
model = xgb.XGBRegressor(n_estimators=10, learning_rate=0.3)

# ✅ GOOD: Slow learning, many trees → better generalization
model = xgb.XGBRegressor(n_estimators=500, learning_rate=0.05)

Pitfall 3: Ignoring Feature Scaling

Good news: Gradient boosting doesn’t require feature scaling!

# No need to scale for tree-based methods
from sklearn.preprocessing import StandardScaler

# ❌ Unnecessary for XGBoost/LightGBM/CatBoost
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# ✅ Use raw features
model.fit(X, y)  # Works great!

Pitfall 4: Not Encoding Categorical Features (except CatBoost)

Code

import pandas as pd

# Data with categorical feature
df = pd.DataFrame({
    'size': [1200, 1500, 1800],
    'neighborhood': ['A', 'B', 'C'],
    'price': [200, 250, 300]
})

# ❌ BAD: XGBoost/LightGBM can't handle strings directly
model = xgb.XGBRegressor()
# model.fit(df[['size', 'neighborhood']], df['price'])  # ERROR!

# ✅ GOOD: Encode first (for XGBoost/LightGBM)
df['neighborhood_encoded'] = df['neighborhood'].astype('category').cat.codes
model.fit(df[['size', 'neighborhood_encoded']], df['price'])

# ✅ BEST: Use CatBoost
model = CatBoostRegressor(cat_features=['neighborhood'])
model.fit(df[['size', 'neighborhood']], df['price'])  # Handles strings!

Learning rate set to 0.016352
0:  learn: 40.6747393   total: 906us    remaining: 906ms
1:  learn: 40.6747390   total: 949us    remaining: 474ms
2:  learn: 40.5084607   total: 1.42ms   remaining: 471ms
3:  learn: 40.5084603   total: 1.46ms   remaining: 363ms
4:  learn: 40.3599814   total: 1.54ms   remaining: 306ms
5:  learn: 40.2124950   total: 1.59ms   remaining: 264ms
6:  learn: 40.0633224   total: 1.64ms   remaining: 233ms
7:  learn: 39.9151440   total: 1.7ms    remaining: 211ms
8:  learn: 39.7688441   total: 1.74ms   remaining: 192ms
9:  learn: 39.7688428   total: 1.77ms   remaining: 175ms
10: learn: 39.7688416   total: 1.78ms   remaining: 160ms
11: learn: 39.6235172   total: 2.4ms    remaining: 197ms
12: learn: 39.4791652   total: 2.45ms   remaining: 186ms
13: learn: 39.3322744   total: 2.53ms   remaining: 178ms
14: learn: 39.3322717   total: 2.56ms   remaining: 168ms
15: learn: 39.1714813   total: 2.65ms   remaining: 163ms
16: learn: 39.1714787   total: 2.68ms   remaining: 155ms
17: learn: 39.0287736   total: 2.71ms   remaining: 148ms
18: learn: 38.8835586   total: 2.77ms   remaining: 143ms
19: learn: 38.7393105   total: 2.8ms    remaining: 137ms
20: learn: 38.7393095   total: 2.83ms   remaining: 132ms
21: learn: 38.5977527   total: 2.87ms   remaining: 128ms
22: learn: 38.5977502   total: 2.91ms   remaining: 124ms
23: learn: 38.5977478   total: 2.93ms   remaining: 119ms
24: learn: 38.5977454   total: 3.07ms   remaining: 120ms
25: learn: 38.4545614   total: 3.18ms   remaining: 119ms
26: learn: 38.3123315   total: 3.22ms   remaining: 116ms
27: learn: 38.3123313   total: 3.23ms   remaining: 112ms
28: learn: 38.1719078   total: 3.26ms   remaining: 109ms
29: learn: 38.0307220   total: 3.3ms    remaining: 107ms
30: learn: 38.0307218   total: 3.31ms   remaining: 103ms
31: learn: 37.8904779   total: 3.34ms   remaining: 101ms
32: learn: 37.7355816   total: 3.4ms    remaining: 99.8ms
33: learn: 37.7355816   total: 3.42ms   remaining: 97.3ms
34: learn: 37.7355816   total: 3.44ms   remaining: 95ms
35: learn: 37.5813185   total: 3.52ms   remaining: 94.3ms
36: learn: 37.4276861   total: 3.58ms   remaining: 93.2ms
37: learn: 37.2900791   total: 3.63ms   remaining: 91.9ms
38: learn: 37.1525784   total: 3.67ms   remaining: 90.6ms
39: learn: 37.0159831   total: 3.72ms   remaining: 89.3ms
40: learn: 36.8803000   total: 3.78ms   remaining: 88.4ms
41: learn: 36.7455237   total: 3.82ms   remaining: 87.1ms
42: learn: 36.6116489   total: 3.85ms   remaining: 85.7ms
43: learn: 36.6116433   total: 3.87ms   remaining: 84.1ms
44: learn: 36.4786620   total: 3.91ms   remaining: 83ms
45: learn: 36.3295373   total: 3.98ms   remaining: 82.6ms
46: learn: 36.1939725   total: 4.02ms   remaining: 81.5ms
47: learn: 36.0460115   total: 4.08ms   remaining: 81ms
48: learn: 36.0460062   total: 4.11ms   remaining: 79.7ms
49: learn: 36.0460011   total: 4.13ms   remaining: 78.4ms
50: learn: 35.9150736   total: 4.16ms   remaining: 77.4ms
51: learn: 35.7850232   total: 4.2ms    remaining: 76.6ms
52: learn: 35.6558448   total: 4.23ms   remaining: 75.6ms
53: learn: 35.5220062   total: 4.27ms   remaining: 74.8ms
54: learn: 35.3890570   total: 4.31ms   remaining: 74.1ms
55: learn: 35.2609227   total: 4.35ms   remaining: 73.4ms
56: learn: 35.1336476   total: 4.39ms   remaining: 72.6ms
57: learn: 35.0072266   total: 4.42ms   remaining: 71.9ms
58: learn: 34.8816546   total: 4.47ms   remaining: 71.4ms
59: learn: 34.7569266   total: 4.52ms   remaining: 70.8ms
60: learn: 34.6330376   total: 4.57ms   remaining: 70.4ms
61: learn: 34.5014976   total: 4.63ms   remaining: 70.1ms
62: learn: 34.3604554   total: 4.69ms   remaining: 69.7ms
63: learn: 34.2303207   total: 4.71ms   remaining: 69ms
64: learn: 34.1079410   total: 4.76ms   remaining: 68.4ms
65: learn: 33.9787542   total: 4.8ms    remaining: 68ms
66: learn: 33.8572831   total: 4.83ms   remaining: 67.3ms
67: learn: 33.7290372   total: 4.92ms   remaining: 67.4ms
68: learn: 33.6016412   total: 5.1ms    remaining: 68.8ms
69: learn: 33.4750899   total: 5.15ms   remaining: 68.4ms
70: learn: 33.3493783   total: 5.19ms   remaining: 68ms
71: learn: 33.2245016   total: 5.25ms   remaining: 67.6ms
72: learn: 33.1004547   total: 5.29ms   remaining: 67.2ms
73: learn: 32.9772327   total: 5.32ms   remaining: 66.5ms
74: learn: 32.8424218   total: 5.4ms    remaining: 66.6ms
75: learn: 32.7081620   total: 6.18ms   remaining: 75.2ms
76: learn: 32.5890722   total: 6.24ms   remaining: 74.7ms
77: learn: 32.4677504   total: 6.29ms   remaining: 74.3ms
78: learn: 32.3495396   total: 6.33ms   remaining: 73.8ms
79: learn: 32.2291060   total: 6.37ms   remaining: 73.3ms
80: learn: 32.1094739   total: 6.42ms   remaining: 72.8ms
81: learn: 31.9782103   total: 6.48ms   remaining: 72.5ms
82: learn: 31.8598607   total: 6.52ms   remaining: 72.1ms
83: learn: 31.7431615   total: 6.56ms   remaining: 71.6ms
84: learn: 31.6272431   total: 6.59ms   remaining: 71ms
85: learn: 31.5098404   total: 6.62ms   remaining: 70.4ms
86: learn: 31.3810282   total: 6.69ms   remaining: 70.2ms
87: learn: 31.2664350   total: 6.74ms   remaining: 69.9ms
88: learn: 31.1526090   total: 6.79ms   remaining: 69.5ms
89: learn: 31.0395458   total: 6.83ms   remaining: 69.1ms
90: learn: 30.9272409   total: 7.03ms   remaining: 70.2ms
91: learn: 30.8008103   total: 7.17ms   remaining: 70.8ms
92: learn: 30.6854504   total: 7.2ms    remaining: 70.3ms
93: learn: 30.5744319   total: 7.24ms   remaining: 69.8ms
94: learn: 30.4599149   total: 7.26ms   remaining: 69.2ms
95: learn: 30.3353947   total: 7.34ms   remaining: 69.1ms
96: learn: 30.2256480   total: 7.37ms   remaining: 68.6ms
97: learn: 30.1124323   total: 7.39ms   remaining: 68ms
98: learn: 30.0034975   total: 7.45ms   remaining: 67.8ms
99: learn: 29.8911089   total: 7.5ms    remaining: 67.5ms
100:    learn: 29.7829801   total: 7.53ms   remaining: 67ms
101:    learn: 29.6755771   total: 7.57ms   remaining: 66.6ms
102:    learn: 29.5640837   total: 7.6ms    remaining: 66.2ms
103:    learn: 29.4574763   total: 7.63ms   remaining: 65.8ms
104:    learn: 29.3370541   total: 7.7ms    remaining: 65.6ms
105:    learn: 29.2268267   total: 7.73ms   remaining: 65.2ms
106:    learn: 29.1173308   total: 7.77ms   remaining: 64.8ms
107:    learn: 29.0120185   total: 7.82ms   remaining: 64.6ms
108:    learn: 28.8934174   total: 7.88ms   remaining: 64.4ms
109:    learn: 28.7851656   total: 7.94ms   remaining: 64.3ms
110:    learn: 28.6776326   total: 7.98ms   remaining: 63.9ms
111:    learn: 28.5603984   total: 8.04ms   remaining: 63.8ms
112:    learn: 28.4567898   total: 8.1ms    remaining: 63.5ms
113:    learn: 28.3538763   total: 8.15ms   remaining: 63.3ms
114:    learn: 28.2516540   total: 8.19ms   remaining: 63ms
115:    learn: 28.1501185   total: 8.23ms   remaining: 62.7ms
116:    learn: 28.0440199   total: 8.26ms   remaining: 62.3ms
117:    learn: 27.9432376   total: 8.29ms   remaining: 62ms
118:    learn: 27.8431332   total: 9.02ms   remaining: 66.8ms
119:    learn: 27.7378777   total: 9.14ms   remaining: 67ms
120:    learn: 27.6333193   total: 9.16ms   remaining: 66.6ms
121:    learn: 27.5203543   total: 9.22ms   remaining: 66.4ms
122:    learn: 27.4169136   total: 9.25ms   remaining: 66ms
123:    learn: 27.3141591   total: 9.28ms   remaining: 65.5ms
124:    learn: 27.2120866   total: 9.31ms   remaining: 65.1ms
125:    learn: 27.1008436   total: 9.4ms    remaining: 65.2ms
126:    learn: 26.9998637   total: 9.44ms   remaining: 64.9ms
127:    learn: 26.9016535   total: 9.47ms   remaining: 64.5ms
128:    learn: 26.8014121   total: 9.51ms   remaining: 64.2ms
129:    learn: 26.6918479   total: 9.55ms   remaining: 63.9ms
130:    learn: 26.5926799   total: 9.59ms   remaining: 63.6ms
131:    learn: 26.4956603   total: 9.64ms   remaining: 63.4ms
132:    learn: 26.3992911   total: 9.68ms   remaining: 63.1ms
133:    learn: 26.3035682   total: 9.72ms   remaining: 62.8ms
134:    learn: 26.2052577   total: 9.76ms   remaining: 62.5ms
135:    learn: 26.1076002   total: 9.79ms   remaining: 62.2ms
136:    learn: 26.0105918   total: 9.83ms   remaining: 61.9ms
137:    learn: 25.9157072   total: 9.87ms   remaining: 61.7ms
138:    learn: 25.8097638   total: 9.94ms   remaining: 61.6ms
139:    learn: 25.7138590   total: 9.97ms   remaining: 61.3ms
140:    learn: 25.6087408   total: 10ms remaining: 61.1ms
141:    learn: 25.5153252   total: 10.1ms   remaining: 60.9ms
142:    learn: 25.4205115   total: 10.1ms   remaining: 60.6ms
143:    learn: 25.3263284   total: 10.2ms   remaining: 60.4ms
144:    learn: 25.2227944   total: 10.2ms   remaining: 60.2ms
145:    learn: 25.1305106   total: 10.3ms   remaining: 60ms
146:    learn: 25.0277771   total: 10.3ms   remaining: 60ms
147:    learn: 24.9364862   total: 10.4ms   remaining: 59.8ms
148:    learn: 24.8458074   total: 10.4ms   remaining: 59.5ms
149:    learn: 24.7532013   total: 10.5ms   remaining: 59.4ms
150:    learn: 24.6631927   total: 10.5ms   remaining: 59.1ms
151:    learn: 24.5623696   total: 10.6ms   remaining: 58.9ms
152:    learn: 24.4619586   total: 10.6ms   remaining: 58.9ms
153:    learn: 24.3732835   total: 10.7ms   remaining: 58.6ms
154:    learn: 24.2852038   total: 10.7ms   remaining: 58.3ms
155:    learn: 24.1941464   total: 10.7ms   remaining: 58.1ms
156:    learn: 24.1067191   total: 10.8ms   remaining: 57.8ms
157:    learn: 24.0081708   total: 11ms remaining: 58.5ms
158:    learn: 23.9181474   total: 11.1ms   remaining: 58.7ms
159:    learn: 23.8317225   total: 11.2ms   remaining: 58.5ms
160:    learn: 23.7342984   total: 11.2ms   remaining: 58.4ms
161:    learn: 23.6372726   total: 11.3ms   remaining: 58.3ms
162:    learn: 23.5521289   total: 11.3ms   remaining: 58.1ms
163:    learn: 23.4675580   total: 11.3ms   remaining: 57.8ms
164:    learn: 23.3716226   total: 11.4ms   remaining: 57.8ms
165:    learn: 23.2760794   total: 11.5ms   remaining: 57.6ms
166:    learn: 23.1882819   total: 11.5ms   remaining: 57.5ms
167:    learn: 23.0934882   total: 11.6ms   remaining: 57.4ms
168:    learn: 23.0105704   total: 11.6ms   remaining: 57.2ms
169:    learn: 22.9237686   total: 11.7ms   remaining: 57.1ms
170:    learn: 22.8414664   total: 11.7ms   remaining: 56.9ms
171:    learn: 22.7552967   total: 11.8ms   remaining: 56.7ms
172:    learn: 22.6696976   total: 11.8ms   remaining: 56.5ms
173:    learn: 22.5770239   total: 11.9ms   remaining: 56.4ms
174:    learn: 22.4847290   total: 12ms remaining: 56.4ms
175:    learn: 22.3928114   total: 12ms remaining: 56.2ms
176:    learn: 22.3012696   total: 12ms remaining: 56ms
177:    learn: 22.2101020   total: 12.1ms   remaining: 56ms
178:    learn: 22.1193071   total: 12.2ms   remaining: 55.9ms
179:    learn: 22.0288834   total: 12.3ms   remaining: 55.8ms
180:    learn: 21.9388293   total: 12.3ms   remaining: 55.6ms
181:    learn: 21.8565389   total: 12.3ms   remaining: 55.4ms
182:    learn: 21.7671894   total: 13.1ms   remaining: 58.6ms
183:    learn: 21.6782051   total: 13.3ms   remaining: 58.8ms
184:    learn: 21.5895846   total: 13.3ms   remaining: 58.7ms
185:    learn: 21.5088384   total: 13.4ms   remaining: 58.5ms
186:    learn: 21.4209103   total: 13.4ms   remaining: 58.3ms
187:    learn: 21.3333416   total: 13.5ms   remaining: 58.1ms
188:    learn: 21.2461309   total: 13.5ms   remaining: 58ms
189:    learn: 21.1592768   total: 13.6ms   remaining: 57.9ms
190:    learn: 21.0727776   total: 13.7ms   remaining: 57.8ms
191:    learn: 20.9866321   total: 13.7ms   remaining: 57.7ms
192:    learn: 20.9083692   total: 13.8ms   remaining: 57.5ms
193:    learn: 20.8228958   total: 13.8ms   remaining: 57.5ms
194:    learn: 20.7377718   total: 13.9ms   remaining: 57.3ms
195:    learn: 20.6529958   total: 13.9ms   remaining: 57.2ms
196:    learn: 20.5685664   total: 14ms remaining: 57.1ms
197:    learn: 20.4844821   total: 14.1ms   remaining: 56.9ms
198:    learn: 20.4007415   total: 14.1ms   remaining: 56.8ms
199:    learn: 20.3248859   total: 14.1ms   remaining: 56.6ms
200:    learn: 20.2417978   total: 14.2ms   remaining: 56.5ms
201:    learn: 20.1680108   total: 14.3ms   remaining: 56.3ms
202:    learn: 20.0855639   total: 14.3ms   remaining: 56.2ms
203:    learn: 20.0034542   total: 14.4ms   remaining: 56.1ms
204:    learn: 19.9216800   total: 14.4ms   remaining: 56ms
205:    learn: 19.8402402   total: 14.5ms   remaining: 55.9ms
206:    learn: 19.7591333   total: 14.6ms   remaining: 55.8ms
207:    learn: 19.6856607   total: 15.1ms   remaining: 57.6ms
208:    learn: 19.6051857   total: 15.3ms   remaining: 57.7ms
209:    learn: 19.5250397   total: 15.3ms   remaining: 57.6ms
210:    learn: 19.4452214   total: 15.4ms   remaining: 57.5ms
211:    learn: 19.3657293   total: 15.4ms   remaining: 57.4ms
212:    learn: 19.2865622   total: 15.5ms   remaining: 57.2ms
213:    learn: 19.2077187   total: 15.5ms   remaining: 57.1ms
214:    learn: 19.1291976   total: 15.6ms   remaining: 57ms
215:    learn: 19.0509974   total: 15.7ms   remaining: 56.8ms
216:    learn: 18.9731170   total: 15.7ms   remaining: 56.6ms
217:    learn: 18.8955549   total: 15.8ms   remaining: 56.5ms
218:    learn: 18.8183098   total: 15.8ms   remaining: 56.4ms
219:    learn: 18.7485412   total: 15.8ms   remaining: 56.2ms
220:    learn: 18.6718972   total: 15.9ms   remaining: 56ms
221:    learn: 18.5955665   total: 15.9ms   remaining: 55.9ms
222:    learn: 18.5195478   total: 16ms remaining: 55.7ms
223:    learn: 18.4438399   total: 16ms remaining: 55.6ms
224:    learn: 18.3684415   total: 16.1ms   remaining: 55.4ms
225:    learn: 18.2933513   total: 16.2ms   remaining: 55.3ms
226:    learn: 18.2257294   total: 16.2ms   remaining: 55.1ms
227:    learn: 18.1512226   total: 16.2ms   remaining: 55ms
228:    learn: 18.0770204   total: 16.3ms   remaining: 54.9ms
229:    learn: 18.0031216   total: 16.3ms   remaining: 54.7ms
230:    learn: 17.9295248   total: 16.4ms   remaining: 54.6ms
231:    learn: 17.8562289   total: 16.4ms   remaining: 54.4ms
232:    learn: 17.7832327   total: 16.5ms   remaining: 54.3ms
233:    learn: 17.7105348   total: 16.6ms   remaining: 54.2ms
234:    learn: 17.6381342   total: 16.6ms   remaining: 54.1ms
235:    learn: 17.5660295   total: 16.7ms   remaining: 53.9ms
236:    learn: 17.4942195   total: 16.7ms   remaining: 53.8ms
237:    learn: 17.4227032   total: 16.8ms   remaining: 53.7ms
238:    learn: 17.3514792   total: 16.8ms   remaining: 53.6ms
239:    learn: 17.2805463   total: 16.9ms   remaining: 53.4ms
240:    learn: 17.2099034   total: 16.9ms   remaining: 53.2ms
241:    learn: 17.1395494   total: 17.4ms   remaining: 54.5ms
242:    learn: 17.0694829   total: 17.5ms   remaining: 54.6ms
243:    learn: 16.9997028   total: 17.6ms   remaining: 54.4ms
244:    learn: 16.9302080   total: 17.6ms   remaining: 54.3ms
245:    learn: 16.8609974   total: 17.7ms   remaining: 54.2ms
246:    learn: 16.7920696   total: 17.7ms   remaining: 54.1ms
247:    learn: 16.7234236   total: 17.8ms   remaining: 54ms
248:    learn: 16.6550583   total: 17.9ms   remaining: 53.9ms
249:    learn: 16.5869724   total: 17.9ms   remaining: 53.8ms
250:    learn: 16.5191649   total: 18ms remaining: 53.7ms
251:    learn: 16.4516345   total: 18ms remaining: 53.5ms
252:    learn: 16.3843802   total: 18.1ms   remaining: 53.4ms
253:    learn: 16.3174009   total: 18.1ms   remaining: 53.3ms
254:    learn: 16.2506953   total: 18.2ms   remaining: 53.1ms
255:    learn: 16.1842625   total: 18.2ms   remaining: 53ms
256:    learn: 16.1181012   total: 18.3ms   remaining: 52.9ms
257:    learn: 16.0522104   total: 18.4ms   remaining: 52.8ms
258:    learn: 15.9865890   total: 18.4ms   remaining: 52.7ms
259:    learn: 15.9212358   total: 18.5ms   remaining: 52.6ms
260:    learn: 15.8561498   total: 18.5ms   remaining: 52.5ms
261:    learn: 15.7913299   total: 18.6ms   remaining: 52.3ms
262:    learn: 15.7267749   total: 18.6ms   remaining: 52.3ms
263:    learn: 15.6624839   total: 18.7ms   remaining: 52.1ms
264:    learn: 15.5984556   total: 18.7ms   remaining: 52ms
265:    learn: 15.5346892   total: 18.8ms   remaining: 51.9ms
266:    learn: 15.4711833   total: 18.9ms   remaining: 51.8ms
267:    learn: 15.4079372   total: 19ms remaining: 51.9ms
268:    learn: 15.3449495   total: 19ms remaining: 51.8ms
269:    learn: 15.2822194   total: 19.1ms   remaining: 51.6ms
270:    learn: 15.2197456   total: 19.2ms   remaining: 51.6ms
271:    learn: 15.1575273   total: 19.2ms   remaining: 51.4ms
272:    learn: 15.0955634   total: 19.3ms   remaining: 51.4ms
273:    learn: 15.0338527   total: 19.4ms   remaining: 51.3ms
274:    learn: 14.9723943   total: 19.4ms   remaining: 51.2ms
275:    learn: 14.9111872   total: 19.5ms   remaining: 51.1ms
276:    learn: 14.8502302   total: 19.5ms   remaining: 51ms
277:    learn: 14.7895225   total: 19.6ms   remaining: 50.8ms
278:    learn: 14.7290629   total: 19.6ms   remaining: 50.7ms
279:    learn: 14.6688505   total: 19.7ms   remaining: 50.6ms
280:    learn: 14.6088842   total: 19.7ms   remaining: 50.5ms
281:    learn: 14.5491631   total: 19.8ms   remaining: 50.4ms
282:    learn: 14.4896862   total: 19.9ms   remaining: 50.3ms
283:    learn: 14.4304523   total: 19.9ms   remaining: 50.2ms
284:    learn: 14.3714606   total: 20ms remaining: 50.1ms
285:    learn: 14.3127101   total: 20ms remaining: 49.9ms
286:    learn: 14.2541997   total: 20.1ms   remaining: 49.8ms
287:    learn: 14.1959286   total: 20.1ms   remaining: 49.7ms
288:    learn: 14.1378956   total: 20.2ms   remaining: 49.6ms
289:    learn: 14.0800999   total: 20.2ms   remaining: 49.5ms
290:    learn: 14.0225405   total: 20.3ms   remaining: 49.4ms
291:    learn: 13.9652163   total: 20.3ms   remaining: 49.3ms
292:    learn: 13.9081265   total: 20.4ms   remaining: 49.2ms
293:    learn: 13.8512701   total: 20.5ms   remaining: 49.1ms
294:    learn: 13.7946461   total: 20.5ms   remaining: 49ms
295:    learn: 13.7382536   total: 20.6ms   remaining: 48.9ms
296:    learn: 13.6820916   total: 20.6ms   remaining: 48.8ms
297:    learn: 13.6261592   total: 20.7ms   remaining: 48.7ms
298:    learn: 13.5704555   total: 20.8ms   remaining: 48.7ms
299:    learn: 13.5149795   total: 21ms remaining: 48.9ms
300:    learn: 13.4597302   total: 21ms remaining: 48.8ms
301:    learn: 13.4047068   total: 21.1ms   remaining: 48.7ms
302:    learn: 13.3499084   total: 21.1ms   remaining: 48.6ms
303:    learn: 13.2953340   total: 21.2ms   remaining: 48.5ms
304:    learn: 13.2409827   total: 21.2ms   remaining: 48.4ms
305:    learn: 13.1868535   total: 21.3ms   remaining: 48.3ms
306:    learn: 13.1329457   total: 21.3ms   remaining: 48.2ms
307:    learn: 13.0792582   total: 21.4ms   remaining: 48.1ms
308:    learn: 13.0257902   total: 21.5ms   remaining: 48ms
309:    learn: 12.9725407   total: 21.5ms   remaining: 47.9ms
310:    learn: 12.9195090   total: 21.6ms   remaining: 47.8ms
311:    learn: 12.8666940   total: 21.6ms   remaining: 47.7ms
312:    learn: 12.8140950   total: 21.7ms   remaining: 47.5ms
313:    learn: 12.7617110   total: 21.7ms   remaining: 47.4ms
314:    learn: 12.7095411   total: 22.5ms   remaining: 48.9ms
315:    learn: 12.6575845   total: 22.6ms   remaining: 48.8ms
316:    learn: 12.6058403   total: 22.6ms   remaining: 48.8ms
317:    learn: 12.5543076   total: 22.7ms   remaining: 48.7ms
318:    learn: 12.5029856   total: 22.9ms   remaining: 48.9ms
319:    learn: 12.4518734   total: 23ms remaining: 49ms
320:    learn: 12.4009702   total: 23.1ms   remaining: 48.9ms
321:    learn: 12.3502750   total: 23.2ms   remaining: 48.8ms
322:    learn: 12.2997871   total: 23.2ms   remaining: 48.7ms
323:    learn: 12.2495055   total: 23.3ms   remaining: 48.6ms
324:    learn: 12.1994296   total: 23.3ms   remaining: 48.4ms
325:    learn: 12.1495583   total: 23.4ms   remaining: 48.3ms
326:    learn: 12.0998909   total: 23.5ms   remaining: 48.3ms
327:    learn: 12.0504265   total: 23.5ms   remaining: 48.2ms
328:    learn: 12.0011644   total: 23.6ms   remaining: 48.1ms
329:    learn: 11.9521036   total: 23.6ms   remaining: 48ms
330:    learn: 11.9032434   total: 23.7ms   remaining: 47.9ms
331:    learn: 11.8545830   total: 23.7ms   remaining: 47.8ms
332:    learn: 11.8061215   total: 23.8ms   remaining: 47.7ms
333:    learn: 11.7578580   total: 23.8ms   remaining: 47.5ms
334:    learn: 11.7097919   total: 23.9ms   remaining: 47.5ms
335:    learn: 11.6619223   total: 23.9ms   remaining: 47.3ms
336:    learn: 11.6142483   total: 24ms remaining: 47.2ms
337:    learn: 11.5667693   total: 24.1ms   remaining: 47.2ms
338:    learn: 11.5194843   total: 24.1ms   remaining: 47ms
339:    learn: 11.4723927   total: 24.2ms   remaining: 47ms
340:    learn: 11.4254936   total: 24.2ms   remaining: 46.9ms
341:    learn: 11.3787861   total: 24.3ms   remaining: 46.8ms
342:    learn: 11.3322697   total: 24.4ms   remaining: 46.7ms
343:    learn: 11.2859433   total: 24.4ms   remaining: 46.6ms
344:    learn: 11.2398064   total: 24.5ms   remaining: 46.5ms
345:    learn: 11.1938581   total: 24.6ms   remaining: 46.4ms
346:    learn: 11.1480976   total: 24.6ms   remaining: 46.3ms
347:    learn: 11.1025242   total: 24.7ms   remaining: 46.2ms
348:    learn: 11.0571370   total: 24.7ms   remaining: 46.1ms
349:    learn: 11.0119355   total: 24.9ms   remaining: 46.2ms
350:    learn: 10.9669187   total: 25ms remaining: 46.2ms
351:    learn: 10.9220859   total: 25.1ms   remaining: 46.1ms
352:    learn: 10.8774364   total: 25.1ms   remaining: 46ms
353:    learn: 10.8329695   total: 25.2ms   remaining: 45.9ms
354:    learn: 10.7886843   total: 25.2ms   remaining: 45.8ms
355:    learn: 10.7445802   total: 25.3ms   remaining: 45.7ms
356:    learn: 10.7006563   total: 25.3ms   remaining: 45.6ms
357:    learn: 10.6569120   total: 25.4ms   remaining: 45.5ms
358:    learn: 10.6133466   total: 25.5ms   remaining: 45.5ms
359:    learn: 10.5699592   total: 25.5ms   remaining: 45.4ms
360:    learn: 10.5267492   total: 25.6ms   remaining: 45.3ms
361:    learn: 10.4837159   total: 25.6ms   remaining: 45.2ms
362:    learn: 10.4408584   total: 25.7ms   remaining: 45.1ms
363:    learn: 10.3981762   total: 25.7ms   remaining: 45ms
364:    learn: 10.3556685   total: 25.8ms   remaining: 44.9ms
365:    learn: 10.3133345   total: 25.8ms   remaining: 44.8ms
366:    learn: 10.2711736   total: 25.9ms   remaining: 44.7ms
367:    learn: 10.2291850   total: 26ms remaining: 44.6ms
368:    learn: 10.1873681   total: 26ms remaining: 44.5ms
369:    learn: 10.1457222   total: 26.1ms   remaining: 44.4ms
370:    learn: 10.1042464   total: 26.1ms   remaining: 44.3ms
371:    learn: 10.0629403   total: 26.2ms   remaining: 44.2ms
372:    learn: 10.0218030   total: 26.2ms   remaining: 44.1ms
373:    learn: 9.9808339    total: 26.3ms   remaining: 44ms
374:    learn: 9.9400322    total: 26.3ms   remaining: 43.9ms
375:    learn: 9.8993974    total: 26.4ms   remaining: 43.8ms
376:    learn: 9.8589286    total: 26.5ms   remaining: 43.7ms
377:    learn: 9.8186253    total: 26.5ms   remaining: 43.6ms
378:    learn: 9.7784868    total: 26.6ms   remaining: 43.6ms
379:    learn: 9.7385123    total: 26.7ms   remaining: 43.5ms
380:    learn: 9.6987013    total: 26.7ms   remaining: 43.4ms
381:    learn: 9.6590530    total: 26.9ms   remaining: 43.6ms
382:    learn: 9.6195668    total: 27ms remaining: 43.6ms
383:    learn: 9.5802420    total: 27.1ms   remaining: 43.5ms
384:    learn: 9.5410780    total: 27.2ms   remaining: 43.4ms
385:    learn: 9.5020741    total: 27.2ms   remaining: 43.3ms
386:    learn: 9.4632296    total: 27.3ms   remaining: 43.2ms
387:    learn: 9.4245439    total: 27.3ms   remaining: 43.1ms
388:    learn: 9.3860164    total: 27.4ms   remaining: 43.1ms
389:    learn: 9.3476463    total: 27.5ms   remaining: 43ms
390:    learn: 9.3094331    total: 27.5ms   remaining: 42.9ms
391:    learn: 9.2713762    total: 27.6ms   remaining: 42.8ms
392:    learn: 9.2334748    total: 27.6ms   remaining: 42.7ms
393:    learn: 9.1957284    total: 27.7ms   remaining: 42.6ms
394:    learn: 9.1581362    total: 27.8ms   remaining: 42.5ms
395:    learn: 9.1206978    total: 27.8ms   remaining: 42.4ms
396:    learn: 9.0834123    total: 27.9ms   remaining: 42.3ms
397:    learn: 9.0462794    total: 27.9ms   remaining: 42.2ms
398:    learn: 9.0092982    total: 28ms remaining: 42.1ms
399:    learn: 8.9724682    total: 28ms remaining: 42ms
400:    learn: 8.9357887    total: 28.1ms   remaining: 42ms
401:    learn: 8.8992592    total: 28.2ms   remaining: 41.9ms
402:    learn: 8.8628790    total: 28.2ms   remaining: 41.8ms
403:    learn: 8.8266476    total: 28.3ms   remaining: 41.7ms
404:    learn: 8.7905643    total: 28.3ms   remaining: 41.6ms
405:    learn: 8.7546284    total: 28.4ms   remaining: 41.6ms
406:    learn: 8.7188395    total: 28.5ms   remaining: 41.5ms
407:    learn: 8.6831969    total: 28.5ms   remaining: 41.4ms
408:    learn: 8.6477000    total: 28.6ms   remaining: 41.3ms
409:    learn: 8.6123482    total: 28.7ms   remaining: 41.2ms
410:    learn: 8.5771409    total: 28.7ms   remaining: 41.1ms
411:    learn: 8.5420776    total: 29.4ms   remaining: 42ms
412:    learn: 8.5071575    total: 29.5ms   remaining: 41.9ms
413:    learn: 8.4723803    total: 29.5ms   remaining: 41.8ms
414:    learn: 8.4377452    total: 29.6ms   remaining: 41.7ms
415:    learn: 8.4032517    total: 29.7ms   remaining: 41.7ms
416:    learn: 8.3688992    total: 29.8ms   remaining: 41.6ms
417:    learn: 8.3346871    total: 29.8ms   remaining: 41.5ms
418:    learn: 8.3006149    total: 29.9ms   remaining: 41.4ms
419:    learn: 8.2666820    total: 29.9ms   remaining: 41.3ms
420:    learn: 8.2328878    total: 30ms remaining: 41.3ms
421:    learn: 8.1992318    total: 30.1ms   remaining: 41.2ms
422:    learn: 8.1657133    total: 30.1ms   remaining: 41.1ms
423:    learn: 8.1323319    total: 30.2ms   remaining: 41ms
424:    learn: 8.0990869    total: 30.2ms   remaining: 40.9ms
425:    learn: 8.0659779    total: 30.3ms   remaining: 40.8ms
426:    learn: 8.0330041    total: 30.3ms   remaining: 40.7ms
427:    learn: 8.0001652    total: 30.4ms   remaining: 40.6ms
428:    learn: 7.9674605    total: 30.4ms   remaining: 40.5ms
429:    learn: 7.9348896    total: 30.5ms   remaining: 40.4ms
430:    learn: 7.9024517    total: 30.6ms   remaining: 40.4ms
431:    learn: 7.8701465    total: 30.6ms   remaining: 40.3ms
432:    learn: 7.8379734    total: 30.7ms   remaining: 40.2ms
433:    learn: 7.8059317    total: 30.7ms   remaining: 40.1ms
434:    learn: 7.7740211    total: 30.8ms   remaining: 40ms
435:    learn: 7.7422409    total: 30.9ms   remaining: 40ms
436:    learn: 7.7105906    total: 31.1ms   remaining: 40ms
437:    learn: 7.6790697    total: 31.1ms   remaining: 39.9ms
438:    learn: 7.6476777    total: 31.2ms   remaining: 39.8ms
439:    learn: 7.6164140    total: 31.2ms   remaining: 39.7ms
440:    learn: 7.5852781    total: 31.2ms   remaining: 39.6ms
441:    learn: 7.5542694    total: 31.3ms   remaining: 39.5ms
442:    learn: 7.5233876    total: 31.3ms   remaining: 39.4ms
443:    learn: 7.4926320    total: 31.4ms   remaining: 39.3ms
444:    learn: 7.4620021    total: 31.5ms   remaining: 39.2ms
445:    learn: 7.4314974    total: 31.5ms   remaining: 39.2ms
446:    learn: 7.4011175    total: 31.6ms   remaining: 39.1ms
447:    learn: 7.3708617    total: 31.6ms   remaining: 39ms
448:    learn: 7.3407296    total: 31.7ms   remaining: 38.9ms
449:    learn: 7.3107207    total: 31.7ms   remaining: 38.8ms
450:    learn: 7.2808345    total: 31.8ms   remaining: 38.7ms
451:    learn: 7.2510704    total: 31.9ms   remaining: 38.7ms
452:    learn: 7.2214281    total: 32ms remaining: 38.6ms
453:    learn: 7.1919069    total: 32ms remaining: 38.5ms
454:    learn: 7.1625064    total: 32.1ms   remaining: 38.4ms
455:    learn: 7.1332260    total: 32.2ms   remaining: 38.4ms
456:    learn: 7.1040654    total: 32.2ms   remaining: 38.3ms
457:    learn: 7.0750240    total: 32.3ms   remaining: 38.2ms
458:    learn: 7.0461013    total: 32.3ms   remaining: 38.1ms
459:    learn: 7.0172968    total: 32.4ms   remaining: 38ms
460:    learn: 6.9886101    total: 32.4ms   remaining: 37.9ms
461:    learn: 6.9600407    total: 32.5ms   remaining: 37.8ms
462:    learn: 6.9315880    total: 32.6ms   remaining: 37.8ms
463:    learn: 6.9032517    total: 32.6ms   remaining: 37.7ms
464:    learn: 6.8750312    total: 32.7ms   remaining: 37.6ms
465:    learn: 6.8469261    total: 32.8ms   remaining: 37.5ms
466:    learn: 6.8189359    total: 32.8ms   remaining: 37.5ms
467:    learn: 6.7910600    total: 33.1ms   remaining: 37.7ms
468:    learn: 6.7632982    total: 33.3ms   remaining: 37.7ms
469:    learn: 6.7356498    total: 33.3ms   remaining: 37.6ms
470:    learn: 6.7081145    total: 33.4ms   remaining: 37.5ms
471:    learn: 6.6806917    total: 33.4ms   remaining: 37.4ms
472:    learn: 6.6533811    total: 33.5ms   remaining: 37.3ms
473:    learn: 6.6261820    total: 33.5ms   remaining: 37.2ms
474:    learn: 6.5990942    total: 33.6ms   remaining: 37.1ms
475:    learn: 6.5721171    total: 33.6ms   remaining: 37ms
476:    learn: 6.5452503    total: 33.7ms   remaining: 36.9ms
477:    learn: 6.5184933    total: 33.7ms   remaining: 36.8ms
478:    learn: 6.4918457    total: 33.8ms   remaining: 36.8ms
479:    learn: 6.4653070    total: 33.9ms   remaining: 36.7ms
480:    learn: 6.4388769    total: 33.9ms   remaining: 36.6ms
481:    learn: 6.4125547    total: 34ms remaining: 36.5ms
482:    learn: 6.3863402    total: 34.1ms   remaining: 36.5ms
483:    learn: 6.3602329    total: 34.1ms   remaining: 36.4ms
484:    learn: 6.3342322    total: 34.2ms   remaining: 36.3ms
485:    learn: 6.3083379    total: 34.2ms   remaining: 36.2ms
486:    learn: 6.2825494    total: 34.3ms   remaining: 36.1ms
487:    learn: 6.2568663    total: 34.4ms   remaining: 36ms
488:    learn: 6.2312883    total: 34.4ms   remaining: 36ms
489:    learn: 6.2058148    total: 34.5ms   remaining: 35.9ms
490:    learn: 6.1804454    total: 34.5ms   remaining: 35.8ms
491:    learn: 6.1551797    total: 34.6ms   remaining: 35.7ms
492:    learn: 6.1300174    total: 34.6ms   remaining: 35.6ms
493:    learn: 6.1049578    total: 34.7ms   remaining: 35.5ms
494:    learn: 6.0800008    total: 34.8ms   remaining: 35.5ms
495:    learn: 6.0551457    total: 34.9ms   remaining: 35.5ms
496:    learn: 6.0303923    total: 35ms remaining: 35.5ms
497:    learn: 6.0057401    total: 35.1ms   remaining: 35.4ms
498:    learn: 5.9811886    total: 35.1ms   remaining: 35.3ms
499:    learn: 5.9567375    total: 35.2ms   remaining: 35.2ms
500:    learn: 5.9323864    total: 35.2ms   remaining: 35.1ms
501:    learn: 5.9081348    total: 35.3ms   remaining: 35ms
502:    learn: 5.8839823    total: 35.3ms   remaining: 34.9ms
503:    learn: 5.8599286    total: 35.4ms   remaining: 34.8ms
504:    learn: 5.8359732    total: 35.5ms   remaining: 34.8ms
505:    learn: 5.8121157    total: 35.5ms   remaining: 34.7ms
506:    learn: 5.7883558    total: 35.6ms   remaining: 34.6ms
507:    learn: 5.7646930    total: 35.7ms   remaining: 34.5ms
508:    learn: 5.7411269    total: 35.7ms   remaining: 34.5ms
509:    learn: 5.7176572    total: 35.8ms   remaining: 34.4ms
510:    learn: 5.6942834    total: 35.9ms   remaining: 34.3ms
511:    learn: 5.6710052    total: 35.9ms   remaining: 34.2ms
512:    learn: 5.6478221    total: 36ms remaining: 34.1ms
513:    learn: 5.6247338    total: 36ms remaining: 34.1ms
514:    learn: 5.6017399    total: 36.1ms   remaining: 34ms
515:    learn: 5.5788400    total: 36.1ms   remaining: 33.9ms
516:    learn: 5.5560337    total: 36.2ms   remaining: 33.8ms
517:    learn: 5.5333207    total: 36.2ms   remaining: 33.7ms
518:    learn: 5.5107004    total: 36.3ms   remaining: 33.6ms
519:    learn: 5.4881727    total: 36.3ms   remaining: 33.5ms
520:    learn: 5.4657370    total: 36.4ms   remaining: 33.5ms
521:    learn: 5.4433931    total: 36.5ms   remaining: 33.4ms
522:    learn: 5.4211405    total: 36.5ms   remaining: 33.3ms
523:    learn: 5.3989789    total: 36.6ms   remaining: 33.2ms
524:    learn: 5.3769079    total: 36.6ms   remaining: 33.1ms
525:    learn: 5.3549271    total: 36.7ms   remaining: 33.1ms
526:    learn: 5.3330361    total: 36.8ms   remaining: 33ms
527:    learn: 5.3112347    total: 36.9ms   remaining: 33ms
528:    learn: 5.2895224    total: 37ms remaining: 33ms
529:    learn: 5.2678988    total: 37.1ms   remaining: 32.9ms
530:    learn: 5.2463636    total: 37.1ms   remaining: 32.8ms
531:    learn: 5.2249165    total: 37.2ms   remaining: 32.7ms
532:    learn: 5.2035570    total: 37.2ms   remaining: 32.6ms
533:    learn: 5.1822849    total: 37.3ms   remaining: 32.5ms
534:    learn: 5.1610997    total: 37.3ms   remaining: 32.4ms
535:    learn: 5.1400011    total: 37.4ms   remaining: 32.3ms
536:    learn: 5.1189888    total: 37.4ms   remaining: 32.3ms
537:    learn: 5.0980624    total: 37.5ms   remaining: 32.2ms
538:    learn: 5.0772215    total: 37.5ms   remaining: 32.1ms
539:    learn: 5.0564658    total: 37.6ms   remaining: 32ms
540:    learn: 5.0357950    total: 37.6ms   remaining: 31.9ms
541:    learn: 5.0152087    total: 37.7ms   remaining: 31.8ms
542:    learn: 4.9947065    total: 37.7ms   remaining: 31.7ms
543:    learn: 4.9742881    total: 37.8ms   remaining: 31.7ms
544:    learn: 4.9539532    total: 37.8ms   remaining: 31.6ms
545:    learn: 4.9337015    total: 37.9ms   remaining: 31.5ms
546:    learn: 4.9135325    total: 37.9ms   remaining: 31.4ms
547:    learn: 4.8934460    total: 38ms remaining: 31.3ms
548:    learn: 4.8734416    total: 38ms remaining: 31.3ms
549:    learn: 4.8535190    total: 38.1ms   remaining: 31.2ms
550:    learn: 4.8336778    total: 38.2ms   remaining: 31.1ms
551:    learn: 4.8139177    total: 38.2ms   remaining: 31ms
552:    learn: 4.7942384    total: 38.3ms   remaining: 30.9ms
553:    learn: 4.7746395    total: 38.4ms   remaining: 30.9ms
554:    learn: 4.7551208    total: 38.4ms   remaining: 30.8ms
555:    learn: 4.7356819    total: 38.5ms   remaining: 30.7ms
556:    learn: 4.7163224    total: 38.5ms   remaining: 30.6ms
557:    learn: 4.6970421    total: 38.6ms   remaining: 30.5ms
558:    learn: 4.6778406    total: 38.6ms   remaining: 30.5ms
559:    learn: 4.6587176    total: 38.7ms   remaining: 30.4ms
560:    learn: 4.6396727    total: 38.7ms   remaining: 30.3ms
561:    learn: 4.6207058    total: 38.9ms   remaining: 30.4ms
562:    learn: 4.6018163    total: 39.1ms   remaining: 30.3ms
563:    learn: 4.5830041    total: 39.1ms   remaining: 30.2ms
564:    learn: 4.5642688    total: 39.2ms   remaining: 30.2ms
565:    learn: 4.5456100    total: 39.2ms   remaining: 30.1ms
566:    learn: 4.5270276    total: 39.3ms   remaining: 30ms
567:    learn: 4.5085211    total: 39.3ms   remaining: 29.9ms
568:    learn: 4.4900903    total: 39.4ms   remaining: 29.9ms
569:    learn: 4.4717348    total: 39.5ms   remaining: 29.8ms
570:    learn: 4.4534543    total: 39.5ms   remaining: 29.7ms
571:    learn: 4.4352486    total: 39.6ms   remaining: 29.6ms
572:    learn: 4.4171173    total: 39.6ms   remaining: 29.5ms
573:    learn: 4.3990601    total: 39.7ms   remaining: 29.5ms
574:    learn: 4.3810768    total: 39.8ms   remaining: 29.4ms
575:    learn: 4.3631669    total: 39.8ms   remaining: 29.3ms
576:    learn: 4.3453303    total: 39.9ms   remaining: 29.2ms
577:    learn: 4.3275666    total: 39.9ms   remaining: 29.2ms
578:    learn: 4.3098755    total: 40ms remaining: 29.1ms
579:    learn: 4.2922567    total: 40ms remaining: 29ms
580:    learn: 4.2747100    total: 40.1ms   remaining: 28.9ms
581:    learn: 4.2572350    total: 40.2ms   remaining: 28.9ms
582:    learn: 4.2398314    total: 40.2ms   remaining: 28.8ms
583:    learn: 4.2224990    total: 40.3ms   remaining: 28.7ms
584:    learn: 4.2052374    total: 40.3ms   remaining: 28.6ms
585:    learn: 4.1880464    total: 40.4ms   remaining: 28.5ms
586:    learn: 4.1709256    total: 40.4ms   remaining: 28.4ms
587:    learn: 4.1538749    total: 40.5ms   remaining: 28.4ms
588:    learn: 4.1368939    total: 40.5ms   remaining: 28.3ms
589:    learn: 4.1199822    total: 40.6ms   remaining: 28.2ms
590:    learn: 4.1031398    total: 40.6ms   remaining: 28.1ms
591:    learn: 4.0863661    total: 40.7ms   remaining: 28.1ms
592:    learn: 4.0696611    total: 40.8ms   remaining: 28ms
593:    learn: 4.0530243    total: 41ms remaining: 28ms
594:    learn: 4.0364555    total: 41ms remaining: 27.9ms
595:    learn: 4.0199545    total: 41.1ms   remaining: 27.8ms
596:    learn: 4.0035209    total: 41.1ms   remaining: 27.7ms
597:    learn: 3.9871545    total: 41.2ms   remaining: 27.7ms
598:    learn: 3.9708550    total: 41.2ms   remaining: 27.6ms
599:    learn: 3.9546222    total: 41.3ms   remaining: 27.5ms
600:    learn: 3.9384557    total: 41.3ms   remaining: 27.4ms
601:    learn: 3.9223553    total: 41.4ms   remaining: 27.4ms
602:    learn: 3.9063207    total: 41.4ms   remaining: 27.3ms
603:    learn: 3.8903516    total: 41.5ms   remaining: 27.2ms
604:    learn: 3.8744479    total: 41.5ms   remaining: 27.1ms
605:    learn: 3.8586091    total: 41.6ms   remaining: 27.1ms
606:    learn: 3.8428352    total: 41.7ms   remaining: 27ms
607:    learn: 3.8271256    total: 41.7ms   remaining: 26.9ms
608:    learn: 3.8114804    total: 41.8ms   remaining: 26.8ms
609:    learn: 3.7958990    total: 41.9ms   remaining: 26.8ms
610:    learn: 3.7803814    total: 41.9ms   remaining: 26.7ms
611:    learn: 3.7649272    total: 42ms remaining: 26.6ms
612:    learn: 3.7495362    total: 42ms remaining: 26.5ms
613:    learn: 3.7342081    total: 42.1ms   remaining: 26.5ms
614:    learn: 3.7189426    total: 42.2ms   remaining: 26.4ms
615:    learn: 3.7037396    total: 42.2ms   remaining: 26.3ms
616:    learn: 3.6885987    total: 42.3ms   remaining: 26.2ms
617:    learn: 3.6735197    total: 42.3ms   remaining: 26.2ms
618:    learn: 3.6585024    total: 42.4ms   remaining: 26.1ms
619:    learn: 3.6435464    total: 42.4ms   remaining: 26ms
620:    learn: 3.6286516    total: 42.5ms   remaining: 25.9ms
621:    learn: 3.6138177    total: 42.6ms   remaining: 25.9ms
622:    learn: 3.5990444    total: 42.6ms   remaining: 25.8ms
623:    learn: 3.5843315    total: 42.7ms   remaining: 25.7ms
624:    learn: 3.5696787    total: 42.7ms   remaining: 25.6ms
625:    learn: 3.5550859    total: 43ms remaining: 25.7ms
626:    learn: 3.5405527    total: 43.1ms   remaining: 25.6ms
627:    learn: 3.5260789    total: 43.1ms   remaining: 25.5ms
628:    learn: 3.5116643    total: 43.2ms   remaining: 25.5ms
629:    learn: 3.4973086    total: 43.2ms   remaining: 25.4ms
630:    learn: 3.4830116    total: 43.3ms   remaining: 25.3ms
631:    learn: 3.4687731    total: 43.3ms   remaining: 25.2ms
632:    learn: 3.4545927    total: 43.4ms   remaining: 25.2ms
633:    learn: 3.4404704    total: 43.4ms   remaining: 25.1ms
634:    learn: 3.4264057    total: 43.5ms   remaining: 25ms
635:    learn: 3.4123986    total: 43.6ms   remaining: 24.9ms
636:    learn: 3.3984487    total: 43.6ms   remaining: 24.9ms
637:    learn: 3.3845558    total: 43.7ms   remaining: 24.8ms
638:    learn: 3.3707198    total: 43.7ms   remaining: 24.7ms
639:    learn: 3.3569403    total: 43.8ms   remaining: 24.6ms
640:    learn: 3.3432171    total: 43.9ms   remaining: 24.6ms
641:    learn: 3.3295500    total: 43.9ms   remaining: 24.5ms
642:    learn: 3.3159388    total: 44ms remaining: 24.4ms
643:    learn: 3.3023833    total: 44ms remaining: 24.3ms
644:    learn: 3.2888831    total: 44.1ms   remaining: 24.3ms
645:    learn: 3.2754382    total: 44.1ms   remaining: 24.2ms
646:    learn: 3.2620482    total: 44.2ms   remaining: 24.1ms
647:    learn: 3.2487129    total: 44.2ms   remaining: 24ms
648:    learn: 3.2354322    total: 44.3ms   remaining: 24ms
649:    learn: 3.2222057    total: 44.4ms   remaining: 23.9ms
650:    learn: 3.2090334    total: 44.4ms   remaining: 23.8ms
651:    learn: 3.1959148    total: 44.5ms   remaining: 23.7ms
652:    learn: 3.1828499    total: 44.5ms   remaining: 23.7ms
653:    learn: 3.1698384    total: 44.6ms   remaining: 23.6ms
654:    learn: 3.1568801    total: 44.6ms   remaining: 23.5ms
655:    learn: 3.1439748    total: 44.7ms   remaining: 23.4ms
656:    learn: 3.1311222    total: 44.8ms   remaining: 23.4ms
657:    learn: 3.1183222    total: 45ms remaining: 23.4ms
658:    learn: 3.1055745    total: 45ms remaining: 23.3ms
659:    learn: 3.0928789    total: 45.1ms   remaining: 23.2ms
660:    learn: 3.0802352    total: 45.1ms   remaining: 23.1ms
661:    learn: 3.0676432    total: 45.2ms   remaining: 23.1ms
662:    learn: 3.0551027    total: 45.3ms   remaining: 23ms
663:    learn: 3.0426134    total: 45.3ms   remaining: 22.9ms
664:    learn: 3.0301752    total: 45.4ms   remaining: 22.9ms
665:    learn: 3.0177879    total: 45.4ms   remaining: 22.8ms
666:    learn: 3.0054512    total: 45.5ms   remaining: 22.7ms
667:    learn: 2.9931649    total: 45.5ms   remaining: 22.6ms
668:    learn: 2.9809288    total: 45.6ms   remaining: 22.6ms
669:    learn: 2.9687428    total: 45.7ms   remaining: 22.5ms
670:    learn: 2.9566066    total: 45.8ms   remaining: 22.4ms
671:    learn: 2.9445200    total: 45.8ms   remaining: 22.4ms
672:    learn: 2.9324828    total: 45.9ms   remaining: 22.3ms
673:    learn: 2.9204948    total: 45.9ms   remaining: 22.2ms
674:    learn: 2.9085558    total: 46ms remaining: 22.1ms
675:    learn: 2.8966656    total: 46ms remaining: 22.1ms
676:    learn: 2.8848241    total: 46.1ms   remaining: 22ms
677:    learn: 2.8730309    total: 46.1ms   remaining: 21.9ms
678:    learn: 2.8612859    total: 46.2ms   remaining: 21.8ms
679:    learn: 2.8495890    total: 46.2ms   remaining: 21.8ms
680:    learn: 2.8379399    total: 46.3ms   remaining: 21.7ms
681:    learn: 2.8263384    total: 46.3ms   remaining: 21.6ms
682:    learn: 2.8147843    total: 46.4ms   remaining: 21.5ms
683:    learn: 2.8032775    total: 46.4ms   remaining: 21.4ms
684:    learn: 2.7918177    total: 46.5ms   remaining: 21.4ms
685:    learn: 2.7804047    total: 46.5ms   remaining: 21.3ms
686:    learn: 2.7690384    total: 46.6ms   remaining: 21.2ms
687:    learn: 2.7577186    total: 46.6ms   remaining: 21.2ms
688:    learn: 2.7464451    total: 46.9ms   remaining: 21.2ms
689:    learn: 2.7352176    total: 47.1ms   remaining: 21.1ms
690:    learn: 2.7240360    total: 47.1ms   remaining: 21.1ms
691:    learn: 2.7129002    total: 47.2ms   remaining: 21ms
692:    learn: 2.7018098    total: 47.3ms   remaining: 20.9ms
693:    learn: 2.6907648    total: 47.3ms   remaining: 20.9ms
694:    learn: 2.6797650    total: 47.3ms   remaining: 20.8ms
695:    learn: 2.6688101    total: 47.4ms   remaining: 20.7ms
696:    learn: 2.6579000    total: 47.5ms   remaining: 20.6ms
697:    learn: 2.6470345    total: 47.5ms   remaining: 20.6ms
698:    learn: 2.6362134    total: 47.6ms   remaining: 20.5ms
699:    learn: 2.6254366    total: 47.7ms   remaining: 20.4ms
700:    learn: 2.6147038    total: 47.7ms   remaining: 20.4ms
701:    learn: 2.6040149    total: 47.8ms   remaining: 20.3ms
702:    learn: 2.5933697    total: 47.8ms   remaining: 20.2ms
703:    learn: 2.5827680    total: 47.9ms   remaining: 20.1ms
704:    learn: 2.5722096    total: 47.9ms   remaining: 20.1ms
705:    learn: 2.5616944    total: 48ms remaining: 20ms
706:    learn: 2.5512222    total: 48ms remaining: 19.9ms
707:    learn: 2.5407928    total: 48.1ms   remaining: 19.8ms
708:    learn: 2.5304061    total: 48.1ms   remaining: 19.8ms
709:    learn: 2.5200618    total: 48.2ms   remaining: 19.7ms
710:    learn: 2.5097598    total: 48.2ms   remaining: 19.6ms
711:    learn: 2.4994999    total: 48.3ms   remaining: 19.5ms
712:    learn: 2.4892819    total: 48.4ms   remaining: 19.5ms
713:    learn: 2.4791057    total: 48.4ms   remaining: 19.4ms
714:    learn: 2.4689711    total: 48.5ms   remaining: 19.3ms
715:    learn: 2.4588780    total: 48.5ms   remaining: 19.3ms
716:    learn: 2.4488261    total: 48.6ms   remaining: 19.2ms
717:    learn: 2.4388153    total: 48.7ms   remaining: 19.1ms
718:    learn: 2.4288454    total: 48.7ms   remaining: 19ms
719:    learn: 2.4189163    total: 49ms remaining: 19.1ms
720:    learn: 2.4090278    total: 49.2ms   remaining: 19ms
721:    learn: 2.3991797    total: 49.2ms   remaining: 19ms
722:    learn: 2.3893718    total: 49.3ms   remaining: 18.9ms
723:    learn: 2.3796041    total: 49.3ms   remaining: 18.8ms
724:    learn: 2.3698762    total: 49.4ms   remaining: 18.7ms
725:    learn: 2.3601882    total: 49.4ms   remaining: 18.6ms
726:    learn: 2.3505397    total: 49.4ms   remaining: 18.6ms
727:    learn: 2.3409307    total: 49.5ms   remaining: 18.5ms
728:    learn: 2.3313610    total: 49.6ms   remaining: 18.4ms
729:    learn: 2.3218304    total: 49.6ms   remaining: 18.4ms
730:    learn: 2.3123388    total: 49.7ms   remaining: 18.3ms
731:    learn: 2.3028859    total: 49.7ms   remaining: 18.2ms
732:    learn: 2.2934717    total: 49.8ms   remaining: 18.1ms
733:    learn: 2.2840960    total: 49.8ms   remaining: 18.1ms
734:    learn: 2.2747586    total: 49.9ms   remaining: 18ms
735:    learn: 2.2654594    total: 49.9ms   remaining: 17.9ms
736:    learn: 2.2561982    total: 50ms remaining: 17.8ms
737:    learn: 2.2469749    total: 50ms remaining: 17.8ms
738:    learn: 2.2377892    total: 50.1ms   remaining: 17.7ms
739:    learn: 2.2286412    total: 50.2ms   remaining: 17.6ms
740:    learn: 2.2195305    total: 50.2ms   remaining: 17.5ms
741:    learn: 2.2104570    total: 50.3ms   remaining: 17.5ms
742:    learn: 2.2014207    total: 50.3ms   remaining: 17.4ms
743:    learn: 2.1924213    total: 50.4ms   remaining: 17.3ms
744:    learn: 2.1834587    total: 50.4ms   remaining: 17.3ms
745:    learn: 2.1745327    total: 50.5ms   remaining: 17.2ms
746:    learn: 2.1656432    total: 50.5ms   remaining: 17.1ms
747:    learn: 2.1567900    total: 50.6ms   remaining: 17ms
748:    learn: 2.1479731    total: 50.6ms   remaining: 17ms
749:    learn: 2.1391922    total: 50.7ms   remaining: 16.9ms
750:    learn: 2.1304472    total: 50.7ms   remaining: 16.8ms
751:    learn: 2.1217379    total: 50.8ms   remaining: 16.7ms
752:    learn: 2.1130642    total: 50.8ms   remaining: 16.7ms
753:    learn: 2.1044260    total: 51ms remaining: 16.6ms
754:    learn: 2.0958231    total: 51.2ms   remaining: 16.6ms
755:    learn: 2.0872554    total: 51.2ms   remaining: 16.5ms
756:    learn: 2.0787227    total: 51.3ms   remaining: 16.5ms
757:    learn: 2.0702249    total: 51.3ms   remaining: 16.4ms
758:    learn: 2.0617618    total: 51.4ms   remaining: 16.3ms
759:    learn: 2.0533333    total: 51.5ms   remaining: 16.3ms
760:    learn: 2.0449393    total: 51.5ms   remaining: 16.2ms
761:    learn: 2.0365796    total: 51.6ms   remaining: 16.1ms
762:    learn: 2.0282540    total: 51.6ms   remaining: 16ms
763:    learn: 2.0199625    total: 51.7ms   remaining: 16ms
764:    learn: 2.0117049    total: 51.8ms   remaining: 15.9ms
765:    learn: 2.0034811    total: 51.8ms   remaining: 15.8ms
766:    learn: 1.9952909    total: 51.9ms   remaining: 15.8ms
767:    learn: 1.9871341    total: 52ms remaining: 15.7ms
768:    learn: 1.9790107    total: 52.6ms   remaining: 15.8ms
769:    learn: 1.9709205    total: 52.7ms   remaining: 15.7ms
770:    learn: 1.9628634    total: 52.7ms   remaining: 15.7ms
771:    learn: 1.9548392    total: 52.8ms   remaining: 15.6ms
772:    learn: 1.9468478    total: 52.8ms   remaining: 15.5ms
773:    learn: 1.9388891    total: 52.9ms   remaining: 15.4ms
774:    learn: 1.9309629    total: 52.9ms   remaining: 15.4ms
775:    learn: 1.9230691    total: 53ms remaining: 15.3ms
776:    learn: 1.9152076    total: 53.2ms   remaining: 15.3ms
777:    learn: 1.9073783    total: 53.3ms   remaining: 15.2ms
778:    learn: 1.8995809    total: 53.3ms   remaining: 15.1ms
779:    learn: 1.8918154    total: 53.4ms   remaining: 15.1ms
780:    learn: 1.8840817    total: 53.4ms   remaining: 15ms
781:    learn: 1.8763796    total: 53.5ms   remaining: 14.9ms
782:    learn: 1.8687089    total: 53.5ms   remaining: 14.8ms
783:    learn: 1.8610696    total: 53.6ms   remaining: 14.8ms
784:    learn: 1.8534616    total: 53.6ms   remaining: 14.7ms
785:    learn: 1.8458846    total: 53.7ms   remaining: 14.6ms
786:    learn: 1.8383387    total: 53.8ms   remaining: 14.6ms
787:    learn: 1.8308235    total: 53.8ms   remaining: 14.5ms
788:    learn: 1.8233391    total: 53.9ms   remaining: 14.4ms
789:    learn: 1.8158853    total: 53.9ms   remaining: 14.3ms
790:    learn: 1.8084620    total: 54ms remaining: 14.3ms
791:    learn: 1.8010690    total: 54.1ms   remaining: 14.2ms
792:    learn: 1.7937062    total: 54.1ms   remaining: 14.1ms
793:    learn: 1.7863735    total: 54.2ms   remaining: 14.1ms
794:    learn: 1.7790708    total: 54.3ms   remaining: 14ms
795:    learn: 1.7717980    total: 54.3ms   remaining: 13.9ms
796:    learn: 1.7645549    total: 54.4ms   remaining: 13.8ms
797:    learn: 1.7573414    total: 54.4ms   remaining: 13.8ms
798:    learn: 1.7501574    total: 54.5ms   remaining: 13.7ms
799:    learn: 1.7430027    total: 54.5ms   remaining: 13.6ms
800:    learn: 1.7358773    total: 54.6ms   remaining: 13.6ms
801:    learn: 1.7287811    total: 54.7ms   remaining: 13.5ms
802:    learn: 1.7217138    total: 54.7ms   remaining: 13.4ms
803:    learn: 1.7146755    total: 54.8ms   remaining: 13.3ms
804:    learn: 1.7076659    total: 54.8ms   remaining: 13.3ms
805:    learn: 1.7006849    total: 54.9ms   remaining: 13.2ms
806:    learn: 1.6937325    total: 54.9ms   remaining: 13.1ms
807:    learn: 1.6868085    total: 55ms remaining: 13.1ms
808:    learn: 1.6799129    total: 55.2ms   remaining: 13ms
809:    learn: 1.6730454    total: 55.3ms   remaining: 13ms
810:    learn: 1.6662060    total: 55.4ms   remaining: 12.9ms
811:    learn: 1.6593945    total: 55.4ms   remaining: 12.8ms
812:    learn: 1.6526109    total: 56.2ms   remaining: 12.9ms
813:    learn: 1.6458551    total: 56.3ms   remaining: 12.9ms
814:    learn: 1.6391268    total: 56.3ms   remaining: 12.8ms
815:    learn: 1.6324260    total: 56.4ms   remaining: 12.7ms
816:    learn: 1.6257527    total: 56.4ms   remaining: 12.6ms
817:    learn: 1.6191066    total: 56.5ms   remaining: 12.6ms
818:    learn: 1.6124877    total: 56.5ms   remaining: 12.5ms
819:    learn: 1.6058959    total: 56.6ms   remaining: 12.4ms
820:    learn: 1.5993310    total: 56.7ms   remaining: 12.4ms
821:    learn: 1.5927929    total: 56.7ms   remaining: 12.3ms
822:    learn: 1.5862815    total: 56.8ms   remaining: 12.2ms
823:    learn: 1.5797968    total: 56.8ms   remaining: 12.1ms
824:    learn: 1.5733386    total: 56.9ms   remaining: 12.1ms
825:    learn: 1.5669068    total: 57ms remaining: 12ms
826:    learn: 1.5605013    total: 57ms remaining: 11.9ms
827:    learn: 1.5541220    total: 57.1ms   remaining: 11.9ms
828:    learn: 1.5477687    total: 57.1ms   remaining: 11.8ms
829:    learn: 1.5414414    total: 57.2ms   remaining: 11.7ms
830:    learn: 1.5351400    total: 57.4ms   remaining: 11.7ms
831:    learn: 1.5288644    total: 57.5ms   remaining: 11.6ms
832:    learn: 1.5226144    total: 57.5ms   remaining: 11.5ms
833:    learn: 1.5163899    total: 57.5ms   remaining: 11.5ms
834:    learn: 1.5101909    total: 57.6ms   remaining: 11.4ms
835:    learn: 1.5040173    total: 57.6ms   remaining: 11.3ms
836:    learn: 1.4978688    total: 57.7ms   remaining: 11.2ms
837:    learn: 1.4917456    total: 57.8ms   remaining: 11.2ms
838:    learn: 1.4856473    total: 57.8ms   remaining: 11.1ms
839:    learn: 1.4795740    total: 57.9ms   remaining: 11ms
840:    learn: 1.4735255    total: 58ms remaining: 11ms
841:    learn: 1.4675017    total: 58ms remaining: 10.9ms
842:    learn: 1.4615026    total: 58.1ms   remaining: 10.8ms
843:    learn: 1.4555279    total: 58.1ms   remaining: 10.7ms
844:    learn: 1.4495777    total: 58.2ms   remaining: 10.7ms
845:    learn: 1.4436519    total: 58.3ms   remaining: 10.6ms
846:    learn: 1.4377502    total: 58.3ms   remaining: 10.5ms
847:    learn: 1.4318727    total: 58.4ms   remaining: 10.5ms
848:    learn: 1.4260192    total: 58.4ms   remaining: 10.4ms
849:    learn: 1.4201896    total: 58.5ms   remaining: 10.3ms
850:    learn: 1.4143839    total: 58.6ms   remaining: 10.3ms
851:    learn: 1.4086019    total: 58.6ms   remaining: 10.2ms
852:    learn: 1.4028435    total: 58.7ms   remaining: 10.1ms
853:    learn: 1.3971087    total: 58.7ms   remaining: 10ms
854:    learn: 1.3913973    total: 58.8ms   remaining: 9.97ms
855:    learn: 1.3857093    total: 58.9ms   remaining: 9.9ms
856:    learn: 1.3800445    total: 58.9ms   remaining: 9.83ms
857:    learn: 1.3744029    total: 59ms remaining: 9.76ms
858:    learn: 1.3687843    total: 59ms remaining: 9.69ms
859:    learn: 1.3631887    total: 59.1ms   remaining: 9.62ms
860:    learn: 1.3576160    total: 59.1ms   remaining: 9.55ms
861:    learn: 1.3520661    total: 59.3ms   remaining: 9.5ms
862:    learn: 1.3465388    total: 59.5ms   remaining: 9.44ms
863:    learn: 1.3410342    total: 59.5ms   remaining: 9.37ms
864:    learn: 1.3355520    total: 59.6ms   remaining: 9.29ms
865:    learn: 1.3300923    total: 59.6ms   remaining: 9.22ms
866:    learn: 1.3246549    total: 59.7ms   remaining: 9.15ms
867:    learn: 1.3192397    total: 59.7ms   remaining: 9.08ms
868:    learn: 1.3138467    total: 59.8ms   remaining: 9.01ms
869:    learn: 1.3084756    total: 59.8ms   remaining: 8.94ms
870:    learn: 1.3031266    total: 59.9ms   remaining: 8.88ms
871:    learn: 1.2977994    total: 60ms remaining: 8.8ms
872:    learn: 1.2924940    total: 60ms remaining: 8.73ms
873:    learn: 1.2872103    total: 60.1ms   remaining: 8.66ms
874:    learn: 1.2819482    total: 60.2ms   remaining: 8.59ms
875:    learn: 1.2767076    total: 60.2ms   remaining: 8.52ms
876:    learn: 1.2714884    total: 60.3ms   remaining: 8.45ms
877:    learn: 1.2662906    total: 60.4ms   remaining: 8.39ms
878:    learn: 1.2611140    total: 60.4ms   remaining: 8.32ms
879:    learn: 1.2559585    total: 60.5ms   remaining: 8.25ms
880:    learn: 1.2508242    total: 60.5ms   remaining: 8.18ms
881:    learn: 1.2457108    total: 60.6ms   remaining: 8.11ms
882:    learn: 1.2406183    total: 60.7ms   remaining: 8.04ms
883:    learn: 1.2355467    total: 60.7ms   remaining: 7.97ms
884:    learn: 1.2304958    total: 60.8ms   remaining: 7.89ms
885:    learn: 1.2254655    total: 60.8ms   remaining: 7.83ms
886:    learn: 1.2204558    total: 60.9ms   remaining: 7.76ms
887:    learn: 1.2154666    total: 60.9ms   remaining: 7.69ms
888:    learn: 1.2104978    total: 61ms remaining: 7.62ms
889:    learn: 1.2055492    total: 61ms remaining: 7.54ms
890:    learn: 1.2006210    total: 61.1ms   remaining: 7.47ms
891:    learn: 1.1957128    total: 61.2ms   remaining: 7.4ms
892:    learn: 1.1908247    total: 61.2ms   remaining: 7.33ms
893:    learn: 1.1859566    total: 61.3ms   remaining: 7.26ms
894:    learn: 1.1811085    total: 61.3ms   remaining: 7.19ms
895:    learn: 1.1762801    total: 61.4ms   remaining: 7.13ms
896:    learn: 1.1714715    total: 61.7ms   remaining: 7.08ms
897:    learn: 1.1666825    total: 61.7ms   remaining: 7.01ms
898:    learn: 1.1619131    total: 61.8ms   remaining: 6.94ms
899:    learn: 1.1571632    total: 61.8ms   remaining: 6.87ms
900:    learn: 1.1524327    total: 61.9ms   remaining: 6.8ms
901:    learn: 1.1477216    total: 61.9ms   remaining: 6.72ms
902:    learn: 1.1430297    total: 61.9ms   remaining: 6.65ms
903:    learn: 1.1383570    total: 62ms remaining: 6.58ms
904:    learn: 1.1337034    total: 62.1ms   remaining: 6.52ms
905:    learn: 1.1290688    total: 62.1ms   remaining: 6.45ms
906:    learn: 1.1244531    total: 62.2ms   remaining: 6.38ms
907:    learn: 1.1198564    total: 62.3ms   remaining: 6.31ms
908:    learn: 1.1152784    total: 62.3ms   remaining: 6.24ms
909:    learn: 1.1107191    total: 62.4ms   remaining: 6.17ms
910:    learn: 1.1061785    total: 62.4ms   remaining: 6.1ms
911:    learn: 1.1016565    total: 62.5ms   remaining: 6.03ms
912:    learn: 1.0971529    total: 62.5ms   remaining: 5.96ms
913:    learn: 1.0926677    total: 62.6ms   remaining: 5.89ms
914:    learn: 1.0882009    total: 62.6ms   remaining: 5.82ms
915:    learn: 1.0837523    total: 62.7ms   remaining: 5.75ms
916:    learn: 1.0793220    total: 62.7ms   remaining: 5.68ms
917:    learn: 1.0749097    total: 62.8ms   remaining: 5.61ms
918:    learn: 1.0705155    total: 62.9ms   remaining: 5.54ms
919:    learn: 1.0661392    total: 62.9ms   remaining: 5.47ms
920:    learn: 1.0617808    total: 63ms remaining: 5.4ms
921:    learn: 1.0574403    total: 63ms remaining: 5.33ms
922:    learn: 1.0531174    total: 63.1ms   remaining: 5.26ms
923:    learn: 1.0488123    total: 63.2ms   remaining: 5.2ms
924:    learn: 1.0445248    total: 63.2ms   remaining: 5.13ms
925:    learn: 1.0402547    total: 63.3ms   remaining: 5.06ms
926:    learn: 1.0360022    total: 63.4ms   remaining: 4.99ms
927:    learn: 1.0317670    total: 63.4ms   remaining: 4.92ms
928:    learn: 1.0275491    total: 63.6ms   remaining: 4.86ms
929:    learn: 1.0233485    total: 63.7ms   remaining: 4.8ms
930:    learn: 1.0191651    total: 63.8ms   remaining: 4.73ms
931:    learn: 1.0149987    total: 63.8ms   remaining: 4.66ms
932:    learn: 1.0108494    total: 63.9ms   remaining: 4.59ms
933:    learn: 1.0067171    total: 63.9ms   remaining: 4.52ms
934:    learn: 1.0026016    total: 64ms remaining: 4.45ms
935:    learn: 0.9985030    total: 64.1ms   remaining: 4.38ms
936:    learn: 0.9944211    total: 64.1ms   remaining: 4.31ms
937:    learn: 0.9903559    total: 64.2ms   remaining: 4.24ms
938:    learn: 0.9863073    total: 64.2ms   remaining: 4.17ms
939:    learn: 0.9822753    total: 64.3ms   remaining: 4.1ms
940:    learn: 0.9782598    total: 64.4ms   remaining: 4.04ms
941:    learn: 0.9742606    total: 64.4ms   remaining: 3.97ms
942:    learn: 0.9702778    total: 64.5ms   remaining: 3.9ms
943:    learn: 0.9663114    total: 64.6ms   remaining: 3.83ms
944:    learn: 0.9623611    total: 64.6ms   remaining: 3.76ms
945:    learn: 0.9584269    total: 64.7ms   remaining: 3.69ms
946:    learn: 0.9545089    total: 64.7ms   remaining: 3.62ms
947:    learn: 0.9506069    total: 64.8ms   remaining: 3.55ms
948:    learn: 0.9467208    total: 64.8ms   remaining: 3.48ms
949:    learn: 0.9428506    total: 64.9ms   remaining: 3.42ms
950:    learn: 0.9389962    total: 64.9ms   remaining: 3.35ms
951:    learn: 0.9351576    total: 65ms remaining: 3.28ms
952:    learn: 0.9313347    total: 65ms remaining: 3.21ms
953:    learn: 0.9275274    total: 65.1ms   remaining: 3.14ms
954:    learn: 0.9237356    total: 65.2ms   remaining: 3.07ms
955:    learn: 0.9199594    total: 65.2ms   remaining: 3ms
956:    learn: 0.9161986    total: 65.3ms   remaining: 2.93ms
957:    learn: 0.9124532    total: 65.3ms   remaining: 2.86ms
958:    learn: 0.9087231    total: 65.4ms   remaining: 2.79ms
959:    learn: 0.9050082    total: 65.4ms   remaining: 2.73ms
960:    learn: 0.9013086    total: 65.5ms   remaining: 2.66ms
961:    learn: 0.8976240    total: 65.7ms   remaining: 2.59ms
962:    learn: 0.8939545    total: 65.9ms   remaining: 2.53ms
963:    learn: 0.8903000    total: 65.9ms   remaining: 2.46ms
964:    learn: 0.8866605    total: 66ms remaining: 2.39ms
965:    learn: 0.8830358    total: 66ms remaining: 2.32ms
966:    learn: 0.8794260    total: 66.1ms   remaining: 2.25ms
967:    learn: 0.8758309    total: 66.1ms   remaining: 2.19ms
968:    learn: 0.8722505    total: 66.2ms   remaining: 2.12ms
969:    learn: 0.8686847    total: 66.2ms   remaining: 2.05ms
970:    learn: 0.8651335    total: 66.3ms   remaining: 1.98ms
971:    learn: 0.8615969    total: 66.4ms   remaining: 1.91ms
972:    learn: 0.8580747    total: 66.4ms   remaining: 1.84ms
973:    learn: 0.8545669    total: 66.5ms   remaining: 1.77ms
974:    learn: 0.8510734    total: 66.5ms   remaining: 1.71ms
975:    learn: 0.8475942    total: 66.6ms   remaining: 1.64ms
976:    learn: 0.8441292    total: 66.7ms   remaining: 1.57ms
977:    learn: 0.8406784    total: 66.7ms   remaining: 1.5ms
978:    learn: 0.8372417    total: 66.8ms   remaining: 1.43ms
979:    learn: 0.8338191    total: 66.8ms   remaining: 1.36ms
980:    learn: 0.8304104    total: 66.9ms   remaining: 1.29ms
981:    learn: 0.8270157    total: 66.9ms   remaining: 1.23ms
982:    learn: 0.8236349    total: 67ms remaining: 1.16ms
983:    learn: 0.8202679    total: 67.1ms   remaining: 1.09ms
984:    learn: 0.8169146    total: 67.1ms   remaining: 1.02ms
985:    learn: 0.8135751    total: 67.2ms   remaining: 954us
986:    learn: 0.8102492    total: 67.3ms   remaining: 885us
987:    learn: 0.8069369    total: 67.3ms   remaining: 817us
988:    learn: 0.8036381    total: 67.4ms   remaining: 749us
989:    learn: 0.8003528    total: 67.4ms   remaining: 681us
990:    learn: 0.7970810    total: 67.5ms   remaining: 613us
991:    learn: 0.7938225    total: 67.7ms   remaining: 546us
992:    learn: 0.7905774    total: 67.9ms   remaining: 478us
993:    learn: 0.7873455    total: 67.9ms   remaining: 409us
994:    learn: 0.7841268    total: 67.9ms   remaining: 341us
995:    learn: 0.7809213    total: 68ms remaining: 273us
996:    learn: 0.7777289    total: 68ms remaining: 204us
997:    learn: 0.7745496    total: 68.1ms   remaining: 136us
998:    learn: 0.7713832    total: 68.1ms   remaining: 68us
999:    learn: 0.7682298    total: 68.2ms   remaining: 0us

CatBoostRegressor(cat_features=['neighborhood'], loss_function='RMSE')

Installation Guide

# XGBoost
pip install xgboost

# LightGBM
pip install lightgbm

# CatBoost
pip install catboost

# All at once
pip install xgboost lightgbm catboost

Key Takeaways

Gradient boosting builds trees sequentially, each correcting errors of previous trees
Learning rate controls how much each tree contributes (lower = better generalization)
XGBoost, LightGBM, and CatBoost are the three leading implementations
LightGBM is fastest on large datasets
CatBoost excels with categorical features
XGBoost is the most popular and well-documented
Always use early stopping with a validation set
Gradient boosting typically outperforms random forests on structured data
No feature scaling needed for tree-based methods
Trade-off: More complex to tune than random forests, but worth it for accuracy

Practice Problems

Conceptual: Explain why gradient boosting can overfit more easily than random forests, even though both use decision trees.
Applied: Load the Boston housing dataset and compare:
- Random Forest
- XGBoost
- LightGBM
- CatBoost
Use cross-validation and report mean RMSE and training time for each.
Hyperparameter Tuning:
- Start with n_estimators=1000, learning_rate=0.01
- Then try n_estimators=100, learning_rate=0.1
- Compare results. Which works better? Why?
Challenge: Create a dataset with a categorical feature that has 100 unique values. Compare:
- XGBoost with one-hot encoding
- XGBoost with label encoding
- CatBoost with native categorical handling
Which performs best? Why?

Summary

Gradient boosting methods represent the state-of-the-art for structured/tabular data. While they require more careful tuning than random forests, the performance gains are substantial. In practice:

Start with XGBoost for most problems (great documentation, wide adoption)
Switch to LightGBM if you have large datasets or need speed
Use CatBoost if you have many categorical features or want minimal tuning

Understanding these methods is essential for any data scientist working with structured data, and they’re often the first choice for Kaggle competitions and production ML systems.

Next steps: Explore ensemble methods that combine different model types, and dive deeper into hyperparameter optimization techniques like Bayesian optimization.