LinearBoost and SEFRBoost Classifiers

Current release: v0.2.0. This version adds SEFRBoost—gradient boosting with linear SEFR splits at tree nodes—via SEFRBoostClassifier and SEFRBoostRegressor (from linearboost import … or linearboost.sefr_boost). It is pronounced /sɛfr-buːst/. LinearBoost security issues have been updated in this release; upgrade from earlier versions to stay patched.

🧪 Quickstart demos

LinearBoost

Want to see how LinearBoost works in practice?

➡️ Run the demo notebook

This Jupyter notebook shows how to:

• Load and prepare data
• Train LinearBoostClassifier
• Evaluate using F1 score and cross-validation

SEFRBoost

• optuna_sefrboost_demo.ipynb — Hyperparameter search for SEFRBoostClassifier with Optuna on sklearn’s Breast Cancer Wisconsin data: default baseline, then 5-fold stratified CV optimizing F1 (install optuna in addition to linearboost and scikit-learn).

LinearBoost is a boosting classifier built on the linear classifier SEFR: it trains SEFR classifiers in sequence with AdaBoost-style sample reweighting (SAMME or SAMME.R), optionally on kernel features. Training and prediction are fast, and memory use is low.

---

🚀 New in Version 0.2.0

SEFRBoost

SEFRBoost provides binary classification and regression with shallow trees whose internal nodes use SEFR hyperplane splits on pseudo-residuals—an oblique-split GBDT-style alternative that lives alongside LinearBoostClassifier.

from linearboost import SEFRBoostClassifier, SEFRBoostRegressor
# or: from linearboost.sefr_boost import SEFRBoostClassifier, SEFRBoostRegressor

Hands-on examples: notebooks/optuna_sefrboost_demo.ipynb (Optuna tuning).

Security updates

LinearBoost-related security issues are addressed in v0.2.0. Upgrade from older releases.

---

🚀 New in Version 0.1.9

Security Updates

Version 0.1.9 included security updates. For the latest fixes, use v0.2.0 or newer.

---

🚀 New in Version 0.1.7

Gradient Boosting Mode

LinearBoost now supports gradient boosting in addition to AdaBoost via the boosting_type parameter:

• boosting_type='adaboost' (default): Classic AdaBoost (SAMME or SAMME.R) that reweights samples by classification error.
• boosting_type='gradient': Fits each base estimator to pseudo-residuals (negative gradient of log-loss). Often better for highly non-linear or XOR-like patterns and smoother decision boundaries.

# Gradient boosting for complex non-linear patterns
clf = LinearBoostClassifier(
    boosting_type='gradient',
    n_estimators=200,
    kernel='rbf'
)

Class Weighting & Custom Loss

• class_weight: Use 'balanced' or a dict of class weights for imbalanced data. Weights are applied in the boosting loop.
• loss_function: Optional callable (y_true, y_pred, sample_weight) -> float for custom optimization objectives.

clf = LinearBoostClassifier(
    class_weight='balanced',  # Adjust for imbalanced classes
    n_estimators=200
)

Default Algorithm

The default algorithm is now 'SAMME.R' for faster convergence and typically lower test error with fewer iterations (when using boosting_type='adaboost').

---

🚀 New in Version 0.1.5

Version 0.1.5 introduced major architectural improvements designed for scalability, robustness on imbalanced data, and training speed.

⚡ Scalable Kernel Approximation

LinearBoost now supports Kernel Approximation via kernel_approx='rff' or kernel_approx='nystrom'.

Why it matters: Previously, non-linear kernels required computing a full (O(n^2)) kernel matrix, which is memory-intensive for large datasets.

New Capability: You can now map inputs to a lower-dimensional feature space using:

• Random Fourier Features (RFF) — for RBF kernels
• Nyström Approximation — for any kernel type

This enables linear time complexity while retaining non-linear decision boundaries.

# Example: Using kernel approximation for scalable non-linear classification
clf = LinearBoostClassifier(
    kernel='rbf',
    kernel_approx='rff',  # or 'nystrom'
    n_components=256
)

🎯 Stochastic Boosting & Regularization

Advanced regularization techniques to prevent overfitting and reduce variance:

• Subsampling (subsample): Enables Stochastic Gradient Boosting by training each estimator on a random fraction of the training data.
• Shrinkage (shrinkage): Scales the contribution of each new estimator (learning rate decay), effectively "slowing down" learning for better generalization.

clf = LinearBoostClassifier(
    subsample=0.8,    # Use 80% of data per iteration
    shrinkage=0.9     # Scale each estimator's contribution by 0.9
)

⚖️ Optimized for Imbalanced Data

The internal boosting logic has been overhauled to prioritize F1-Score optimization:

• Adaptive Class Weighting: The algorithm dynamically adjusts sample weights based on class frequencies within the boosting loop, aggressively correcting errors on minority classes.
• F1-Based Estimator Weighting: Estimators are rewarded not just for accuracy, but specifically for their F1 performance.

⏱️ Early Stopping

Training can now stop automatically when validation scores plateau:

• Standard validation splits via validation_fraction
• Out-of-Bag (OOB) Evaluation: When using subsampling (subsample < 1.0), LinearBoost utilizes unused samples for validation without reducing training set size.

clf = LinearBoostClassifier(
    n_estimators=500,
    early_stopping=True,
    validation_fraction=0.1,  # 10% held out for validation
    n_iter_no_change=5,       # Stop after 5 iterations with no improvement
    tol=1e-4
)

# Or with OOB evaluation (automatic when subsampling)
clf = LinearBoostClassifier(
    n_estimators=500,
    subsample=0.8,            # Enables OOB evaluation
    early_stopping=True,
    n_iter_no_change=5
)

---

🚀 New Major Release (v0.1.3)

The LinearBoost and SEFR classifiers use kernels to solve non-linear problems. Kernels work by projecting data into a different perspective, allowing a simple linear model to capture complex, curved patterns.

---

Linear Kernel

This is the default setting and performs no transformation. It's the fastest option and works best when the data is already simple and doesn't require complex boundaries.

---

Polynomial Kernel

This kernel creates feature interactions to model curved or complex relationships in the data.

• degree: Sets the complexity of the feature combinations.
• coef0: Balances the influence between high-degree and low-degree feature interactions.

---

RBF Kernel

A powerful and flexible kernel that can handle very complex boundaries. It works by considering the distance between data points, making it a strong general-purpose choice.

• gamma: Controls the reach of a single training point's influence. Small values create smoother boundaries, while large values create more complex ones.

---

Sigmoid Kernel

Inspired by neural networks, this kernel is useful for certain classification tasks that follow a sigmoid shape.

• gamma: Scales the data's influence.
• coef0: Shifts the decision boundary.

🚀 New Major Release (v0.1.2)

Version 0.1.2 of LinearBoost Classifier is released. Here are the changes:

• The codebase is refactored into a new structure.
• SAMME.R algorithm is returned to the classifier.
• Both SEFR and LinearBoostClassifier classes are refactored to fully adhere to Scikit-learn's conventions and API. Now, they are standard Scikit-learn estimators that can be used in Scikit-learn pipelines, grid search, etc.
• Added unit tests (using pytest) to ensure the estimators adhere to Scikit-learn conventions.
• Added fit_intercept parameter to SEFR similar to other linear estimators in Scikit-learn (e.g., LogisticRegression, LinearRegression, etc.).
• Removed random_state parameter from LinearBoostClassifier as it doesn't affect the result, since SEFR doesn't expose a random_state argument. According to Scikit-learn documentation for this parameter in AdaBoostClassifier:

  it is only used when estimator exposes a random_state.

• Added docstring to both SEFR and LinearBoostClassifier classes.
• Used uv for project and package management.
• Used ruff and isort for formatting and lining.
• Added a GitHub workflow (.github/workflows/ci.yml) for CI on PRs.
• Improved Scikit-learn compatibility.

Get Started and Documentation

The documentation is available at https://linearboost.readthedocs.io/.

Recommended Parameters for LinearBoost

Suggested starting values for the main parameters:

• n_estimators: A range of 10 to 200 is suggested, with higher values potentially improving performance at the cost of longer training times. When using early_stopping=True, you can set a higher value (e.g., 500) and let training stop automatically.

• learning_rate: Values between 0.01 and 1 typically perform well. Adjust based on the dataset's complexity and noise.

• algorithm (when boosting_type='adaboost'): Use either SAMME or SAMME.R (default). SAMME.R typically converges faster with lower test error.

  • SAMME: May be better for datasets with clearer separations between classes.
  • SAMME.R: Uses class probabilities; often better for nuanced boundaries.

  Note: As of scikit-learn v1.6, the algorithm parameter is deprecated and will be removed in v1.8. LinearBoostClassifier will only implement the 'SAMME' algorithm in newer versions.

• boosting_type (new in v0.1.7):

  • 'adaboost': Classic AdaBoost (default).
  • 'gradient': Gradient boosting on pseudo-residuals; try for highly non-linear or XOR-like data.

• class_weight (new in v0.1.7): Use 'balanced' for imbalanced datasets so class weights are adjusted automatically.

• scaler: The following scaling methods are recommended based on dataset characteristics:

  • minmax: Best for datasets where features are on different scales but bounded.
  • robust: Effective for datasets with outliers.
  • quantile-uniform: Normalizes features to a uniform distribution.
  • quantile-normal: Normalizes features to a normal (Gaussian) distribution.

• kernel (new in v0.1.3): Choose based on data complexity:

  • linear: Fastest, for linearly separable data.
  • rbf: Most flexible, works well for complex non-linear patterns.
  • poly: For polynomial relationships.
  • sigmoid: For sigmoid-like decision boundaries.

• kernel_approx (new in v0.1.5): For large datasets with non-linear kernels:

  • None: Use full kernel matrix (default, exact but (O(n^2)) memory).
  • 'rff': Random Fourier Features (only with kernel='rbf').
  • 'nystrom': Nyström approximation (works with any kernel).

• subsample (new in v0.1.5): Values in (0, 1] control stochastic boosting. Use 0.8 for variance reduction while maintaining speed.

• shrinkage (new in v0.1.5): Values in (0, 1] scale each estimator's contribution. Use 0.8-0.95 to improve generalization.

• early_stopping (new in v0.1.5): Set to True with n_iter_no_change=5 and tol=1e-4 to automatically stop training when validation performance plateaus.

These parameters should serve as a solid starting point for most datasets. For fine-tuning, consider using hyperparameter optimization tools like Optuna.

Future Developments

These are not yet supported in this current version, but are in the future plans:

• Supporting categorical variables natively
• Adding regression support (LinearBoostRegressor)
• Multi-output classification

Reference Paper

The paper is written by Hamidreza Keshavarz (Independent Researcher based in Berlin, Germany) and Reza Rawassizadeh (Department of Computer Science, Metropolitan college, Boston University, United States). It will be available soon.

License

This project is licensed under the terms of the MIT license. See LICENSE for additional details.

Acknowledgments

Some portions of this code are adapted from the scikit-learn project (https://scikit-learn.org), which is licensed under the BSD 3-Clause License. See the licenses/ folder for details. The modifications and additions made to the original code are licensed under the MIT License © 2025 Hamidreza Keshavarz, Reza Rawassizadeh. The original code from scikit-learn is available at scikit-learn GitHub repository

Special Thanks to:

• Mehdi Samsami – for software engineering, refactoring, and ensuring compatibility.