Better Evaluation Standards

One major improvement in tabular data modeling is the emergence of more rigorous benchmarking. TabArena (leaderboard) and TALENT provide standardized datasets, consistent evaluation metrics, and transparency in preprocessing, which addresses the apples-to-oranges comparison problems from earlier papers. TabArena, which will be the primary reference here, has set a new standard for evaluating tabular DL models with:

• A curated set of tabular datasets for binary/multiclass classification and regression tasks
• Consistent evaluation metrics across different model types
• A public leaderboard tracking model performance
• Transparency in preprocessing and hyperparameter choices
• Focus on heterogeneous datasets that better reflect real-world tabular data scenarios
• Use of Elo-style model ranking for easier model comparisons

These new benchmarks address one of the major frustrations from my previous post summaries: papers often used different datasets, preprocessing pipelines, and hyperparameter tuning budgets, which made fair comparisons difficult at best. They also often focused primarily on very homogeneous data, which isn’t representative of typical real-world tabular data, and sometimes focused only classification settings. This is a far better assessment, and gives us more confidence when comparing methods.

Current Leaderboard Landscape

As of March 2026, the TabArena leaderboard shows some interesting patterns. Here are the ELO rankings for tested models using default, tuned and tuned + ensembled versions of each model type:

The key insight from this is that some newer DL models are not only competitive with boosting, a few typically win in head-to-head comparisons against boosting models. For example, among the tuned + ensemble models the top 4 DL models win > 50% of the time against LGBM, XGB, and CatBost.

One important side note: performance is typically not going to be statistically different for models that are similarly ranked, and papers will often provide something like a critical difference diagram to show this. So while the top 4 DL models are all ranked above the boosting models, that doesn’t necessarily mean they’re statistically better than boosting models.

Take the following image from the TabArena paper based on all datasets. It shows that the top six models, which includes the big 3 boosting models, are not statistically different from each other¹.

Practical Implementation Tools

The even bigger story is accessibility. Modern tools make it trivial to use these methods, so let’s try it out for ourselves!

Using pytabkit

pytabkit provides implementations of state-of-the-art tabular DL models with:

• Optimized hyperparameter defaults based on extensive benchmarking
• Unified API similar to scikit-learn
• Automated preprocessing for heterogeneous data
• Support for both classification and regression

This is significant because one of the historical barriers to using tabular DL was implementation difficulty. While boosting modules were already easy to add to typical ML pipelines and often worked well out of the box, DL models often required custom code, careful tuning, and extensive experimentation to get good results.

Let’s see it in action with a simple example. We’ll generate a synthetic dataset with mixed numeric and categorical features, then train a RealMLP model and compare it to an XGBoost baseline.

from pytabkit import RealMLP_TD_Classifier

from sklearn.model_selection import train_test_split
from sklearn.datasets import make_classification
from sklearn.metrics import accuracy_score, roc_auc_score, log_loss, f1_score
from sklearn.model_selection import StratifiedKFold

import pandas as pd
import numpy as np
import string


# Generate a classification dataset
X, y = make_classification(
    n_samples=5000,
    n_features=15,
    n_informative=8,
    n_redundant=3,
    n_classes=2,
    random_state=42,
    # make it a more difficult problem
    flip_y=0.2,
    class_sep=0.1,
    n_clusters_per_class=2,
)


# Convert to DataFrame for easier handling
feature_names = [f"Feature_{i}" for i in range(X.shape[1])]
X = pd.DataFrame(X, columns=feature_names)

# Add three categorical columns with 3, 5, and 12 categories, using letters as group names
X['Cat_3'] = pd.cut(X['Feature_0'], bins=3, labels=list(string.ascii_uppercase[:3])).astype('category')
X['Cat_5'] = pd.cut(X['Feature_1'], bins=5, labels=list(string.ascii_uppercase[:5])).astype('category')
X['Cat_12'] = pd.cut(X['Feature_2'], bins=12, labels=list(string.ascii_uppercase[:12])).astype('category')


# Use a single train/test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)

# Example CV splitter for pytabkit (if needed)
cv_splitter = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
# Get train/validation indices from the splitter
# X_val, y_val = X_train.iloc[val_idx], y_train[val_idx]

val_idxs_list = [val_idxs for train_idxs, val_idxs in cv_splitter.split(X_train, y_train)]

# make sure that each validation set has the same length, so we can exploit vectorization
max_len = max([len(val_idxs) for val_idxs in val_idxs_list])
val_idxs_list = [val_idxs[:max_len] for val_idxs in val_idxs_list]
val_idxs = np.asarray(val_idxs_list)

print(f"Train size: {X_train.shape[0]}, Test size: {X_test.shape[0]}, Baseline rate: {np.mean(y):.2f}")

With data in hand, we can train a RealMLP model.

model_realmmlp = RealMLP_TD_Classifier(
    device='mps',
    random_state=0,
    n_cv=5,
    val_metric_name='cross_entropy',
    use_ls=False,
    verbosity=2,  # will show per epoch loss
    # example modifications
    # n_refit=1,
    # n_repeats=1,
    # n_epochs=256,
    # batch_size=256,
    # hidden_sizes=[256] * 3,
    # lr=0.04,
)

model_realmmlp.fit(
    X_train,
    y_train,
    val_idxs=val_idxs,
    cat_col_names=['Cat_3', 'Cat_5', 'Cat_12']
)

On an old Macbook M1, this ran just at around one minute with the default 256 epochs, and it doesn’t look like we needed that many (Figure 4). We definitely don’t need to sweat that kind of time on a dataset of this size regardless. Now let’s evaluate the model on the test set. Note that inference is near instant for this model.

y_pred = model_realmmlp.predict(X_test)
y_proba = model_realmmlp.predict_proba(X_test)[:, 1]

metrics_realmlp = {
    "Accuracy": accuracy_score(y_test, y_pred),
    "ROC AUC": roc_auc_score(y_test, y_proba),
    "Log Loss": log_loss(y_test, y_proba),
    "F1 Score": f1_score(y_test, y_pred)
}

metrics_realmlp = pd.DataFrame(metrics_realmlp, index=["RealMLP"])
print(metrics_realmlp)

Now we can compare to XGBoost as the representative of the boosting approach. With pytabkit we can use default parameter settings derived from models used across a variety of data sets (TD = tuned defaults), making it easy to get started very well. Model training with data of this size took just a couple seconds.

from pytabkit import XGB_TD_Classifier

model_xgb = XGB_TD_Classifier(
    device='mps',
    random_state=0,
    n_cv=5
)

model_xgb.fit(
    X_train,
    y_train,
    val_idxs=val_idxs,
    cat_col_names=['Cat_3', 'Cat_5', 'Cat_12']
)

y_pred = model_xgb.predict(X_test)
y_proba = model_xgb.predict_proba(X_test)[:, 1]

metrics_xgb = {
    "Accuracy": accuracy_score(y_test, y_pred),
    "ROC AUC": roc_auc_score(y_test, y_proba),
    "Log Loss": log_loss(y_test, y_proba),
    "F1 Score": f1_score(y_test, y_pred)
}

While boosting performs well, in very little time we can get not only competitive but even better results with a modern DL approach. Of course, this is just one (simulated) dataset that shows how easy it is, but the TabArena results suggest this generalizes well beyond also. There we see that RealMLP is one of the top performers across a variety of datasets, and beats XGBoost in head-to-head comparisons more than 75% of the time (Figure 2).

Even Easier: AutoGluon

AutoGluon is a popular AutoML library that has recently added support for some of the newer tabular DL models, including many of those in pytabkit. This means you can get competitive performance with minimal code and tuning for multiple models, whether boosting, DL, or other approaches. Here we use 3 DL models and 3 boosting models².

from autogluon.tabular import TabularDataset, TabularPredictor

X_train_ag = TabularDataset(X_train.assign(target=y_train))
X_test_ag = TabularDataset(X_test)

model_ag = (
    TabularPredictor(label='target', eval_metric='log_loss')
    .fit(
        X_train_ag,
        num_bag_folds=5,
        # num_bag_sets=1,
        # num_stack_levels=0,
        # you can set these and/or let AutoGluon search 
        hyperparameters={
            'REALTABPFN-V2.5': {'device': 'mps'},
            'REALMLP': {},
            'TABM': {},
            'GBM': {},
            'XGB': {},
            'CAT': {},
        },
        # presets='best_quality', # multiple stacking, typically better performance
                                  # but much longer training time
    )
)

y_pred = model_ag.predict(X_test_ag)
y_proba = model_ag.predict_proba(X_test_ag).loc[:, 1]

One very nice aspect of AutoGluon is its easily accesible ‘leaderboard’ so you can compare and contrast your models immediately. The best non-ensemble model was RealTabPFN-v2.5. It also makes very clear just how much more time it took to get the best performance (in seconds), which is a lot. Whether it’s worth it will depend on your context.

ag_leaderboard = model_ag.leaderboard(X_test_ag.assign(target=y_test))

Now let’s compare all our approaches. Standard XGB with tuned defaults did pretty well, but our RealMLP model did better, and AutoGluon did even better still in terms of loss.

Note that a fairer comparison would be to give the boosting and RealMLP models the same HPO/ensembling treatment, which is just as easy by using the *_HPO_Classifier versions of those models in pytabkit, and would likely improve their performance as well. However, I wanted to show the performance you can get with the viable defaults that would still be quick to train.

The Promise

Recent work has explored pre-training models on large collections of tabular datasets and/or synthetic data, then fine-tuning for specific tasks. This is similar to the paradigm that revolutionized NLP and vision. But do they work?

The idea is that just like in other domains, we can leverage large amounts of tabular data to learn generalizable representations that can be fine-tuned for specific tasks. This could potentially allow us to:

• Achieve better performance with less task-specific data
• Handle heterogeneous features more effectively
• Enable zero-shot or few-shot learning on new tabular tasks

Key models include TabPFN, Mitra (Amazon), TabICL, and others.

The Reality

Results are hit and miss. Some foundation models obviously do well when we look at the leaderboard… if the data is not large, with some even restricted to smaller sizes. Note that TabArena dataset sizes max out at 150k, which is considered ‘medium’ size. Even then, training and inference time may be prohibitive in many cases. Right now you could use MLP approaches like TabM and RealMLP with faster yet comparable results. But more to the point, it is very common in the tabular domain to see much larger datasets, and boosting models are still working fine even with millions of rows, and can utilize GPUs as well.

Previous posts in this series

• This is definitely not all you need (2021)
• Deep Learning for Tabular Data (2022)

Papers and other resources

• TabArena Leaderboard
• TabArena Paper
• pytabkit GitHub
• TALENT Paper
• The State of Tabular Foundation Models

Comparison of some current DL Tabular models

• TFM: Tabular Foundation Model
• MLP: Multi-Layer Perceptron
• Ensemble*: TabM is an efficient imitation of an ensemble of MLPs, but is not itself an ensemble model in the traditional sense.
• Synthetic Data+: Pre-trained on synthetic data, but also augmented with real data during training
• SSL: Self-Supervised Learning

Elo vs. Inference Time

The following shows that when it comes to inference time, boosting holds a clear advantage, but MLP models are also good. This advantage is lessened for ensemble approaches, where DL models also gain more in terms of improvability.

Contributing Factors

Several key factors contribute to prediction uncertainty, among them:

• Amount of Data: While more data can lead us to feel more confident about predictions in a general sense by providing additional learning examples of varying kinds, this is limited. Additional bad or uninformative data will not improve a model or its predictions!

• Model Complexity: More complex models may capture nuanced relationships leading to better predictions, but can also introduce instability in those predictions if additional steps aren’t taken. Conversely, having too simple a model yields relatively poor predictions in general.

• Precision of Measurement: The quality of the data collection process definitely impacts any predictions we might ultimately make. For example, if sales are entered with possible error, or a lag for some reporting units and not for others, this can lead to uncertainty in our predictions of future sales. In addition, the precision of measurement can vary across features. For example, if price is a feature, it may be that price fluctuates within a month but sales are recorded for a month, meaning we might have to take an average or other measure of price.

So any number of things might contribute to the uncertainty in our predictions and, in general, some aspects contributing to prediction uncertainty may be under our control, while others may not be. Different methods produce different results, and some intervals may be too narrow or too wide in some settings, giving false confidence or excessive caution. In what follows, we’ll explore various uncertainty estimation approaches with a focus on conformal prediction, demonstrating its implementation and comparing its advantages to alternative methods. We’ll start by discussing the importance of uncertainty estimation and the different approaches to it. Next, we will discuss conformal prediction and show a simple demonstration of how it can be used to estimate uncertainty. Finally, we will discuss the advantages and disadvantages of conformal prediction compared to other approaches to uncertainty estimation.

Statistical Uncertainty

Probably the most common approach to estimate uncertainty is to use a statistical model. Given a statistical distribution and the estimated parameters of the model, we can get a sense of uncertainty in the prediction. For example, let’s say we have a standard linear regression model. In that case, we are assuming the data generating process for the target/outcome is a normal distribution. The mean for that distribution comes from our model via the linear combination of features, and the variance is estimated separately¹. Given those and other model-specific assumptions hold (e.g. homoscedasticity, independence of observations), we can then obtain an interval estimate for a prediction, typically using a standard formula. This approach extends well beyond both linear regression and the normal distribution, but the basic idea is the same. We assume a distribution for the target, estimate the parameters of that distribution via the data, and then use those estimates to ultimately obtain an interval estimate for our prediction.

Bootstrapping

Bootstrapping is another technique for estimating uncertainty in machine learning models. By repeatedly resampling our original dataset with replacement, we create multiple versions of our data that capture its inherent variability. This process mirrors what would happen if we could collect many different samples from the population. Each resampled dataset produces slightly different model parameters, and consequently, different predictions for the same input. The variation across these predictions directly quantifies our uncertainty - wider spread indicating higher uncertainty, narrower spread suggesting more confidence.

Think of it this way – we take our data, resample the observations with replacement, and train a new model on the resampled data. We then use the new model to make predictions on the original or new/test data. Now we do that many times, taking the average for our final prediction. We then use the quantiles of the prediction distribution (e.g. corresponding to the 5th and 95th percentiles) across the bootstrap samples to get an interval estimate for our prediction. This is a very powerful technique, as it allows us to estimate the uncertainty in our predictions without making any assumptions about the underlying distribution of the data.

Bayesian Approaches

Bootstrap and Bayesian approaches are kindred spirits in that they both generate a distribution for our predictions. The Bayesian approach does not require resampling the data, but it does have an assumption about the distributions of the parameters, which ultimately means it still makes assumptions similar to those in traditional statistical models. We use the Bayesian approach to provide a distribution for the parameters we’re attempting to estimate. For our predictions, we then take random draws from that parameter distribution, feed it into our model to get a prediction, and then repeat that process many times. Again, we may take the average for our final prediction, and the quantiles of interest for our interval estimate.

Conformal Prediction

Conformal prediction is yet another technique for estimating uncertainty in machine learning models, and one of its primary strengths is that it is model agnostic and theoretically can work for any model, from linear regression to deep learning. It is based on the idea that we can estimate the uncertainty in our predictions by looking at the distribution of the predictions from the model, or more specifically, the prediction error. Using the observed error on a calibration set that was not used to train the model, we can order those errors and find the quantile corresponding to the desired uncertainty coverage/error rate². When predicting on new data, we assume it (and its error) comes from a similar distribution as what we’ve seen already in our training/calibration process, with no particular assumption about that distribution. We then use that quantile from our previous distribution to create upper and lower bounds for the new prediction.

from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

def split_conformal(X, y, new_data, alpha = 0.05, seed = 123):
    # Set random seed for reproducibility
    np.random.seed(seed)

    # Splitting the data into training and calibration sets
    train_data, cal_data, train_y, cal_y = train_test_split(X, y, test_size = 0.5)
    N = train_data.shape[0]

    # Train the base model
    model = LinearRegression()
    model.fit(train_data, train_y)
    
    # Calculate residuals on calibration set
    cal_preds = model.predict(cal_data)
    residuals = np.array(np.abs(cal_y - cal_preds))
    
    # Sort residuals and find the quantile corresponding to (1-alpha)
    residuals.sort()
    quantile = np.quantile(residuals, (1 - alpha) * (N / (N + 1)))
    
    # Make predictions on new data and calculate prediction intervals
    preds = model.predict(new_data)
    lower_bounds = preds - quantile
    upper_bounds = preds + quantile
    
    # Return predictions and prediction intervals
    return(
        dict(
            cp_error = quantile, 
            preds = preds, 
            lower_bounds = lower_bounds, 
            upper_bounds = upper_bounds
        )
    )

cp_error_py = split_conformal(X_train, y_train, X_test, alpha = .1)['cp_error']

split_conformal = function(X, y, new_data, alpha = .05, seed = 123) {
    # Set random seed for reproducibility
    set.seed(seed)

    # Splitting the data into training and calibration sets
    idx = sample(1:nrow(X), size = nrow(X) / 2)
    
    train_data = X |> slice(idx)
    cal_data = X |> slice(-idx)
    train_y = y[idx]
    cal_y = y[-idx]

    N = nrow(train_data)

    # Train the base model
    model = lm(train_y ~ ., data = train_data)

    # Calculate residuals on calibration set
    cal_preds = predict(model, newdata = cal_data)
    residuals = abs(cal_y - cal_preds)

    # Sort residuals and find the quantile corresponding to (1-alpha)
    residuals = sort(residuals)
    quantile  = quantile(residuals, (1 - alpha) * (N / (N + 1)))

    # Make predictions on new data and calculate prediction intervals
    preds = predict(model, newdata = new_data)
    lower_bounds = preds - quantile
    upper_bounds = preds + quantile

    # Return predictions and prediction intervals
    return(
        list(
            cp_error = quantile, 
            preds = preds, 
            lower_bounds = lower_bounds, 
            upper_bounds = upper_bounds
        )
    )
}

cp_error_r = split_conformal(X_train, y_train, X_test, alpha = .1)[['cp_error']]

from mapie.regression import MapieRegressor

initial_fit = LinearRegression().fit(X_train, y_train)
model = MapieRegressor(initial_fit, method = 'naive')
y_pred, y_pis = model.fit(X_train, y_train).predict(X_test, alpha = 0.1)

# take the first difference between upper and lower bounds,
# since it's constant for all predictions in this setting
cp_error_mapie = (y_pis[0, 1, 0] - y_pis[0, 0, 0]) / 2  

Traditional Statistical Prediction Intervals

Advantages:

• Ease: For many models, prediction intervals can be computed easily and are automatically provided by various modeling packages.
• Interpretability: The statistical theory behind the prediction intervals is generally straightforward.
• Computational Efficiency: Unlike bootstrapping or Bayesian approaches that require multiple model fits or MCMC sampling, traditional statistical intervals can typically be calculated directly from the fitted model parameters with minimal computational overhead.

Disadvantages:

• Distribution Assumption: They typically require (sometimes strong) assumptions about the underlying data distribution.
• Model-Specific Estimation: The intervals depend on a specific statistical model, and different estimation approaches are needed for different models.
• Model Complexity: Even common models, e.g., those using penalty terms, can make prediction uncertainty using distributional assumptions difficult⁶.

Bootstrapping

Advantages:

• Simple: Bootstrapping is a relatively simple approach to estimating uncertainty.
• Less Restrictive Assumptions: Like conformal prediction, non-parametric bootstrapping doesn’t require assumptions of the underlying data distribution.
• Model-Agnostic: It operates directly on the data irrespective of your model.

Disadvantages:

• Computationally Intensive: It might require a large number of resamples, which can be computationally intensive for some model-data combinations.
• Data issues: For very small datasets, extreme values/tails, estimates could be problematic, but this is likely true for most approaches.
• Naive approach is limited: The naive resampling approach tends to underestimate uncertainty, so additional steps are required to get a more accurate estimate of uncertainty.

Bayesian Approaches

Advantages:

• Probabilistic Interpretation: It provides a full distribution of plausible values for our parameters or predictions.
• Incorporates Prior Information: We can use prior beliefs about parameters we estimate. For example, we can use last year’s data to inform our prior beliefs about the parameters of this year’s model.
• Confidence in Uncertainty Estimate: We can be more confident in our uncertainty estimates than we can with some statistical approaches that often use workarounds to estimate uncertainty (e.g. mixed models), and there are many diagnostic tools available to spot problematic models.

Disadvantages:

• Computationally Intensive: MCMC sampling for model estimation can be slow, particularly for complex models.
• Choice of Prior: The choice of prior can significantly influence the results, especially with small data.
• Statistical Assumptions: As the Bayesian models are an alternative way to estimate statistical models, they depend on a specific statistical model and its likelihood.

Conformal Prediction

Advantages:

• Distribution-Free & Model-Agnostic: Generates valid prediction intervals regardless of underlying data distribution and works with any model, providing a unified framework.
• Theoretical Guarantees: Given a significance level, conformal prediction provides valid coverage even with misspecified models.
• Efficiency: Conformal prediction is relatively computationally efficient compared to other methods, and nonconformity measures based on residuals are straightforward to compute during the training process.
• Generalizable: Other approaches that might be used for uncertainty prediction, like quantile regression or Bayesian methods, can be ‘conformalized’ to produce appropriate coverage.

Disadvantages:

• Implementation Challenges: The intervals can be unstable with small data changes, and are sometimes overly conservative when the nonconformity measure isn’t chosen properly.
• Data Splitting Requirement: A portion of the data needs to be held out to estimate the nonconformity scores, which can reduce the data available for model training.
• Theoretical Limitations: While theoretically sound, various practical implementations (like split-conformal) introduce trade-offs between computational efficiency and theoretical guarantees.
• Exchangeability Assumption: Still requires data exchangeability assumption, which must be accounted for in time series or other structured data.

Goals

To guide our exploration, we have the following goals:

• Explain class imbalance generally
• Review examples and consequences of class imbalance
• Cover some common techniques for addressing the problem
• Investigate the effects of classification thresholds on performance metrics
• Explore a Case Study to see these concepts in action

By the end, you should have a good understanding of the challenges posed by imbalanced data, and how to address them in your own work.

Real-world Examples

Class imbalance is common in many real-world scenarios, such as:

• Fraud detection: As in our opening example, the vast majority of transactions are legitimate, but a small percentage are fraudulent.
• Click-through rate: Most users on a website do not click on ads, but a small percentage do.
• Disease detection: Consider classifying medical images for a rare disease. The vast majority of images may represent normal tissue, while only a small fraction show signs of the disease.
• Spam detection: In email datasets, the majority of emails are legitimate (non-spam), while only a small fraction are spam emails.

As one can see, it’s quite common and natural to have imbalanced data in many real-world applications. So how does it affect our modeling?

Is imbalance a problem?

Is imbalance a problem? Well, not exactly, at least not inherently. The real challenge often arises from its implications, such as the sample size (of the minority class), model limitations, and the metrics we use to evaluate the model.

For example, let’s say you had 10 million observations and 1% of them were in the minority class. Despite the imbalanced classes, you would likely have enough to work with to discover the feature-target relationships, and obtain good estimates of a variety of metrics even after test and validation splits. But with a smaller total sample or even rarer class prevalence, the problems of imbalance can be exacerbated due to the lack of data to work with, making it difficult to learn meaningful patterns or evaluate the model reliably.

In other cases, our model may not produce good predictions for the minority class, even if we have enough data. Maybe the model is not well suited to the problem, or the features aren’t strong enough to make good classifications. In this case, even if your data was balanced you’d potentially still have poor performance, and it will likely be worse with imbalance.

And finally, some metrics, even common ones such as accuracy, can be deceptive in the imbalanced setting. As an example, they may reflect the majority class performance while ignoring the minority class.

Addressing these challenges requires a thoughtful modeling approach and use of performance metrics, which we will explore in the following sections.

The Role of Thresholds in Balancing Trade-offs

As noted previously, when creating a classification label, we use a threshold, or cutoff point, to determine which class to assign to the observation based on predicted probability for that observation. Anything beyond the threshold is classified as the positive class, and anything below is classified as the negative class. So, the choice of threshold directly affects the resulting classifications, and the performance metrics derived from them.

It’s important to note that there is not an inherently right or wrong choice here. For example:

• A lower threshold, which would lead to more positive classifications, may be appropriate when missing positive cases (false negatives) is costly, such as in disease detection³.
• A higher threshold, leading to fewer positive classifications, may be better when false positives are more problematic, such as in fraud detection.

The impact of thresholds highlights the trade-offs inherent in classification tasks, and selecting a threshold that aligns with the specific goals and constraints of your application is critical. Now let’s take a closer look at how thresholds can affect performance metrics. If you feel you already have the gist, you can skip ahead to the case study to see how this plays out in practice.

The Confusion Matrix

Many commonly used metrics such as accuracy, precision, recall, and F1 score, are derived from the confusion matrix. The confusion matrix is the initial summary of the relationship between the predicted and observed classes. However, any metric derived in this way will be sensitive to the classification threshold.

On the other hand, many other metrics derived from predicted probabilities. Examples include the area under a receiver operating curve (AUROC) and Brier score, and these metrics are independent of the classification threshold, and so can provide an additional view of model performance. These metrics are particularly valuable in imbalanced settings, where the threshold-dependent metrics may fail to capture the nuances of model behavior. Let’s now see how this works.

Example: Classification with a .5 Threshold

In the following hypothetical example, the positive rate for the binary target is low - the true average underlying probability of the positive outcome is less than 0.1. Though in practice we’d not know the true probabilities, here we can visualize what their distribution might look like in the following density plot, where the mean probability of the positive class is marked by the line.

Now assume we have a model that predicts probabilities for the target outcome. It may even produce probabilities that correspond well to the true probabilities, as shown in the following plot.

Now, what might a confusion matrix for such data look like with a standard threshold of 0.5? In our hypothetical dataset, we have 100 total for the sample, but there are only five positive observations. Using the default threshold leads to ineffective classification of the minority class, as we only classified two observations as positive, and only one was correct. In this case our precision is 50%, meaning that half of the positive classifications were correct, but our recall is only 20%, meaning we only classified one out of five possible positive observations correctly.

Abbreviations:

• ACC: Accuracy
• TPR: True Positive Rate/Sensitivity/Recall
• TNR: True Negative Rate/Specificity
• PPV: Positive Predictive Value/Precision
• NPV: Negative Predictive Value

Although classifying one observation out of five true positive cases is not something we’d probably be satisfied with, the accuracy score is 95% because almost all the negative observations were correctly classified (true negative rate = 99%). Without any additional context, someone might think 95% accuracy was a good result, but it’s actually no better than guessing, or just labeling all observations as the most common (negative) class, which would have also resulted in 95% accuracy. The other metrics in the confusion matrix like precision (PPV) and recall (TPR) provide the additional context that we need to understand the model’s performance - namely, that we are not doing so well with the positive class, which is typically not a desirable outcome.

Example: Classification with a Lower Threshold

The next plot shows different possible threshold values we could use for the predicted probabilities. With each threshold, we’d potentially create different classifications, resulting in a new confusion matrix and different metrics based on those predicted classes.

Now let’s see what happens to the performance metrics if we try using a lower classification threshold than 0.5, say something closer to ~.1. Here is what the resulting table might look like.

In this setting, the accuracy score has dropped from 95% to 84%, but we really shouldn’t too be concerned with this, as accuracy isn’t usually our focus in the imbalanced setting. Again, we could have guessed and gotten 95% correct. One new result is that the true positive rate (recall) has notably increased - from 20% to 80% - there are now four out of five possible correct positive classifications. This is much better! But looking further, it did come at the cost of a higher false positive rate - ~16% for this result vs. 1% previously⁴. The positive predictive value (PPV) has also dropped from 50% to 21%, meaning that only about one in five of the positive classifications were correct. This is a common trade-off in imbalanced settings, where increasing recall often comes at the cost of precision, and vice versa. In general, this result suggests that the model is now doing a better job of capturing the positive class. But while we gain in some metrics, we’ve lost ground in others. Again, it’s a trade-off.

The following table provides some additional metrics to consider for the two confusion matrices. In particular, we see the F1 score, the balanced accuracy score, the area under a receiver operating curve (AUROC) score. These are common alternatives used in the imbalanced setting.

The F1 score is the harmonic mean of precision (positive predictive value) and recall (true positive rate), and balanced accuracy is the average of the true positive rate and true negative rate, or the average recall for both positive and negative classes. These are less sensitive to the class imbalance than accuracy, but they are still sensitive to the threshold used to create the classifications. In this case, both improve with a lower threshold, though the F1 score only changes slightly.

In contrast, the AUROC score is based on the predicted probabilities, and so is independent of the threshold, which is why it is constant in Table 3⁵. It is a useful metric in this context to consider because it considers the relationship between the true positive rate and true negative rate at many different thresholds, not just the single threshold you happened to select (Google (2025)). As such, this score is often a preferred metric, even in the balanced classification setting.

We’ve now seen a few performance metrics, but there are many available, and we’ll see even more later. Each provides a different facet with which to view your model’s performance. With this in mind, you should decide which aspects of your model’s performance are most important to your use case, and choose your metrics accordingly.

Table Metrics

The first table for logistic and LGBM models shows the following metrics. An arrow (↑ or ↓) indicates whether a higher or lower value is better for the metric.

• Threshold: the threshold value used for classification.
• Mean Pred: the mean predicted probability for the positive class. Ideally this should be close to the observed proportion of positive cases.
• Precision: the proportion of correctly predicted positive cases among all predicted positive cases. ↑
• Recall: the proportion of correctly predicted positive cases among all actual positive cases. ↑
• F1 Score: the harmonic mean of precision and recall. ↑
• F2 Score: a variant of the F1 score that gives more weight to recall. ↑
• FPR: False Positive Rate, the proportion of incorrectly predicted positive cases among all actual negative cases. ↓
• FNR: False Negative Rate, the proportion of incorrectly predicted negative cases among all actual positive cases. ↓
• Total Pos Pred: the total number of positive predictions.
• Threshold Method: the method used to select the classification threshold.

The second table shows the following metrics:

• Threshold: the threshold/cutoff value used for classification.
• Mean Pred: the mean predicted probability for the positive class.
• AUROC: the area under the receiver operating characteristic curve. ↑
• Avg Prec: the average precision (area under the precision-recall curve). ↑
• Brier Score: a measure of the mean squared difference between predicted probabilities and the true outcome. ↓
• ACE: the adaptive calibration error (Nixon et al. (2020)). ↓

Brief Discussion of the Multiclass Setting

While we’ve focused on binary classification, many real-world problems involve more than two classes. Multiclass settings can present additional challenges with imbalanced data, as the relationships between classes become more complex and the interpretation of performance metrics needs adjustment.

In multiclass settings, we often use the same core metrics as in binary classification (precision, recall, F1, etc.), but we need to consider how to aggregate these metrics across classes. Two common approaches are:

Weighted Average: This approach takes into account the number of instances in each class. Each class’s metric is weighted by the proportion of instances in that class. It may be useful when you want to account for class imbalance to ensure that metrics from larger classes have a greater impact on the overall score.

Macro Average: This approach calculates the metric independently for each class and then takes the average of these metrics. Each class is treated equally, regardless of its size. It is useful when you want to ensure that the model performance assessment is consistent across all classes, rather than being dominated by the performance on the more frequent classes.

As in the binary case, the choice between these approaches depends on your specific goals. Use weighted averaging when you want metrics to reflect the natural class distribution, and macro averaging when you want to ensure consistent performance across all classes.

Choosing a Classification Threshold

As we have noted, the default threshold of 0.5 is often used as a classification threshold, but it may not be optimal in many cases, particularly in the imbalanced setting. Each of these and other similar methods have their own advantages and may be suitable for different scenarios. Here are some common methods for selecting a classification threshold:

1. Metric Optimization: Choose a threshold that maximizes or minimizes a specific metric:

   • F-Beta Scores: Adjust the β parameter to weight the relative importance of precision vs. recall. gives equal weight, favors recall, and favors precision.
   • Precision-Recall Trade-off: Select a threshold based on the precision-recall curve to achieve desired balance between these metrics.
   • ROC-based: Choose the threshold that maximizes the geometric mean of sensitivity and specificity.

2. Cost-Sensitive Thresholding: This involves selecting a threshold that minimizes the overall cost of misclassification. This can be done by assigning different weights to false positives and false negatives and choosing the threshold that results in the lowest weighted error. This does not require an algorithmic approach though, as you can just select a threshold based on domain knowledge.

3. Other Statistical Approaches:

   • Youden’s J Statistic: Youden’s J is defined as sensitivity + specificity - 1. This approach maximizes the sum of sensitivity and specificity, and is equivalent to maximizing vertical distance from ROC curve to diagonal.
   • Equal Error Rate (EER): Sets threshold where false positive rate equals false negative rate.

4. Data-driven: Use the observed class proportion (base rate) as the threshold, which may be appropriate when you want predictions to reflect the natural class distribution. This is simple and good for a quick baseline, but may not be optimal in all cases.

For this post, we explored using different thresholds out of curiosity, and here are the results for the LGBM (Basic) models, and only those metrics that are actually affected by the threshold. The different methods explored included optimizing AUROC (ROC), AUPRC (PRC), F.5, F1, and F2 (F*), and simply using the base rate/observed proportion as a cutoff (Obs. Prop.). In the following, the untuned model parameters are identical, and the tuned models are also identical. Within each of those groups, the only difference is the threshold method used. All metrics regard the test set.

In our case study, thresholding that was relatively more focused on precision (F.5/AUPRC) led to a high threshold, few positive predictions, and a low false positive rate. Thresholding that was relatively more focused on recall (F2) led to a notably lower threshold than .5, more positive predictions, and a higher false positive rate. The geometric mean/ROC approach led to the most positive predictions, and thus a notably high false positive rate, but also a lower false negative rate. By coincidence, the ROC and F2 approaches led to the same classification for the tuned model.

We ultimately chose the F2 threshold approach for the reported tables because we were interested in recall, and the ROC approach led to too many positives for the untuned model. However, there’s no reason for this choice other than our personal preference, and the other approaches could be just as valid depending on your own goals for wine classification. Cheers! 🍷

Setup

Now we are ready to prep things for specific use with pytorch. I will not use the same terminology as in Jeremy’s original post, so for us, target = ‘dependent variable’ and X is our feature matrix⁶. Both of these will be pytorch tensors, which for our purposes is just another word for an array of arbitrary size.

from torch import tensor
device = torch.device('cpu')
target = tensor(df_titanic_train.Survived)

dummies = ['Sex_male', 'Sex_female', 'Pclass_1', 'Pclass_2', 'Pclass_3', 'Embarked_C', 'Embarked_Q', 'Embarked_S']
all_features = ['Age', 'SibSp', 'Parch', 'LogFare'] + dummies 

X = df_titanic_train[all_features].apply(pd.to_numeric).astype(float)
X = tensor(X.values, dtype = torch.float)

X.shape

torch.Size([891, 12])

Doing a Gradient Descent Step

We can continue our journey onward to actually estimating the weights rather than specifying them directly, since we definitely don’t want to just keep guessing! This is an iterative process where we still start with an initial (random) guess, then, at each step, refine our guess in a way that lowers our loss. For neural networks we call these steps epochs, and getting our next guess requires calculating what’s called a gradient. Here are some resources for more detail:

• How Does a Neural Net Really Work?: great intro by Jeremy Howard
• Some by-hand code using gradient descent for linear regression, R, Python: By yours truly

In any case, this is basic functionality within pytorch, and it will keep track of each step taken.

coeffs.requires_grad_()

tensor([-0.4629,  0.1386,  0.2409, -0.2262, -0.2632, -0.3147,  0.4876,  0.3136,
         0.2799, -0.4392,  0.2103,  0.3625], requires_grad=True)

loss = calc_loss(X_sc, coeffs, target)
loss

tensor(1.3960, grad_fn=<MseLossBackward0>)

We use backward to calculate the gradients and inspect them.

loss.backward()

coeffs.grad

tensor([-0.9311,  0.6245,  0.4957, -0.7423,  0.6008, -0.6008, -0.9158,  0.0938,
         0.7127, -1.7183,  0.1715,  1.3974])

Each time backward is called, the gradients are added to the previous values. We can see here that they’ve now doubled.

loss = calc_loss(X_sc, coeffs, target)

loss.backward()

coeffs.grad

tensor([-1.8621,  1.2491,  0.9914, -1.4847,  1.2015, -1.2015, -1.8317,  0.1877,
         1.4254, -3.4366,  0.3431,  2.7947])

What we want instead is to set them back to zero after they are used for our estimation step. The following does this.

loss = calc_loss(X_sc, coeffs, target)

loss.backward()

with torch.no_grad():
    coeffs.sub_(coeffs.grad * 0.1)     # sub subtracts in place
    coeffs.grad.zero_()                # zeros out in place
    print(calc_loss(X, coeffs, target))

tensor([-0.1836, -0.0488,  0.0922, -0.0035, -0.4435, -0.1345,  0.7624,  0.2854,
         0.0661,  0.0763,  0.1588, -0.0567], requires_grad=True)
tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
tensor(37.9424)

Training the Linear Model

We typically would split our data into training and test. We can do so here, or keep this data as training and import test.csv for the test set. The latter is actually used for the Kaggle submission, but that’s not a goal here. We’ll use scikit-learn for the splitting.

from sklearn.model_selection import train_test_split

# test size .2 in keeping with fastai RandomSplitter default
train_x, valid_x, train_y, valid_y = train_test_split(
  X_sc, 
  target.float(), 
  test_size = 0.2, 
  random_state = 808
)
  

len(train_x), len(valid_x) # might be one off of the original notebook

(712, 179)

As before, we’ll create functions to help automate our steps:

• one to initialize the weights
• a function to update weights
• one to do a full epoch (using weights to calculate loss, updating weights)
• one to train the entire model (run multiple times/epochs)

As mentioned, the approach here is to create functions that are general enough to take any X or target, so they look a little different from the original notebook. I also add in a verbosity option so you can see the loss at each verbose value epoch (e.g. verbose = 10 means you’ll see the latest loss value every 10 epochs), so you can watch the iterations for as long as you like without it printing constantly (possibly not too big a deal depending on your IDE).

def init_weights(n_wts): 
    return (torch.rand(n_wts) - 0.5).requires_grad_()

def update_weights(weights, lr):
    weights.sub_(weights.grad * lr)
    weights.grad.zero_()

def one_epoch(X, weights, target, lr, verbose = 1, i = 1):
    loss = calc_loss(X, weights, target)
    loss.backward()
    
    with torch.no_grad(): update_weights(weights, lr)
    
    if verbose != 0:
        if i % verbose == 0:
            print(f'{loss: 3f}', end = '\n ')

def train_model(X, target, epochs = 30, lr = 1e-3, verbose = 1):
    torch.manual_seed(442)
    coeffs = init_weights(X.shape[1])
    
    for i in range(epochs): 
        one_epoch(X, coeffs, target, lr = lr, i = i, verbose = verbose)
    return coeffs

Try out the functions if you like (not shown).

calc_loss(X_sc, init_weights(X_sc.shape[1]), target).backward()

one_epoch(train_x, init_weights(train_x.shape[1]), train_y, .01)

Now train the model for multiple epochs. The loss drops very quickly before becoming more steady.

coeffs_est = train_model(train_x, train_y, epochs = 50, verbose = 5, lr = .2)

 1.375618
  0.296216
  0.284019
  0.281221
  0.280271
  0.279923
  0.279794
  0.279746
  0.279728
  0.279721
 

Let’s create a function to show our estimated parameters/weights/coefficients in a pretty fashion.

def show_coeffs(estimates): 
  coef_dict = dict(zip(all_features, estimates.requires_grad_(False).numpy()))
  return pd.DataFrame(coef_dict, index = ['value']).T

show_coeffs(coeffs_est)

               value
Age        -0.090825
SibSp      -0.054449
Parch      -0.016111
LogFare     0.046320
Sex_male   -0.406538
Sex_female -0.171426
Pclass_1    0.408707
Pclass_2    0.335766
Pclass_3    0.329800
Embarked_C  0.057091
Embarked_Q  0.032813
Embarked_S  0.039464

Measuring Accuracy

It’s one thing to get accuracy on the trained data, but a better estimate of model performance is to measure it on our test/validation data. The following function will convert our estimates to a binary value like our target, and compares them to the target. Depending on how you did your training setup, it might be pretty bad or at least better than guessing.

def acc(X, weights, target): 
    return (target.bool() == (calc_preds(X, weights) > 0.5)).float().mean()

acc(train_x, coeffs_est, train_y), acc(valid_x, coeffs_est, valid_y)

(tensor(0.7051), tensor(0.6425))

Using sigmoid

Nothing about the previous setup actually produces a result on the probability scale, so basing a cutoff of .5 is meaningless. you can inspect them and might see values are above 1 or below zero, which we generally don’t want⁸. However we do have a solution. The sigmoid function⁹ allows us to transform our predictions to values between 0 and 1, i.e. probabilities in this context, and in particular, the probability of survival. Then our acc function will be more appropriate, where any probability > .5 will be given a value of 1 (or True technically), while others will be 0/False.

def calc_preds(X, weights):
    return torch.sigmoid((X*weights).sum(axis = 1))

We also will do more iterations, and fiddle with the learning rate (a.k.a. step size)

coeffs_est = train_model(
  train_x,
  train_y,
  epochs = 500,
  lr = 1,
  verbose = 100
)

 0.314158
  0.154329
  0.154237
  0.154232
  0.154232
 

Not too shabby!

acc(train_x, coeffs_est, train_y), acc(valid_x, coeffs_est, valid_y)

(tensor(0.7823), tensor(0.7989))

show_coeffs(coeffs_est)

               value
Age        -0.516476
SibSp      -0.423656
Parch      -0.179623
LogFare     0.396468
Sex_male   -0.927410
Sex_female  0.349448
Pclass_1    0.713895
Pclass_2    0.320935
Pclass_3    0.078919
Embarked_C  0.107378
Embarked_Q  0.082943
Embarked_S -0.036137

In implementing the sigmoid, let’s go ahead and optimize how we calculate the predictions using a matrix shorthand for getting the predictions (which is also much more efficient/faster)¹⁰. To do this, the coefficients will need to be a column vector, so we change our init_coeffs function slightly¹¹.

def calc_preds(X, weights): 
    return torch.sigmoid(X@weights)

def init_coeffs(n_wts): 
    return (torch.rand(n_wts, 1) * 0.1).requires_grad_()

Now our functions are more like the mathematical notation we’d usually see for linear regression.

Compare to Linear/Logistic Regression

Before getting too excited, let’s compare our results to basic linear and logistic regression. The linear regression is more like our model before using the sigmoid transformation, while the logistic is more like when we used it. Depending on your settings, the logistic regression is probably doing better at this point.

from sklearn import linear_model
from sklearn.metrics import accuracy_score


reg = linear_model.LinearRegression()
reg.fit(train_x, train_y)

LinearRegression()

LinearRegression()

acc(valid_x, coeffs_est, valid_y), acc(valid_x, reg.coef_.T, valid_y)

(tensor(0.7989), tensor(0.7821))

reg = linear_model.LogisticRegression()
reg.fit(train_x, train_y)

LogisticRegression()

LogisticRegression()

accuracy_score(valid_y.numpy(), reg.predict(valid_x))

0.7821229050279329

It looks like our coefficient estimates are similar to the logistic regression ones.

show_coeffs(coeffs_est).assign(logreg = np.round(reg.coef_.T, 4))

               value  logreg
Age        -0.516476 -0.4799
SibSp      -0.423656 -0.4191
Parch      -0.179623 -0.1265
LogFare     0.396468  0.3441
Sex_male   -0.927410 -0.6262
Sex_female  0.349448  0.6262
Pclass_1    0.713895  0.3941
Pclass_2    0.320935  0.0675
Pclass_3    0.078919 -0.3945
Embarked_C  0.107378  0.0546
Embarked_Q  0.082943  0.0655
Embarked_S -0.036137 -0.0890

Import & Setup

To get things started we’ll use the tidyverse for some additional data processing, and lubridate for any date processing, for example, converting to weekdays.

# Data Processing

library(tidyverse)
library(lubridate)


# Misc

START_DT = '2008-06-01'
END_DT   = '2018-12-31'
SPLIT_DT = '2017-06-01'

df = data.table::fread('data/time-series/processed_df.csv')

df_start = df %>% 
  as_tibble() %>% 
  select(-contains('_attributes'), -(tsun:wt22)) %>% 
  mutate(
    date      = as_date(date), # remove IDATE class
    rides_log = log(rides),
    year      = year(date),
    year_fac  = factor(year),
    month     = month(date, label = TRUE),
    day       = factor(wday(date, label = TRUE), ordered = FALSE),
    year_day  = lubridate::yday(date),
    line      = factor(line),
    snow_scaled = scale(snow)[, 1],
    colors = as.character(line),
    colors = ifelse(colors == 'purple_express', 'purple4', colors),
    red_line_modernization = 
      ifelse(
        between(date, as_date('2013-05-19'), as_date('2013-10-20')), 
        1, 
        0
      )
  ) %>% 
  arrange(date, line)

Training and Validation

We split our data into training and validation sets, such that everything before 2017-06-01 is used for training, while everything after will be used for testing model performance².

df_train = df_start %>% 
  filter(date < SPLIT_DT, !is.na(rides))

df_validate = df_start %>% 
  filter(date >= SPLIT_DT, !is.na(rides))

red_line_train = df_train %>% 
  filter(line == 'red')

red_line_validate = df_validate %>% 
  filter(line == 'red')

Other

Holidays are available via the prophet package, which we’ll be demonstrating a model with later. The data we’re using already has a ‘holiday vs. not’ variable for simplicity, though it comes from a different source. The prophet version has both the actual date and the observed date counted as a holiday, and I prefer to use both.

holidays = prophet::generated_holidays %>% 
  filter(country == 'US') %>% 
  mutate(ds = as.numeric(as_date(ds))) %>%
  droplevels()

We’ll take a quick look at the red line similar to Cody’s post, so we can feel we have the data processed as we should.

With the data ready to go, we are ready for modeling, so let’s get started!

Intro

Classical times series from an econometrics perspective often considers a error model that accounts for the correlation a current observation has with past observations. A traditional example is the so-called autoregressive, or AR, model, which lets a current observation be predicted by past observations up to a certain point. For example, would could start by just using the last observation to predict the current one. Next we extend this to predict the current based on the previous two observations, and so on. How many lags we use is part of the model exploration.

We can extend this to include not just past observations but also past residuals, called a moving average. So formally, our ARMA (p, q) model now looks like this for an observation at time :

We can also use differencing, for example subtracting the previous time value from the current observation value for all values, to come to the final ARIMA (p, d, q) model. See Hyndman and Athanasopoulos (2021) for more details.

Model

Even base R comes with basic time series models such as this. However, as mentioned, we typically don’t know what to set the values of an ARIMA(p, d, q) to. A quick way to explore this is via the forecast package, which will search over the various hyperparameters and select one based on AIC. Note that fable, a package we will be using later, will also allow such an approach, and if you’d like to go ahead and start using it I show some commented code below.

model_arima = forecast::auto.arima(
  red_line_train$rides_scaled
)

# model_arima = red_line_train %>%
#   select(date, rides_scaled) %>%
#   tsibble::as_tsibble() %>%
#   fabletools::model(fable::ARIMA(
#     rides_scaled ~ 0 + PDQ(0,0,0),
#     stepwise = FALSE,
#     approximation = FALSE
#   ))
# fabletools::report(model_arima)

Explore

In this case we have a selected AR of 3 and MA of 4 for the centered value. But looking at the predictions, we can see this is an almost useless result for any number of days out, and does little better than guessing.

broom::tidy(model_arima)

# plot(acf(residuals(model_arima))) # weekly autocorrelation still exists

red_line_validate %>% 
  slice(1:30)  %>%
  mutate(pred = predict(model_arima, n.ahead = 30)$pred) %>%
  # mutate(pred = forecast(model_arima, h = 30)$.mean) %>%
  ggplot(aes(date, rides_scaled)) +
  geom_line(alpha = .25) +
  geom_line(aes(y = pred), alpha = .25, color = 'darkred')

We’ll use yardstick to help us evaluate performance for this and subsequent models. In this case however, the visualization told us enough- a basic ARIMA isn’t going cut it.

library(yardstick)

# this function will be used for all subsequent models!
metric_score = metric_set(rmse, mae, rsq) 

# validation
data.frame(
  pred = predict(model_arima, newdata = red_line_validate, n.ahead = 30)$pred,
  observed = red_line_validate$rides_scaled[1:30]
) %>%
  metric_score(truth = observed, estimate = pred)

One nice thing about the forecast package is that it can include additional features via the xreg argument, which is exactly what we need- additional information. Now our model looks something like this, where is our model matrix of features and their corresponding regression weights.

Adding these is not exactly straightforward, since it requires a matrix rather than a data frame, but this is not too big a deal once you are used to creating model matrices.

mm = model.matrix(
  ~ . - 1, 
  data = red_line_train %>% 
    select(is_weekend:is_sox_game, tmax_scaled, prcp_scaled, red_line_modernization)
)

model_arima_xreg = forecast::auto.arima(
  red_line_train$rides_scaled,
  max.p = 10,
  max.q = 10,
  xreg  = mm
)

This is looking much better! We can also see how notably different the ARMA structure is relative to the previous model. We also see that adding weekend and holiday effects result in a huge drop in ridership as expected, while baseball games and good weather will lead to an increase.

In the following code, we create a model matrix similar to the training data that we can feed into the predict function. The forecast package also offers a glance method if desired.

nd = red_line_validate %>%
  select(is_weekend:is_sox_game, tmax_scaled, prcp_scaled, red_line_modernization) %>% 
  model.matrix( ~ . - 1, data = .)

preds = predict(model_arima_xreg, newxreg = nd, n.ahead = nrow(red_line_validate))$pred

p_arima_red = red_line_validate %>% 
  mutate(pred = preds) %>% 
  ggplot(aes(date, rides_scaled)) +
  geom_line(alpha = .25) +
  geom_line(aes(y = pred), alpha = .25, color = 'red') +
  labs(x = '', y = 'Rides (scaled)', subtitle = 'ARIMA')

p_arima_red

And here we can see performance is notably improved (restrict to first 30 obs for a direct comparison to the previous).

Intro

More generally, we can think of that original AR error as a random effect, such that after the linear predictor is constructed, we add a random effect based on the correlation structure desired, in this case, autoregressive. In the mixed model setting, it is actually quite common to use an AR residual structure within a cluster or group, and here we can do so as well, as the data is naturally grouped by line.

To make this a bit more clear, we can state the AR effect more formally as follows for a single line at time :

Where t,s are different time points, e.g. within a line.

If we were to simulate it for 4 time points, with autocovariance value of .5, we could do so as follows³.

n_clusters   = 1
n_timepoints = 4
mu  = 0
var = 1  # not actually used if the value is 1
S = .5^as.matrix(dist(1:n_timepoints))

S

z = MASS::mvrnorm(mu = rep(mu, n_timepoints), Sigma = S)

z

And here is our typical model with a single random effect, e.g. for line:

The X may be at either line or observation level, and potentially the could vary by line.

Putting it all together, we’re just adding the AR random effect to the standard mixed model for a single line.

Data Prep

So let’s try this! First some minor data prep to add holidays.

df_train_mixed = df_train %>% 
  mutate(date = as.numeric(date)) %>% 
  left_join(holidays, by = c('date' = 'ds', 'year' = 'year')) %>% 
  mutate(holiday = factor(ifelse(is.na(holiday), '0', as.character(holiday))))

df_validate_mixed = df_validate %>% 
  mutate(date = as.numeric(date)) %>% 
  left_join(holidays, by = c('date' = 'ds', 'year' = 'year')) %>% 
  mutate(holiday = factor(ifelse(is.na(holiday), '0', as.character(holiday))))

Model

For the model, we can now easily think of it as we do other standard modeling scenarios. Along with standard features, we’ll add random effects for line, day, day x line interaction, etc. Finally we also add an AR random effect. For each line, we have an autoregressive structure for days, such that days right next to each other are correlated, and this correlation tapers off as days are further apart. This is not our only option, but seems straightforward to me.

Depending on what you include in the model, you may have convergence issues, so feel free to reduce the complexity if needed. For example, most of the day effect is captured by weekend vs. not, and a by line year trend wasn’t really necessary. In addition, the way the AR random effect variance is estimated as noted above, this essentially captures the line intercept variance.

model_mixed = 
  rides_scaled ~ 
  is_weekend +
  is_cubs_game +
  is_sox_game +
  tmax_scaled + 
  prcp_scaled + 
  snow_scaled +
  # year_day +
  ar1(0 + day|line) +     # the 0 + is a nuance of tmb's approach
  (1|holiday) +           # as RE with all holidays instead of just holiday vs. not
  (1|year) +     
  (1 | red_line_modernization:line) +  # the project shifted ridership from red to other lines
  # (1|day) #+ 
  # (1|line) +
  (1|day:line) #+
  # (1 + year_day|line)

library(glmmTMB)

fit_mixed = glmmTMB(model_mixed, data = df_train_mixed)

Explore

The mixed model approach is nice because it is highly interpretable. We get both standard regression coefficients, and variance components to help us understand how the rest of the variance breaks down. For example, I would interpret the following that that line and weekend are the biggest contributors to the variability seen, and that we have high autocorrelation, as expected.

library(mixedup)

summarise_model(fit_mixed, digits = 4)

extract_cor_structure(fit_mixed, which_cor = 'ar1')

We can visually inspect how well it matches the data. In the following the colored lines are the predictions, while the observed is gray. It looks like performance tapers for more recent time periods, and holiday effects are not as prevalent for some lines (e.g. yellow). The latter could be helped by adding a holiday:line random effect.

library(glmmTMB)

p_mixed = df_validate_mixed %>% 
  droplevels() %>% 
  mutate(pred = predict(fit_mixed, newdata = ., allow.new.levels=TRUE)) %>%
  mutate(date = as_date(date)) %>% 
  ggplot(aes(date, rides_scaled)) +
  geom_line(alpha = .1) +
  geom_line(aes(y = pred, color = I(colors)), alpha = .25) +
  facet_grid(rows = vars(line), scales = 'free_y') +
  labs(x = '', y = 'Rides (scaled)', subtitle = 'Mixed')

p_mixed

As before we can measure performance via yardstick. This model does appears to do very well.

# validation
data.frame(
  pred = predict(fit_mixed, newdata = df_validate_mixed, allow.new.levels = TRUE),
  observed = df_validate_mixed$rides_scaled
) %>%
  metric_score(truth = observed, estimate = pred)

For more on autocorrelation structure in the mixed model setting, see my mixed model document here⁴.

Intro

We can generalize mixed models even further to incorporate nonlinear components, which may include cyclic or other effects. Such models are typically referred to as generalized additive models (GAMs). AR processes themselves can be seen as a special case of gaussian processes, which can potentially be approximated via GAMs. As GAMs can accommodate spatial, temporal, nonlinear, and other effects, they are sometimes more generally referred to as structured additive regression models, or STARs.

Data Prep

The data prep for the GAM is the same as with the mixed model, so we’ll just use that data.

df_train_gam = df_train_mixed

df_validate_gam = df_validate_mixed

Model

With data in place we are ready to conduct the model. We have numerous options for how we’d like to take this. However, as an example, I tried various smooths, but didn’t really see much difference, which is actually a good thing. For any further improvements we’d likely have to tweak the core model itself. I also use bam for a quicker result, but this isn’t really necessary, as it didn’t even take a minute to run with standard gam. As with the mixed model, we will use holiday as a random effect, but we add the holiday by line interaction since we saw that need. In addition, our year-day by line interaction should help some with the tailing off of more recent predictions.

library(mgcv)

# for year, use year (numeric) or use year_fac, but for latter, it will not be
# able to predict any year not in the training data unless you use
# drop.unused.levels.
model_gam = 
  rides_scaled ~ 
  is_weekend +
  is_cubs_game +
  is_sox_game +
  s(tmax_scaled) + 
  s(prcp_scaled) + 
  s(snow_scaled) +
  s(red_line_modernization, line, bs = 're') +
  s(holiday,  bs = 're') +
  s(holiday, line,  bs = 're') +
  s(year_fac, bs = 're') +      
  s(day,  bs = 're') + 
  s(line, bs = 're') + 
  s(line, day, bs = 're') + 
  s(year_day, by = line, bs = c('ds', 'fs'))


# will take a while!
# fit_gam = gam(
#   model_gam, 
#   data     = df_train_gam,
#   drop.unused.levels = FALSE, 
#   method   = "REML"
# )

# fast even without parallel
fit_gam = bam(
  model_gam, 
  data     = df_train_gam,
  drop.unused.levels = FALSE, 
  method   = "fREML",
  discrete = TRUE     # will fit the model in a second rather than a couple seconds
  # nthreads = 8,     # this option assumes a cluster is available. not necessary for this.
)

Explore

As with glmmTMB, I use a custom function to summarize the model, or extract different components from it. From the initial glance we can see things that we expect (e.g. line and weekend effects are large).

mixedup::summarise_model(fit_gam)

Variance Components:

                       Group                 Effect Variance   SD SD_2.5
                 tmax_scaled              Intercept     0.01 0.09   0.04
                 prcp_scaled              Intercept     0.00 0.01   0.00
                 snow_scaled              Intercept     0.00 0.01   0.00
                        line red_line_modernization     0.09 0.31   0.19
                     holiday              Intercept     0.05 0.22   0.14
                     holiday                   line     0.04 0.21   0.18
                    year_fac              Intercept     0.00 0.06   0.04
                         day              Intercept     0.00 0.00   0.00
                        line              Intercept     0.49 0.70   0.42
                        line                    day     0.05 0.22   0.18
           year_day:lineblue              Intercept     0.00 0.00   0.00
          year_day:linebrown              Intercept     0.00 0.00   0.00
          year_day:linegreen              Intercept     0.00 0.00   0.00
         year_day:lineorange              Intercept     0.00 0.00   0.00
           year_day:linepink              Intercept     0.00 0.00   0.00
         year_day:linepurple              Intercept     0.00 0.00   0.00
 year_day:linepurple_express              Intercept     0.00 0.00   0.00
            year_day:linered              Intercept     0.00 0.00   0.00
         year_day:lineyellow              Intercept     0.00 0.00   0.00
                    Residual                     NA     0.02 0.13   0.13
      SD_97.5 Var_prop
 2.100000e-01     0.01
 2.000000e-02     0.00
 2.000000e-02     0.00
 4.900000e-01     0.13
 3.300000e-01     0.06
 2.400000e-01     0.06
 1.000000e-01     0.01
 4.162009e+69     0.00
 1.150000e+00     0.65
 2.700000e-01     0.06
 0.000000e+00     0.00
 0.000000e+00     0.00
 0.000000e+00     0.00
 0.000000e+00     0.00
 0.000000e+00     0.00
 0.000000e+00     0.00
 0.000000e+00     0.00
 0.000000e+00     0.00
 0.000000e+00     0.00
 1.300000e-01     0.02

Fixed Effects:

         Term Value   SE     t P_value Lower_2.5 Upper_97.5
    Intercept -0.28 0.24 -1.14    0.25     -0.76       0.20
   is_weekend -0.54 0.06 -8.27    0.00     -0.66      -0.41
 is_cubs_game  0.04 0.00 13.81    0.00      0.03       0.04
  is_sox_game  0.01 0.00  3.81    0.00      0.00       0.02

Now we can visualize test predictions broken about by line. The greater flexibility of the GAM for fixed and other effects allows it to follow the trends more easily than the standard linear mixed model approach.

We can also see improvement over our standard mixed model approach, and our best performance yet.

Intro

Prophet is an approach from Facebook that uses Stan to estimate a time series model taking various trends, seasonality, and other factors under consideration. By default, it only uses Stan for optimization (e.g. via ‘BFGS’), but you can switch to fully Bayesian if desired, and take advantage of all that the Bayesian approach has to offer.

Data Prep

The prophet package in R takes some getting used to. We have to have specific names for our variables, and unfortunately have to do extra work to incorporate categorical features. We can use recipes (like yardstick, part of the tidymodels ’verse) to set up the data (e.g. one-hot encoding).

library(prophet)
library(recipes)

df_train_prophet = df_train %>% 
  arrange(date, line) %>% 
  rename(y  = rides_scaled,
         ds = date)

rec = recipes::recipe(~., df_train_prophet)

df_train_prophet = rec %>% 
  step_dummy(line, one_hot = TRUE, keep_original_cols = TRUE) %>% 
  prep(training = df_train_prophet) %>% 
  bake(new_data = df_train_prophet) %>% 
  rename_with(.cols = starts_with('line_'), str_remove, 'line_')

df_validate_prophet = df_validate %>% 
  arrange(date, line)%>%
  rename(ds = date, y = rides_scaled)

rec = recipe(~., df_validate_prophet)

df_validate_prophet = rec %>% 
  step_dummy(line, one_hot = TRUE, keep_original_cols = TRUE) %>% 
  prep(training = df_train_prophet) %>% 
  bake(new_data = df_validate_prophet) %>% 
  rename_with(.cols = starts_with('line_'), str_remove, 'line_')

Model

With data in place, we are ready to build the model. Note that later we will compare results to fable.prophet, which will mask some of the functions here, or vice versa depending on which you load first. I would suggest only doing the prophet model, or only doing the fable model, rather than trying to do both at the same time, to avoid any mix-up.

After setting up the model, you have to add additional features in separate steps. Prophet has a nice way to incorporate holidays though. When you run this model, you may have to wait for a minute or so.

# use prophet::prophet in case you have fable.prophet loaded also
model_prophet = prophet::prophet(
  holidays = generated_holidays %>% filter(country == 'US'),
  yearly.seasonality = FALSE,
  seasonality.mode = "multiplicative",
  changepoint.prior.scale = .5
)

line_names = c(
  'blue',
  'brown',
  'green',
  'orange',
  'pink',
  'purple',
  'purple_express',
  'red',
  'yellow'
)

predictors = c(
  'is_weekend',
  'is_cubs_game',
  'is_sox_game',
  # 'is_holiday',
  'tmax_scaled',
  'prcp_scaled',
  'snow_scaled',
  line_names
)

for (i in predictors) {
  model_prophet = add_regressor(model_prophet, i, standardize = FALSE, mode = 'additive')
}

model_prophet = add_country_holidays(model_prophet, country_name = 'US')

fit_prophet = fit.prophet(model_prophet, df = df_train_prophet)

forecast = predict(fit_prophet, df_validate_prophet)

Explore

We now visualize predictions as we did with the GAM. But one of the nice things with prophet is that you can plot the various parts of the model results via the plot method or prophet_plot_components (not shown). Unfortunately, our baseline effort seems to undersmooth our more popular lines (blue, red), and overreacts to some of the others (purple, yellow). This is because it’s not really changing the predictions according to line.

We can also assess performance as before, but note that prophet has it’s own cross-validation capabilities which would be better to utilize if this was your primary tool. Between the previous visualization and these metrics, our first stab doesn’t appear to do as well as the GAM, so you might like to go back and tweak things.

Intro

I came across fable.prophet as a possibly easier way to engage prophet. It is an extension of fable and related packages, which are very useful for time series processing and analysis. Note that it is 0.1.0 version development, and hasn’t had much done with it in the past year, so your mileage may vary with regard to utility by the time you read this⁵. But with it we can specify the model in more of an R fashion, and we now don’t have as much data pre-processing.

Data Prep

One key difference using fable.prophet is that it works with tsibble objects, and thus must have unique date observations. We can do this by setting line as the key⁶.

library(fable.prophet)

df_train_fable = df_train_prophet %>% 
  as_tsibble(index = ds, key = line)

df_validate_fable = df_validate_prophet %>% 
  as_tsibble(index = ds, key = line)

holidays_fable = holidays %>% 
  filter(country == 'US') %>% 
  mutate(ds = as_date(ds)) %>% 
  as_tsibble()

Model

Beyond this we use functions within our formula to set the various components. With line as the key, fable is actually running separate prophet models for each line, and we can do so in parallel if desired.

model_prophet = fable.prophet::prophet(
  y ~ 
    growth('linear', changepoint_prior_scale = 0.5) +
    season("week", type = "multiplicative") +
    holiday(holidays_fable) +
    xreg(
      is_weekend,
      is_cubs_game,
      is_sox_game,
      # is_holiday,
      tmax_scaled,
      prcp_scaled,
      snow_scaled
    ) 
)

# library(future)
# plan(multisession)

# furrr is used under the hood, and though it wants a seed, it doesn't
# automatically use one so will give warnings. I don't think it can be passed
# via the model function, so expect to see ignorable warnings (suppressed here).

fit_fable = model(df_train_fable, mdl = model_prophet)

forecast_fable = fit_fable %>% 
  forecast(df_validate_fable) 

# plan(sequential)

Explore

With fable.prophet visualization, we have the more automatic plots, but again we’ll stick with the basic validation plot we’ve been doing.

components(fit_fable)
components(fit_fable) %>%
  autoplot()
forecast_fable %>%
  autoplot(level = 95, color = '#ff5500')

This model does well, and better than basic prophet, but we can see its limitations, for example, with the yellow line, and more recent ridership in general.

And we check performance as before. The fable model is doing almost as well as our GAM approach did.

One nice thing about the fable approach is its internal performance metrics, which are easily obtained. It will give us results for each line⁷, validation data results shown. We see that we have more error for the popular lines as before, but in terms of percentage error, the other lines are showing more difficulty. You can find out more about the additional metrics available here.

accuracy(fit_fable)
accuracy(forecast_fable, df_validate_fable)

The fable results suggests what we already knew from our GAM and mixed model approach, that interactions of the series with line are important. We weren’t easily able to do this with the default prophet (it would likely require adding time x line interaction regresssors), so it’s nice that we have the option here.

Intro

This part is actually new from when I first wrote up this post over a year ago. I basically wanted to test if a boosting approach would work decently out of the box without doing anything special regarding the structure of the data. I don’t add it to the summary visualizations, but provide the standard results here.

Data Prep

I’ll use lightgbm which I’ve been increasingly using of late. It requires a matrix object for input, and so some additional processing as well.

library(lightgbm)

# lightgbm only accepts a matrix input
df_train_lgb_init = df_train %>% 
  select(
    rides_scaled,
    is_weekday,
    is_holiday,
    is_cubs_game,
    is_sox_game,
    tmax_scaled,
    prcp_scaled,
    line, 
    red_line_modernization,
    year,
    month,
    year_day
  ) %>% 
  mutate(
    line = as.integer(line),
    month = as.integer(month)
  )

X_train = as.matrix(df_train_lgb_init %>% select(-rides_scaled))

X_test  = df_validate %>% 
  select(
    is_weekday,
    is_holiday,
    is_cubs_game,
    is_sox_game,
    tmax_scaled,
    prcp_scaled,
    line, 
    red_line_modernization,
    year,
    month,
    year_day
  ) %>% 
  mutate(
    line = as.integer(line),
    month = as.integer(month)
  ) %>% 
  as.matrix()


df_train_lgb = lgb.Dataset(
  X_train,
  label = df_train_lgb_init$rides_scaled,
  categorical_feature = c(
    'is_weekday',
    'is_holiday',
    'is_cubs_game',
    'is_sox_game',
    'line',
    'year',
    'month'
  )
)

df_test_lgb  = lgb.Dataset.create.valid(
  df_train_lgb, 
  X_test,
  label = df_validate$rides_scaled
)

Model

Typically we would perform some sort of search over the (many) parameters available to tweak with lightgbm, like the number of trees, learning rate, regularizer parameters and more. I ignore that, but I did fiddle with the learning rate and bumped up the nrounds (trees), but that’s it.

params = list(
  objective       = "regression"
  , metric        = "l2"
  , min_data      = 10L
  , learning_rate = .01
)

fit_gbm = lgb.train(
  params    = params
  , data    = df_train_lgb
  , nrounds = 2000L
)

Explore

Some may be surprised at how well this does, but regular users of boosting probably are not. We didn’t have to do much and it’s already the best performing model.

Data

As usual we’ll need some data prep, both for initial training-test split creation, but also specifically for usage with Torch.

library(tsibble)
library(lubridate)
library(torch)


df_train_torch = df_train %>%
  filter(line == 'red', year < 2017) %>%
  pull(rides_scaled) %>%
  as.matrix()

df_validate_torch = df_validate %>%
  filter(line == 'red', year >= 2017) %>%
  pull(rides_scaled) %>%
  as.matrix()

df_test_torch = df_validate %>%
  filter(line == 'red', date > '2017-12-24') %>%
  pull(rides_scaled) %>%
  as.matrix()

train_mean = mean(df_train_torch)
train_sd   = sd(df_train_torch)

Torch data

For our data, we will use a week behind lag to predict the following week. This seems appropriate for this problem, but for any particular time series problem we’d want to probably think hard about this and/or test different settings.

n_timesteps = 7    # we use a week instead of 14 days in original blog
n_forecast  = 7    # look ahead one week

cta_dataset <- dataset(
  name = "cta_dataset",

  initialize = function(x, n_timesteps, sample_frac = 1) {

    self$n_timesteps <- n_timesteps
    self$x <- torch_tensor((x - train_mean) / train_sd)

    n <- length(self$x) - self$n_timesteps - 1

    self$starts <- sort(sample.int(
      n = n,
      size = n * sample_frac
    ))

  },

  .getitem = function(i) {

    start <- self$starts[i]
    end <- start + self$n_timesteps - 1
    lag <- 1

    list(
      x = self$x[start:end],
      y = self$x[(start+lag):(end+lag)]$squeeze(2)
    )

  },

  .length = function() {
    length(self$starts)
  }
)

batch_size = 32

train_ds = cta_dataset(df_train_torch, n_timesteps)
train_dl = dataloader(train_ds, batch_size = batch_size, shuffle = TRUE)

valid_ds = cta_dataset(df_validate_torch, n_timesteps)
valid_dl = dataloader(valid_ds, batch_size = batch_size)

test_ds  = cta_dataset(df_test_torch, n_timesteps)
test_dl  = dataloader(test_ds, batch_size = 1)

Model

I leave it to the blog for details, but briefly, there are four components to the model:

• Encoder: takes input, and produces outputs and states via RNN
• Decoder: takes the last predicted value as input and current context to make a new prediction
• Seq2Seq: essentially encodes once, and calls the decoder in a loop
• Attention: allows output from the encoder at a specific time point to provide ‘context’ for the decoder

net =
  seq2seq_module(
    "gru",
    input_size     = 1,
    hidden_size    = 32,
    attention_type = "multiplicative",
    attention_size = 8,
    n_forecast     = n_forecast
  )

b = dataloader_make_iter(train_dl) %>% dataloader_next()

net(b$x, b$y, teacher_forcing_ratio = 1)

Training

With data in place, we’re ready to train the model. For the most part, not much is going on here that would be different from other deep learning situations, e.g. choosing an optimizer, number of epochs, etc. We’ll use mean squared error as our loss, and I create an object to store the validation loss over the epochs of training. I played around with it a bit, and you’re probably not going to see much improvement after letting it go for 100 epochs.

optimizer = optim_adam(net$parameters, lr = 0.001)

num_epochs = 100

train_batch <- function(b, teacher_forcing_ratio) {

  optimizer$zero_grad()
  output <- net(b$x, b$y, teacher_forcing_ratio)
  target <- b$y

  loss <- nnf_mse_loss(output, target[ , 1:(dim(output)[2])])
  loss$backward()
  optimizer$step()

  loss$item()

}

valid_batch <- function(b, teacher_forcing_ratio = 0) {

  output <- net(b$x, b$y, teacher_forcing_ratio)
  target <- b$y

  loss <- nnf_mse_loss(output, target[ , 1:(dim(output)[2])])

  loss$item()

}


all_valid_loss = c()

for (epoch in 1:num_epochs) {

  net$train()
  train_loss <- c()

  coro::loop(for (b in train_dl) {
    loss <- train_batch(b, teacher_forcing_ratio = 0.0)
    train_loss <- c(train_loss, loss)
  })

  cat(sprintf("\nEpoch %d, training: loss: %3.5f \n", epoch, mean(train_loss)))

  net$eval()
  valid_loss <- c()

  coro::loop(for (b in valid_dl) {
    loss <- valid_batch(b)
    valid_loss <- c(valid_loss, loss)
  })
  
  all_valid_loss = c(all_valid_loss, mean(valid_loss))

  cat(sprintf("\nEpoch %d, validation: loss: %3.5f \n", epoch, mean(valid_loss)))
}

Evaluations

net$eval()

test_preds = vector(mode = "list", length = length(test_dl))

i = 1

coro::loop(for (b in test_dl) {

  if (i %% 100 == 0)
    print(i)

  output <- net(b$x, b$y, teacher_forcing_ratio = 0)
  preds <- as.numeric(output)

  test_preds[[i]] <- preds
  i <<- i + 1
})

For this visualization, we do things a little different. In our current setup, we have 7 timesteps predicting 7 day windows. We started our test set at the beginning of December so that the first prediction is January first, and then proceeds accordingly.

# same as test
df_eval_torch = df_validate %>%
  filter(line == 'red', date > '2017-12-01') %>%
  select(rides_scaled, date) %>%
  as_tsibble()

test_preds_plot = vector(mode = "list", length = length(test_preds))

for (i in 1:(length(test_preds_plot)-  n_forecast)) {
  test_preds_plot[[i]] =
    data.frame(
      pred = c(
        rep(NA, n_timesteps + (i - 1)),
        test_preds[[i]] * train_sd + train_mean,
        rep(NA, nrow(df_eval_torch) - (i - 1) - n_timesteps - n_forecast)
      )
    )
}

df_eval_torch_plot0 =
  bind_cols(df_eval_torch, bind_cols(test_preds_plot))

A visualization of the predictions makes this more clear. Each 7 day segment is making predictions for the next 7 days. The following predictions are for the last two months, with each column a set of 7 predictions for that time point.

So for our red line plot, we’ll just use the average prediction at each date to make it comparable to the other plots. In general it looks to be doing okay, even armed with no contextual information. Certainly better than the base ARIMA plot.

However, we can see that there is much information lost just adhering to the series alone.