Training a Model

In the last notebook, we learned how to write stock indicators in PyBroker. Indicators are a good starting point for developing a trading strategy. But to create a successful strategy, it is likely that a more sophisticated approach using predictive modeling will be needed.

One of the main features of PyBroker is the ability to train and backtest machine learning models. These models can utilize indicators as features to make more accurate predictions about market movements. Once trained, these models can be backtested using a popular technique known as Walkforward Analysis, which simulates how a strategy would perform during actual trading.

We’ll explain Walkforward Analysis more in depth later in this notebook. But first, let’s get started with some needed imports:

import numpy as np
import pybroker
from numba import njit
from pybroker import Strategy, StrategyConfig, YFinance

As with DataSource and Indicator data, PyBroker can also cache trained models to disk. You can enable caching for all three by calling pybroker.enable_caches:

pybroker.enable_caches("walkforward_strategy")

In the last notebook, we implemented an indicator that calculates the close-minus-moving-average (CMMA) using NumPy and Numba. Here’s the code for the CMMA indicator again:

def cmma(bar_data, lookback):

    @njit  # Enable Numba JIT.
    def vec_cmma(values):
        # Initialize the result array.
        n = len(values)
        out = np.array([np.nan for _ in range(n)])

        # For all bars starting at lookback:
        for i in range(lookback, n):
            # Calculate the moving average for the lookback.
            ma = 0
            for j in range(i - lookback, i):
                ma += values[j]
            ma /= lookback
            # Subtract the moving average from value.
            out[i] = values[i] - ma
        return out

    # Calculate for close prices.
    return vec_cmma(bar_data.close)


cmma_20 = pybroker.indicator("cmma_20", cmma, lookback=20)

Train and Backtest

Next, we want to build a model that predicts the next day’s return using the 20-day CMMA. Using simple linear regression is a good approach to begin experimenting with as a baseline model. Please note that this is just a basic exercise. Standard linear regression is generally not ideal for forecasting financial returns, so I have skipped the usual assumption checks (like normality and homoscedasticity) for the sake of brevity. You should always verify these assumptions for your own strategies.

Below we import a LinearRegression model from scikit-learn:

from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score

We create a train_slr function to train the LinearRegression model:

def train_slr(symbol, train_data, test_data):
    # Train
    # Previous day close prices.
    train_prev_close = train_data["close"].shift(1)
    # Calculate daily returns.
    train_daily_returns = (
        train_data["close"] - train_prev_close
    ) / train_prev_close
    # Predict next day's return.
    train_data["pred"] = train_daily_returns.shift(-1)
    train_data = train_data.dropna()
    # Train the LinearRegession model to predict the next day's return
    # given the 20-day CMMA.
    X_train = train_data[["cmma_20"]]
    y_train = train_data[["pred"]]
    model = LinearRegression()
    model.fit(X_train, y_train)

    # Test
    test_prev_close = test_data["close"].shift(1)
    test_daily_returns = (
        test_data["close"] - test_prev_close
    ) / test_prev_close
    test_data["pred"] = test_daily_returns.shift(-1)
    test_data = test_data.dropna()
    X_test = test_data[["cmma_20"]]
    y_test = test_data[["pred"]]
    # Make predictions from test data.
    y_pred = model.predict(X_test)
    # Print goodness of fit.
    r2 = r2_score(y_test, np.squeeze(y_pred))
    print(symbol, f"R^2={r2}")

    # Return the trained model and columns to use as input data.
    return model, ["cmma_20"]

The train_slr function uses the 20-day CMMA as the input feature, or predictor, for the LinearRegression model. The function then fits the LinearRegression model to the training data for that stock symbol.

After fitting the model, the function uses the testing data to evaluate the model’s accuracy, specifically by computing the R-squared score. The R-squared score provides a measure of how well the LinearRegression model fits the testing data.

The final output of the train_slr function is the trained LinearRegression model specifically for that stock symbol, along with the cmma_20 column, which is to be used as input data when making predictions. PyBroker will use this model to predict the next day’s return of the stock during the backtest. The train_slr function will be called for each stock symbol, and the trained models will be used to predict the next day’s return for each individual stock.

Once the function to train the model has been defined, it needs to be registered with PyBroker. This is done by creating a new ModelSource instance using the pybroker.model function. The arguments to this function are the name of the model ('slr' in this case), the function that will train the model (train_slr), and a list of indicators to use as inputs for the model (in this case, cmma_20).

model_slr = pybroker.model("slr", train_slr, indicators=[cmma_20])

To create a trading strategy that uses the trained model, create a new Strategy object using the YFinance data source, and specify the start and end dates for the backtest period:

config = StrategyConfig()
strategy = Strategy(YFinance(), "3/1/2017", "3/1/2022", config)
strategy.add_execution(None, ["NVDA", "AMD"], models=model_slr)

The add_execution method is then called on the Strategy object to specify the details of the trading execution. In this case, a None value is passed as the first argument, which means that no trading function will be used during the backtest.

The last step is to run the backtest by calling the backtest method on the Strategy object. Set the train_size to 0.5 to specify that the model should be trained on the first half of the data and tested on the second half.

strategy.backtest(train_size=0.5)

Backtesting: 2017-03-01 00:00:00 to 2022-03-01 00:00:00

Loading bar data...

[*********************100%***********************]  2 of 2 completed

Loaded bar data: 0:00:00

Computing indicators...

100% (2 of 2) |##########################| Elapsed Time: 0:00:00 Time:  0:00:00

Train split: 2017-03-02 00:00:00 to 2019-08-29 00:00:00
AMD R^2=-0.006951269993268561
NVDA R^2=-0.004188579889899735
Finished training models: 0:00:00

Finished backtest: 0:00:01

TestResult(start_date=datetime.datetime(2017, 3, 1, 0, 0), end_date=datetime.datetime(2022, 3, 1, 0, 0), portfolio=Empty DataFrame
Columns: []
Index: [], positions=Empty DataFrame
Columns: []
Index: [], orders=Empty DataFrame
Columns: []
Index: [], trades=Empty DataFrame
Columns: []
Index: [], metrics=EvalMetrics(trade_count=0, initial_market_value=0, end_market_value=0, total_pnl=0, unrealized_pnl=0, total_return_pct=0, annual_return_pct=None, total_profit=0, total_loss=0, total_fees=0, max_drawdown=0, max_drawdown_pct=0, max_drawdown_date=None, win_rate=0, loss_rate=0, winning_trades=0, losing_trades=0, avg_pnl=0, avg_return_pct=0, avg_trade_bars=0, avg_profit=0, avg_profit_pct=0, avg_winning_trade_bars=0, avg_loss=0, avg_loss_pct=0, avg_losing_trade_bars=0, largest_win=0, largest_win_pct=0, largest_win_bars=0, largest_loss=0, largest_loss_pct=0, largest_loss_bars=0, max_wins=0, max_losses=0, sharpe=0, sortino=0, calmar=None, profit_factor=0, ulcer_index=0, upi=0, equity_r2=0, std_error=0, annual_std_error=None, annual_volatility_pct=None), metrics_df=Empty DataFrame
Columns: []
Index: [], bootstrap=None, signals=None, stops=None, symbols=frozenset({'NVDA', 'AMD'}))

Walkforward Analysis

PyBroker employs a powerful algorithm known as Walkforward Analysis to perform backtesting. The algorithm partitions the backtest data into a fixed number of time windows, each containing a train-test split of data.

The Walkforward Analysis then proceeds to “walk forward” in time, in the same manner that a trading strategy would be executed in the real world. The model is first trained on the earliest window and then evaluated on the test data in that window.

As the algorithm moves forward to evaluate the next window in time, the test data from the previous window is added to the training data. This process continues until all of the time windows are evaluated.

By using this approach, the Walkforward Analysis is able to simulate the real-world performance of a trading strategy, and produce more reliable and accurate backtesting results.

Let’s consider a trading strategy that generates buy and sell signals from the LinearRegression model that we trained earlier. The strategy is implemented as the hold_long function:

def hold_long(ctx):
    if not ctx.long_pos():
        # Buy if the next bar is predicted to have a positive return:
        if ctx.preds("slr")[-1] > 0:
            ctx.buy_shares = 100
    else:
        # Sell if the next bar is predicted to have a negative return:
        if ctx.preds("slr")[-1] < 0:
            ctx.sell_shares = 100


strategy.clear_executions()
strategy.add_execution(hold_long, ["NVDA", "AMD"], models=model_slr)

The hold_long function opens a long position when the model predicts a positive return for the next bar, and then closes the position when the model predicts a negative return.

The ctx.preds(‘slr’) method is used to access the predictions made by the 'slr' model for the current stock symbol being executed in the function (NVDA or AMD). The predictions are stored in a NumPy array, and the most recent prediction for the current stock symbol is accessed using ctx.preds('slr')[-1], which is the model’s prediction of the next bar’s return.

Now that we have defined a trading strategy and registered the 'slr' model, we can run the backtest using the Walkforward Analysis algorithm.

Run the backtest by calling the walkforward method on the Strategy object, passing the desired number of time windows and the train/test split ratio. In this case, we use 3 time windows, each with a 50/50 train-test split.

Additionally, because our 'slr' model predicts one bar into the future, we must set the lookahead parameter to 1. This prevents training data from leaking across the test boundary. Always set the lookahead parameter to the number of future bars being predicted.

result = strategy.walkforward(
    warmup=20, windows=3, train_size=0.5, lookahead=1, calc_bootstrap=True
)

Backtesting: 2017-03-01 00:00:00 to 2022-03-01 00:00:00

Loaded cached bar data.

Loaded cached indicator data.

Train split: 2017-03-06 00:00:00 to 2018-06-01 00:00:00
AMD R^2=-0.007950114729117885
NVDA R^2=-0.04203365318219077
Finished training models: 0:00:00

Test split: 2018-06-04 00:00:00 to 2019-08-30 00:00:00

100% (314 of 314) |######################| Elapsed Time: 0:00:00 Time:  0:00:00

Train split: 2018-06-04 00:00:00 to 2019-08-30 00:00:00
AMD R^2=0.0006422677593683757
NVDA R^2=-0.0235917302991151
Finished training models: 0:00:00

Test split: 2019-09-03 00:00:00 to 2020-11-27 00:00:00

100% (314 of 314) |######################| Elapsed Time: 0:00:00 Time:  0:00:00

Train split: 2019-09-03 00:00:00 to 2020-11-27 00:00:00
AMD R^2=-0.015508227883924253
NVDA R^2=-0.4567200811037322
Finished training models: 0:00:00

Test split: 2020-11-30 00:00:00 to 2022-02-28 00:00:00

100% (314 of 314) |######################| Elapsed Time: 0:00:00 Time:  0:00:00

Calculating bootstrap metrics: bars=941, samples=10000...
Calculated bootstrap metrics: 0:00:00

Finished backtest: 0:00:00

During Walkforward Analysis, the 'slr' model is trained on a given window’s training data, and the hold_long function runs on that same window’s test data.

The model uses the training data to predict the next day’s price movements. The hold_long function then relies on these predictions to make buy or sell decisions for the current day’s trading session.

This process repeats for each time window in the backtest. We can view the results to evaluate the strategy:

result.metrics_df

	name	value
0	trade_count	43
1	initial_market_value	100000.0
2	end_market_value	107287.5
3	total_pnl	10409.0
4	unrealized_pnl	-3121.5
5	total_return_pct	10.409
6	total_profit	12129.0
7	total_loss	-1720.0
8	total_fees	0.0
9	max_drawdown	-6129.6
10	max_drawdown_pct	-5.537077
11	max_drawdown_date	2022-01-27 00:00:00
12	win_rate	76.744186
13	loss_rate	23.255814
14	winning_trades	33
15	losing_trades	10
16	avg_pnl	242.069767
17	avg_return_pct	5.268372
18	avg_trade_bars	25.488372
19	avg_profit	367.545455
20	avg_profit_pct	9.236364
21	avg_winning_trade_bars	19.151515
22	avg_loss	-172.0
23	avg_loss_pct	-7.826
24	avg_losing_trade_bars	46.4
25	largest_win	2004.0
26	largest_win_pct	23.82
27	largest_win_bars	201
28	largest_loss	-754.0
29	largest_loss_pct	-27.04
30	largest_loss_bars	32
31	max_wins	13
32	max_losses	2
33	sharpe	0.037646
34	sortino	0.054195
35	profit_factor	1.13091
36	ulcer_index	0.984188
37	upi	0.007807
38	equity_r2	0.906783
39	std_error	3117.783269

result.bootstrap.conf_intervals

		lower	upper
name	conf
Profit Factor	97.5%	0.907461	1.415745
	95%	0.941685	1.365034
	90%	0.982327	1.310340
Sharpe Ratio	97.5%	-0.025962	0.102061
	95%	-0.015605	0.091741
	90%	-0.003145	0.080173

result.bootstrap.drawdown_conf

	amount	percent
conf
99.9%	-17114.10	-14.753068
99%	-12992.42	-11.386747
95%	-9880.41	-8.791603
90%	-8543.49	-7.641047

We have now completed the process of training and backtesting a linear regression model using PyBroker, with the help of Walkforward Analysis.

Beyond linear regression, PyBroker can train other model types such as gradient boosted machines, neural networks, or any other architecture that we choose. To help support these models, PyBroker offers extensive customization options. For example, you can specify an input_data_fn to control how the input data is built, and provide your own predict_fn to customize how predictions are generated (overriding the model’s default predict function).

Additionally, PyBroker v2 now supports training time series models and multi-symbol models.