Trouble doing time series analysis in Python

There are quite a lot of bugs in your code. I'll explain one by one in the following (check comments between ''' '''):

'''
small note, here you defined the variable as units and below you want to use a column called "variable".
Not a big problem, most probably you were reading the data from a file anyway, just something to keep in mind
'''
df = pd.DataFrame({'date':pd.date_range(begin_date, periods=len(units)),'units':units})
'''
The following two lines do not work like that. 
First, the dataframe is not indexed by a datetime
Second, to reference the index you need to use .iloc. Alternatively you can use .loc
'''
# Create estimation data set
est_data = df['2022-08-01':'2022-08-30'] 
# And observation data
obs_data = df['2022-09-01':'2022-09-14']
'''
Here you are fitting according to est_data.
using the m.predict() function will give you the fitted points of est_data.
This will be important later
'''
# Estimate a model predicting stock price with market return
m = smf.ols('variable ~ date', data = est_data).fit()
# Get AR
# Using mean of estimation return
'''
you don't need np.mean for this, just use est_data['variable'].mean()
Also it is most probably not needed to have the mean in your script.
You can directly subtract using obs_data['variable'] - est_data['variable'].mean()
'''
var_return = np.mean(est_data['variable'])
obs_data['AR_mean'] = obs_data['variable'] - var_return
'''
This will not always work, and in this case it does not.
m.predict() returns the predictions based on the data in est_data. The same number of points will be outputed
In order for this to work, obs_data needs to have the same number of points as est_data
'''
obs_data['risk_pred'] = m.predict()
obs_data['AR_risk'] = obs_data['variable'] - obs_data['risk_pred']

Am currently working on fixing the bugs, will give you a working example soon. For this can you please leave me answers to the following question:

• Do you really want to fit the model according to est_data? If so, how are you gonna combine this with obs_data?

Edit 1: how to separate the data

The following code references the dates in the data frame:

est_data_Start = pd.to_datetime('2022-08-01')
est_data_End = pd.to_datetime('2022-08-30')
obs_data_Start = pd.to_datetime('2022-09-01')
est_data = df[df["date"].between(est_data_Start,est_data_End)]
obs_data = df[df["date"]>obs_data_Start]

The results for est_data are:

    date    variable
0   2022-08-01  0.916301
1   2022-08-02  0.483948
2   2022-08-03  0.551259
3   2022-08-04  0.147971
4   2022-08-05  0.617462
5   2022-08-06  0.957460
6   2022-08-07  0.905076
7   2022-08-08  0.274262
8   2022-08-09  0.861609
9   2022-08-10  0.285915
10  2022-08-11  0.989687
11  2022-08-12  0.866146
12  2022-08-13  0.074251
13  2022-08-14  0.209507
14  2022-08-15  0.082519
15  2022-08-16  0.215795
16  2022-08-17  0.953852
17  2022-08-18  0.768329
18  2022-08-19  0.380686
19  2022-08-20  0.623940
20  2022-08-21  0.155944
21  2022-08-22  0.495746
22  2022-08-23  0.084551
23  2022-08-24  0.519966
24  2022-08-25  0.706618
25  2022-08-26  0.872301
26  2022-08-27  0.707696
27  2022-08-28  0.760617
28  2022-08-29  0.213848
29  2022-08-30  0.703866

And the rest is going to obs_data.

Edit 2: fitting and predicting

The following code uses OLS to fit a model to the est_data. Then, the model is used to predict the values based on the data found in obs_data:

XTrain = est_data.index # get the training predictor
XTrain = sm.add_constant(XTrain) # add constant term to account for any intercept
m = sm.OLS(est_data["variable"], XTrain).fit() # fit according to training
XTest = obs_data.index # get the testing predictors
XTest = sm.add_constant(XTest) # and add a constant term
obs_data["risk_pred"] = m.predict(XTest) # predict based on the new data
# the following two calculations I just copied from you...
obs_data["AR_mean"] = obs_data["variable"] - est_data["variable"].mean()
obs_data["AR_risk"] = obs_data["variable"] - obs_data["risk_pred"]

The following code plots the results:

plt.figure()
plt.plot(est_data["date"], est_data["variable"], "-o", label = "Estimated")
plt.plot(obs_data["date"], obs_data["variable"], "-o", label = "Observed")
plt.plot(est_data["date"], m.predict(XTrain), label = "Train fit")
plt.plot(obs_data["date"], m.predict(XTest), label = "Test fit")
plt.legend(ncols =4, bbox_to_anchor=[0.5, 1.1, 0.5, 0])
plt.grid()
locator = mdates.AutoDateLocator(minticks = 7) 
formatter = mdates.ConciseDateFormatter(locator) 
plt.gca().xaxis.set_major_locator(locator) 
plt.gca().xaxis.set_major_formatter(formatter) 

The results and the imports are in the following section:

Imports:

import pandas as pd
import numpy as np
import statsmodels.api as sm
%matplotlib notebook
import matplotlib.pyplot as plt
import matplotlib.dates as mdates