Model Analysis
I chose to use an ANN for this project. Since the data is not sequential it didn’t make much sense to use an LSTM. Also, since this isn’t image data a CNN likely wouldn’t perform better than the ANN. However, this is potentially something I could look into in the future.
In general, the following image is what the architecture of an ANN looks like. There is an input layer where there is a node for each column/feature in your data. For this project, there are 22 nodes in the input layer since there are 22 columns/features in the data. Then there can be any number of hidden layers each with their own number of hidden nodes. Finally, there’s an output layer where there is one node for a binary classification model or nodes equal to the number of classes for a multiclass classification model. This project is a binary classification problem, so like in the image there will only be one node in the output layer. The number and type of hidden layers, the number of hidden nodes in each layer, and the activation function of each layer are all dependent on the specific data being inputted to the model. So, the challenge is building an architecture by determining what those numbers should be based on your data.
Code
The code for the modeling as well as the prepped data can be viewed and downloaded below.
# -*- coding: utf-8 -*-
"""
Created on Sat Dec 2 11:32:52 2023
@author: casey
"""
# ---------------------------------------------------------------------------------------------------------------------------- #
## IMPORTS
import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from sklearn.model_selection import train_test_split
import tensorflow.keras
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import LSTM, Dense, Dropout, LSTM
from tensorflow.keras.optimizers import Adam
import matplotlib.pyplot as plt
from tensorflow.keras import layers
# ------------------------------------------------------------------------------------------------ #
## LOAD PREPPED DATA
nn_df = pd.read_csv('C:/Users/casey/OneDrive/Documents/MSDS_Courses/Fall_2023/Neural_Nets/Project/prepped_data/gbg_nn.csv')
# ------------------------------------------------------------------------------------------------ #
## CREATE TRAIN, VAL, TEST SETS
# X will contain all variables except the labels (the labels are the first column 'survived')
X = nn_df.iloc[:,:-1]
# y will contain the labels (the labels are the first column 'survived')
y = nn_df.iloc[:,-1:]
# split the data vectors randomly into 80% train and 10% test and val
# X_train contains the quantitative variables for the training set
# X_test contains the quantitative variables for the testing set
# X_val contains the quantitative variables for the validation set
# y_train contains the labels for training set
# y_test contains the labels for the testing set
# y_val contains the labels for the validation set
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=616)
X_train, X_test, y_train, y_test = train_test_split(X_train, y_train, test_size=616)
# ------------------------------------------------------------------------------------------------ #
## CONVERT DATAFRAMES TO TENSORS
X_train = tf.convert_to_tensor(X_train)
X_test = tf.convert_to_tensor(X_test)
X_val = tf.convert_to_tensor(X_val)
y_train = tf.convert_to_tensor(y_train)
y_test = tf.convert_to_tensor(y_test)
y_val = tf.convert_to_tensor(y_val)
## What does the data look like?
print("The first value of x_test is \n", X_test[0])
print("The shape of x_test is \n", X_test.shape)
print("The first value of x_train is \n", X_train[0])
print("The shape of x_train is \n", X_train.shape)
print("The first value of x_val is \n", X_val[0])
print("The shape of x_val is \n", X_val.shape)
print("The first value of y_test is \n", y_test[0])
print("The shape of y_test is \n", y_test.shape)
print("The first value of y_train is \n", y_train[0])
print("The shape of y_train is \n", y_train.shape)
print("The first value of y_val is \n", y_val[0])
print("The shape of y_val is \n", y_val.shape)
print("The first few values of y_test are \n", y_test[0:3])
# ---------------------------------------------------------------------------------------------------------------------------- #
# ---------------------------------------------------------------------------------------------------------------------------- #
## MODELING
# ---------------------------------------------------------------------------------------------------------------------------- #
# ---------------------------------------------------------------------------------------------------------------------------- #
# ---------------------------------------------------------------------------------------------------------------------------- #
## BUILD INITIAL ANN MODEL
ANN_Model = tf.keras.models.Sequential([
tf.keras.layers.Dense(8, activation='sigmoid'),
tf.keras.layers.Dense(16, activation='relu'),
#tf.keras.layers.Dense(8, activation='relu'),
#tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(1, activation='sigmoid'),
])
ANN_Model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
optimizer='adam',
metrics=["accuracy"])
Hist=ANN_Model.fit(X_train,y_train, epochs=200, validation_data=(X_val, y_val))
## Save the Model
ANN_Model.save("Sports_Betting_ANN_Model")
ANN_Model.summary()
# ---------------------------------------------------------------------------------------------------------------------------- #
## BUILD INTERMEDIATE ANN MODEL
ANN_Model = tf.keras.models.Sequential([
tf.keras.layers.Dense(100, activation='relu'),
tf.keras.layers.Dense(100, activation='relu'),
tf.keras.layers.Dropout(0.4),
tf.keras.layers.Dense(1, activation='sigmoid'),
])
ANN_Model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
optimizer='adam',
metrics=["accuracy"])
Hist=ANN_Model.fit(X_train,y_train, epochs=200, validation_data=(X_val, y_val))
## Save the Model
ANN_Model.save("Sports_Betting_ANN_Model")
ANN_Model.summary()
# ---------------------------------------------------------------------------------------------------------------------------- #
## BUILD FINAL ANN MODEL
ANN_Model = tf.keras.models.Sequential([
tf.keras.layers.Dense(8, activation='relu'),
tf.keras.layers.Dense(16, activation='relu'),
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Dense(1, activation='sigmoid'),
])
ANN_Model.compile(loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),
optimizer='adam',
metrics=["accuracy"])
Hist=ANN_Model.fit(X_train,y_train, epochs=75, validation_data=(X_val, y_val))
## Save the Model
ANN_Model.save("Sports_Betting_ANN_Model")
ANN_Model.summary()
# ---------------------------------------------------------------------------------------------------------------------------- #
# ---------------------------------------------------------------------------------------------------------------------------- #
## VISUALIZATIONS
# ---------------------------------------------------------------------------------------------------------------------------- #
# ---------------------------------------------------------------------------------------------------------------------------- #
# ---------------------------------------------------------------------------------------------------------------------------- #
## ACCURACY AND LOSS PLOTS
# train accuracy and val accuracy
plt.plot(Hist.history['accuracy'], label='train_accuracy')
plt.plot(Hist.history['val_accuracy'], label = 'val_accuracy')
plt.xlabel('Epoch')
plt.ylabel('Accuracy')
plt.ylim([0.4, 1])
plt.legend(loc='lower right')
# train loss and val loss
plt.plot(Hist.history['loss'], label='train_loss')
plt.plot(Hist.history['val_loss'], label = 'val_loss')
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.ylim([0, 1])
plt.legend(loc='lower right')
# ---------------------------------------------------------------------------------------------------------------------------- #
## ACCURACY ON TEST SET
Test_Loss, Test_Accuracy = ANN_Model.evaluate(X_test, y_test)
print(Test_Accuracy)
# ---------------------------------------------------------------------------------------------------------------------------- #
## GET MODEL PREDICTIONS ON TEST SET
ANNpredictions=ANN_Model.predict([X_test])
print(ANNpredictions)
print(ANNpredictions.shape)
# since using sigmoid, a prediction greater than 0.5 will be positive and less than 0.5 will be negative
ANNpredictions[ANNpredictions >= .5] = 1
ANNpredictions[ANNpredictions < .5] = 0
print(ANNpredictions)
# ---------------------------------------------------------------------------------------------------------------------------- #
## GENERATE CONFUSION MATRIX FOR PREDICTIONS
from sklearn.metrics import confusion_matrix
ANN_CM=confusion_matrix(y_pred=ANNpredictions, y_true=y_test)
print(ANN_CM)
# ---------------------------------------------------------------------------------------------------------------------------- #
## PLOT A PRETTY CONFUSION MATRIX
import seaborn as sns
import matplotlib.pyplot as plt
class_names = ['Over', 'Under']
fig, ax = plt.subplots(figsize=(15,15))
sns.heatmap(ANN_CM, annot=True, fmt='g', ax=ax, annot_kws={'size': 18})
#annot=True to annotate cells, ftm='g' to disable scientific notation
# annot_kws si size of font in heatmap
# labels, title and ticks
ax.set_xlabel('Predicted labels')
ax.set_ylabel('True labels')
ax.set_title('Confusion Matrix: Using an ANN to Predict the Total Result of an NFL Game')
ax.xaxis.set_ticklabels(["0:Over","1:Under"],rotation=90, fontsize = 18)
ax.yaxis.set_ticklabels(["0:Over","1:Under"],rotation=0, fontsize = 18)