Regularization, Hyperparameter Tuning of Low Rank Autoregressive Models

1. Introduction

I conducted Regularization and Hyperparameter Tuning Experiments on the Low Rank Autoregressive Models used in this paper: Active Learning of Neural Population Dynamics using two-photon holographic optogenetics Some motivation for why is that there is a strong need for techniques that minimize the amount of data needed to learn neural population dynamics due to experimental time and resource constraints. The long-term goal that the above paper is working towards is for a model to be able to actively learn the patterns of neurons that have the most informative neural responses to quickly learn the neural population dynamics (brain activity). This project used experimental data where a region of a mice brain was photostimulated with a laser. Specifically, patterns of neurons were photostimulated and the rest of region neuron's response (spikes) to that stimulation was recorded.
The project’s active learning technique takes advantage of the low-rank structure of the neural population dynamics to determine the most informative photostimulation patterns. It uses SVD to create low rank autoregressive models that predict neural activity (spikes).

2. L2 Regularization and Hyperparameter Tuning

These Low Rank Models are the models I was tasked with experimenting on. I used cross validation loops to tune hyperparameters such as...

• Lambda (L2 Regularization strength)
• Epochs
• Learning Rate
• Weight Initialization
• k (number of autoregressive time steps used to make prediction)

My best model resulted in a 25% improvement in performance (MSE) over the baselines the team had before I came on.