Comparison of Three Methods

From the findings of my last post, I decided to both log the intensity and the output energies. I used the following hyper-parameters for each model

• NN (3x64x16x3): Epochs = 35, learning rate = 1e-3 , batch size = 32
• GPR: tolerance = 1e-4, learning rate = 1e-1
• SVR(RBF): C = 1, epsilon = 0.1, tolerance = 1e-4

Time Comparison

First, I plotted the amount of time it would take to train a model as a function of time

We can see that the neural network takes much longer than the GPR which in turn takes much longer than the SVR. One caveat is that the SVR and GP was only running for Max Energy and not Total/Average Energy. This would effectively triple their compute times. I am working on solving for all three energies for GPR and SVR to make a better comparison.

Error as a function of Noise Level

Interestingly enough, all three methods seem to work well for Noise levels of 10\% or greater

However, for noise levels of 5\% or less, we see the SVR perform poorly (for the 21000 point dataset). Interestingly enough, its only at this 21000 model and not most of the others

And if we look at when there is no noise at all, we still get errors for both SVR and GP

The SVR seems to converge around 5\%. Maybe this is some tolerance or epsilon value that I need to adjust. But the GPR is less clear. I need to find a way to make these regressions work better.

Things to Do

• Increase size of testing data to some fixed value, see if that mitigates issues with a low number of testing points
• Train Average and Total Proton Energy with GPR and SVR
• Fix Issues with 0 and 5 percent for SVR and GPR. Model needs to work on noiseless data, otherwise its useless
• Look into ways to measure noise
• Get Average GPU Usage Metrics and Compare
• Change seed to see how mape(exact, noisy) compares for a small training data set size: why is it underestimated initially
• Re-run sims for low number of training points and fine tune exactly how many points are needed to train.