GPU vs TPU training speed for timeseries data

Currently a NVIDIA P100 is being used for timeseries data training, however, as it goes, LTSM with a batch size of 1 just goes slow. Some can be relatively large datasets. I am looking at other options, either something newer than the P100, or using Google TPU's. Does anyone have experience with this, and if so, can you please let me know what type of performance and speed increases could be expected, or that you experienced, of your LSTM time series data training.

[Joss Knight]

Why are you using a batch size of 1?

[Walter Roberson]

"relatively large datasets"... probably running out of memory with larger batch size.

[Robert]

Batch size is 1 because it's time series data, and a prediction is made at each datapoint, and the 0/1 prediction is also fed back into the LSTM as part of the next prediction. (No issues running out of memory)

[Joss Knight]

For stateful sequence networks you can split sequences and MATLAB will manage the state propagation between minibatches. See the documentation This would be the best way to get good parallelism with long sequences. With only one sequence per mini-batch there's almost nothing useful the GPU can do to parallelize what is a fundamentally serial algorithm.

An alternative would be to convert to a solution involving temporal convolution .