Encog - How to Download Learning Data for Neural Network

Question

The NeuralDataSet objects that I saw in action were not something like XOR, which is just two small arrays of data ... I could not figure out anything from the documentation for MLDataSet .

It seems that everything should be loaded right away. However, I would like to loop the learning data until I reach the EOF, and then count it as 1 era. However, everything I saw, all the data should be loaded into 1 2D array from the very beginning. How can I get around this?

I read this question and the answers did not help me . And besides, I did not find a similar question asked here.

Answer 1

This is possible, you can either use an existing implementation of a dataset that supports streaming control, or you can implement your own on top of any source that you have. Check the BasicMLDataSet and SQLNeuralDataSet interface for an example. You will need to implement a codec if you have a specific format. For CSV there is an implementation already, I have not tested if it is based on memory.

Remember, when you do this, that your data is completely transferred for each era and from my experience, which is a much more bottleneck than the actual calculation of the network.