Regression Learner: Importing batches from a database as input data

My problem is that I have a large sqlite database and I would prefer not to import all the rows in memory at once and instead feed the data in batches.

I was experimenting with the fetch() method, but the Regression Learner only accepts tables from workspace and again, I would prefer if I dont have to import the full database into a table in workspace.

Is there a way to achieve this in Matlab?

Answer 1

[Ishaan Mehta]

Hi Dominik

As I understand, your problem can be solved by importing the table's rows in batches, instead of fetching the entire table at once. This can be done using the LIMIT clause in SQLite.

LIMIT clause accepts two parameters, the number of rows to fetch and the offset i.e. number of rows to skip from the top of your table.

Here is a code snippet to fetch data in batches using a for loop:

for i = 1:5 % set upper limit according to number of rows in your table.
    rowsToFetch = 10; % fetch 5 rows at once
    offset = (i-1) * rowsToFetch;
    sqlquery = sprintf('SELECT * from tablename LIMIT %d,%d', offset, rowsToFetch);
    partialResults = fetch(conn, sqlquery)
end

Hope it helps

Ishaan Mehta