Fix: SQLite "too many SQL variables" error in Importer

[pixelastronauts]

When importing large datasets with SQLite, the process fails with too many SQL variables error because SQLite has a variable limit in queries.

This PR chunks the items into smaller batches.

Tested with 5,000+ row XML import - successfully processes without errors.

Fixes #96

[duncanmcclean]

Thanks for this pull request!

However, I think this might cause an issue with the UpdateCollectionTreeJob job which needs to be run after everything has been imported, not after 500 of them have been imported. 🤔

[pixelastronauts]

Good catch. Maybe we can do something like this then:

        $items = match ($import->get('type')) {
            'csv' => (new Csv($import))->getItems($import->get('path')),
            'xml' => (new Xml($import))->getItems($import->get('path')),
        };

        $chunks = $items->chunk(500);
        $pendingBatches = [];

        foreach ($chunks as $chunk) {
            $pendingBatch = Bus::batch($chunk->map(fn (array $item) => new ImportItemJob($import, $item)))
                ->before(fn (Batch $batch) => $import->batchId($batch->id)->save());
            
            $pendingBatches[] = $pendingBatch;
            $pendingBatch->dispatch();
        }

        // Only dispatch the UpdateCollectionTreeJob after the last batch
        if (count($pendingBatches) > 0 && $import->get('destination.type') === 'entries') {
            $lastBatch = array_pop($pendingBatches);
            
            Bus::chain([
                function () use ($import) {
                    UpdateCollectionTreeJob::dispatch($import);
                }
            ])->afterLastBatch($lastBatch)->dispatch();
        }

[duncanmcclean]

We might be able to do something with lazy collections but I'm not sure. Will need to have a think.

[duncanmcclean]

Bus::chain([
    function () use ($import) {
        UpdateCollectionTreeJob::dispatch($import);
    }
])->afterLastBatch($lastBatch)->dispatch();

Unfortunately, it doesn't look like the afterLastBatch method exists (I can't find it in the Laravel codebase), otherwise this would have been a good solution.

I'll need to have a think about the best way to approach this.

[pixelastronauts]

Already discovered this while testing. Really thought this was a method.

See my latest commit 966b3ca which I was already using in a dev branch. I think this fixes it and only dispatches the UpdateCollectionTreeJob once, right?

Not that elegant, so maybe needs some tinkering I guess.

[duncanmcclean]

Unfortunately, I think might be some edge cases with dispatching it during the last batch vs dispatching it when all jobs have been completed.

For example: it's possible that the last batch could be completed before all batches from previous batches have completed, which would mean the UpdateCollectionTreeJob would be run prematurely.

[duncanmcclean]

If it's easiest, I'm happy to look into this, but I'd need to test with a large enough import. Would you be able to send the file you're importing from to [email protected]?