Split In Batches

Process items in chunks to avoid API rate limits or handle large datasets.

Written By pvdyck

Last updated About 5 hours ago

Split In Batches

The Split In Batches node (also called "Loop Over Items") divides a large set of items into smaller groups and processes them one batch at a time in a loop.

Parameters

ParameterDescription
Batch SizeNumber of items per batch (default: 10)

Options

OptionDescription
ResetWhen enabled, resets the node's internal counter on each workflow execution. Useful when the node is called multiple times in a single workflow

How It Works

  1. The node receives all items
  2. It outputs the first batch (e.g., 10 items) through the "loop" output
  3. After processing, connect the loop back to Split In Batches
  4. It outputs the next batch, and so on
  5. When all batches are done, items flow out the "done" output

Use Cases

  • Rate limiting: Process 10 items at a time with a Wait node between batches to avoid API throttling
  • Memory management: Handle large datasets without loading everything at once
  • Progress tracking: Access $node["Split In Batches"].context["currentRunIndex"] to track which batch is being processed

Tips

  • Add a Wait node in the loop to space out API calls
  • Use the "done" output to trigger post-processing after all batches complete
  • Set batch size based on your API's rate limits (e.g., batch size 5 with 1-second waits for APIs allowing 5 requests/second)

Limitations

  • None. Full support.

Related