Batching - large data sets
When performing tasks like data migrations or other operations that deal with large amounts of data it is not possible to have the data in its entirety in memory.
When working with APIs there are often limits placed on the amount of data that an API endpoint will accept in a single call.
From a speed perspective, it is much faster to call an endpoint once with 100 records of data that to call an endpoint 100 times with one unit of data.
All of these situations require batching in order to optimise for available memory and for speed.
Blocks that support batching
Javascript and Utility blocks that deal with large amounts of data support this feature.
Example 1
There are several Utility Blocks that support batching. Below is one such example using the Hubspot Custom Utility Block.
This will read a batch of company records from the Hubspot API at a time.
It then stores each batch in the Data Store. This might be the basis for an export or a data migration where the data should be stored before being transferred to the target platform in another Flow.
The Hubspot Block is hard-coded to read in batches of 100 records.
Example 2
This example reads in data from a Postgres database and then uploads it to the company object in Hubspot.
We can specify the batch size with the SQL Block. We've set it to 20.
Note the
SELECT * FROM customers ... BATCH ##command. This replaces BATCH ## with theVALUES()at runtime.
Example - Javascript
You can batch with Javascript as shown in the following example. Refer to the Javascript Block for more details.
The first Javascript initiates the batch with batch(BATCH_SIZE). When there is no data left to process. it calls batchEnd().
The second Block shows how you can get batchIteration() and batchOffset() values as the batch loops
Note the Batch End Block is the point at which execution loops back until batchEnd() is called.
Other Blocks that support batching
Hubspot Block