Page through large responses in Operations Hub

Notes

When Operations Hub introduced scheduled workflows, I got very excited. I saw the opportunity to run cron jobs on large external databases, extract the relevant information and push that into HubSpot. 

And it works really well when you're looking to update single records on a recurring basis. 

But it works terribly if you're looking to update a large number of records based on interaction with a single record. 

For example, we recently worked on a project where we needed to do a daily lookup of registrants for an event. We had `Events` as a custom object and, each day, a custom-coded workflow action asked the events database for all new registrants. 

But there could be thousands of new registrants. The Operations Hub limit of 20 seconds for a serverless function just wouldn't be long enough to process all that data. 

We needed to page through that data with multiple serverless functions. To make this happen, we created three workflows: 

• two that had near identical code, except one was triggered by a property called `pageAgain` being `true`. 
• one that ensured the record was out of the second workflow before triggering it a second time (a classic HubSpot problem of a record being refused enrolment because it's already enrolled). 

The first two workflows ended with this branch based on the output of the custom code: 

 

Here's the third workflow in its entirety: 

 

The details of the custom-coded workflow aren't the point here. What's really important is the detail of the callback (which you can see in the code snippet below) and ensuring that you push that data back out to the workflow itself, which you can do in the `Data Outputs` section of the coded step: 

 

You'll also want to ensure that the second workflow (the one that's nearly identical to the first), includes the pageInput in its inputs. This will allow your code to know which page of the data to ask for. 

Code

        
Language:  JavaScript


        
 
function triggerCallback() {
    console.log("triggerCallback firing");
    callback({
         outputFields: {
              lastSearchDateTime: lastSearchDateTime,
              recordsOutput: recordsOutput,
              pagesOutput: pagesOutput,
              pagesInput: parseInt(pageInput) + 1,
              pageAgain: pageAgain
              }
         });
     }
 
      

Back to listing