Posts

Scaling 1,000+ Item Syncs to Sitecore XM Cloud with p-limit in Next.js

Thursday, June 4, 2026

The Use Case

If you've tried firing hundreds of requests at once, you've probably seen 429 errors, dropped requests, or long-running processes crash. The fix isn't just "slower" - it's controlled concurrency.

This post shows how to use p-limit to reliably sync ~1,000 items from a Next.js app into Sitecore XM Cloud-with batching, retries, and backoff.

Why p-limit?

p-limit gives you a simple way to say:
"Run many async tasks, but only N at a time."

Instead of launching 1,000 requests in parallel, you cap concurrency (e.g., 5-10). That alone dramatically reduces throttling and improves success rates.

But p-limit shines when combined with:

Batching - process items in chunks
Retries with exponential backoff
Small delays between batches

The Target Pattern

Batch → Limit Concurrency → Retry → Pause → Repeat

Batch size: 25–100
Concurrency: 3–5 (start here)
Catch all failed items and retry on failed
Delay: 200–500ms between batches

Step 1: Install and set up p-limit

npm install p-limit

import pLimit from "p-limit";

const CONCURRENCY = 5;
const limit = pLimit(CONCURRENCY);

const CONCURRENCY = 5; Defines that no more than 5 async tasks should run simultaneously.
const limit = pLimit(CONCURRENCY); Creates a function (limit) that wraps async operations and ensures that at most 5 are running at any given time.

Step 2: Retry with exponential backoff

Handle transient failures like 429 (rate limit) and 5xx:

const sleep = (ms: number) => new Promise(res => setTimeout(res, ms));

  const retryWithBackoff = async (
  fn: () => Promise<any>,
  retries = 3,
  delay = 500
) => {
  try {
    return await fn();
  } catch (err: any) {
    const status = err?.response?.status;

    if (retries > 0 && (status === 429 || status >= 500)) {
      await sleep(delay);
      return retryWithBackoff(fn, retries - 1, delay * 2);
    }

    throw err;
  }
};

It tries to execute an async function (fn). If the operation fails because the server is overloaded or temporarily unavailable, it waits for some time and tries again. Each retry waits longer than the previous one. It handles Rate limiting (HTTP 429), Temporary server failures (HTTP 500, 502, 503, 504, etc.) and Unstable network conditions.

Step 3: Wrap your Sitecore call

Create a single function responsible for syncing one item:

const syncToSitecore = async (item: any) => {
  return retryWithBackoff(async () => {
    const res = await fetch("/api/sitecore-sync", {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
      },
      body: JSON.stringify(item),
    });

    if (!res.ok) {
      const error: any = new Error("Request failed");
      error.response = { status: res.status };
      throw error;
    }

    return res.json();
  });
};

syncToSitecore(item)

↓

POST /api/sitecore-sync

Success (200-299)

↓

Return JSON response

Failure

↓

Throw error with status

↓

retryWithBackoff()

↓

Retry if 429 or 5xx

Step 4: Add batching + p-limit

Now combine batching with controlled concurrency:

const BATCH_SIZE = 50;

const processItems = async (items: any[]) => {
  for (let i = 0; i < items.length; i += BATCH_SIZE) {
    const batch = items.slice(i, i + BATCH_SIZE);

    console.log(`Processing batch ${i / BATCH_SIZE + 1}`);

    const tasks = batch.map(item =>
      limit(() => syncToSitecore(item))
    );

    await Promise.all(tasks);

    await sleep(300); // small pause to avoid throttling
  }
};

This code processes a large list of items and sends them to Sitecore in batches, while also limiting concurrency and adding retries. It

Splits the items into groups of 50.
Processes one group at a time.
Within each group, only 5 requests run simultaneously (because of your earlier p-limit(5)).
Retries failed requests using retryWithBackoff().
Waits 300 ms between batches.

200 Items

↓

Batch 1 (1–50)

↓

Batch 2 (51–100)

↓

Batch 3 (101–150)

↓

Batch 4 (151–200)

Step 5: Add failure tracking (don’t skip this)

const failedItems: any[] = [];
const safeSync = async (item: any) => {
  try {
    return await syncToSitecore(item);
  } catch (err) {
    failedItems.push(item);
  }
};
Then use:
const tasks = batch.map(item =>
  limit(() => safeSync(item))
);

This lets you:

retry later
log errors
avoid silent data loss

Optional: Adaptive throttling

let dynamicDelay = 300;

if (res.status === 429) {
  dynamicDelay += 200;
}
Then:
await sleep(dynamicDelay);

Where to run this code

You can run this in a .tsx component (e.g., useEffect or a button handler), but for 1,000+ items:

Better options

Next.js API route (/api/sync)
Background worker (Node, Azure Function, etc.)

Why

avoids browser limits
prevents UI blocking
more reliable for long-running jobs

What NOT to do

❌ Don't do this

await Promise.all(items.map(syncToSitecore));

This causes:

request spikes
throttling (429)
unstable syncs

Tuning guidelines

Start conservative:

Concurrency: 5
Batch size: 50
Delay: 300ms

Then adjust:

Increase concurrency if stable
Reduce if you see 429s
Monitor response times

Final Takeaway

For syncing large datasets into Sitecore XM Cloud:

p-limit is your concurrency guardrail—but batching, retry, and pacing make it production-ready.

If Used together, they give you:

high success rates
predictable performance
resilience under load

Scaling 1,000+ Item Syncs to Sitecore XM Cloud with p-limit in Next.jsThursday, June 4, 2026