Shopify Bulk Operations: Importing Large Catalogues with productSet

Importing a large catalogue through ordinary GraphQL calls is a long fight with the cost-based rate limiter: every product is a request, every request spends budget, and a 100,000-product import takes hours of careful throttling. The bulk mutation API moves that loop to Shopify’s side. You hand over a file of variables, Shopify runs the mutation once per line, and you collect the results. The mechanics are simple; the parts that cause trouble are file limits, ordering, and productSet‘s deliberately destructive sync semantics.

Choosing the right write path

The bulk path is the right default once a job is measured in thousands of products. For the event-driven trickle of single-product updates between full syncs, the normal API is simpler and gives you an immediate answer. If you only need to read a large catalogue, use bulkOperationRunQuery instead; the export side is covered briefly in Shopify App Development: OAuth, Webhooks and Rate Limits.

The limits that shape the design

All of these are from Shopify’s bulk import documentation and bulk query documentation as of API 2026-10.

The concurrency change matters. Older guidance, including some of my own earlier notes, says only one bulk operation runs per shop at a time. That was true before API 2026-01. If your app is pinned to an older version it still is, so check the version before building a parallel queue.

productSet: a declaration, not a patch

productSet is the right mutation for catalogue ingestion because it treats each call as the complete state of a product. That is also why it needs care:

• List fields are fully synced. Variants, metafields and collections that are missing from the input are deleted. Send a product with two of its five variants and three variants disappear.
• Other fields are left alone if omitted, but sending an empty value clears them: tags: [] removes all tags.
• Identify products deliberately. The identifier argument accepts an id, a handle or a customId (a unique metafield value). Upserting by handle or by your ERP’s product ID in a unique metafield means you do not need to store Shopify GIDs before the first import.
• Products default to a limit of 2,048 variants.
• Inside a bulk operation, productSet always runs synchronously; the synchronous argument is ignored.

The practical rule: the source system must send complete products. If your feed only has partial data for some products (say, a price update without variant options), route those through productVariantsBulkUpdate instead of building a partial productSet that will delete things.

The pipeline, step by step

1. Generate the JSONL

Each line is a JSON object holding the variables for one mutation call. Stream it rather than building it in memory, and roll over to a new file well before 100MB.

// Node.js 20+, ES modules. `products` is an async iterable from your source system.
import {createWriteStream} from 'node:fs';

const MAX_BYTES = 90 * 1024 * 1024; // headroom under the 100MB cap

export async function writeChunks(products) {
  const files = [];
  let out, bytes = 0;

  for await (const p of products) {
    if (!out || bytes > MAX_BYTES) {
      out?.end();
      const path = `catalogue-${files.length}.jsonl`;
      files.push(path);
      out = createWriteStream(path);
      bytes = 0;
    }
    const line = JSON.stringify({
      identifier: {handle: p.handle},
      input: {
        handle: p.handle,
        title: p.title,
        vendor: p.vendor,
        productOptions: p.options,   // [{name, values: [{name}]}]
        variants: p.variants,        // every variant that should exist
      },
    }) + '\n';
    bytes += Buffer.byteLength(line);
    out.write(line);
  }
  out?.end();
  return files;
}

2. Reserve an upload target and upload the file

stagedUploadsCreate returns a signed URL and a set of form parameters. The upload is a multipart POST in which every returned parameter is sent as a form field and the file is the last field. The value of the key parameter is the path you pass to the bulk mutation.

import fs from 'node:fs';

const STAGE = `mutation {
  stagedUploadsCreate(input: [{
    resource: BULK_MUTATION_VARIABLES,
    filename: "catalogue.jsonl",
    mimeType: "text/jsonl",
    httpMethod: POST
  }]) {
    stagedTargets { url parameters { name value } }
    userErrors { field message }
  }
}`;

async function upload(admin, filePath) {
  const {stagedUploadsCreate: s} = await admin(STAGE);
  const target = s.stagedTargets[0];

  const form = new FormData();
  for (const {name, value} of target.parameters) form.append(name, value);
  form.append('file', await fs.openAsBlob(filePath)); // must be last

  const res = await fetch(target.url, {method: 'POST', body: form});
  if (!res.ok) throw new Error(`Staged upload failed: ${res.status}`);

  return target.parameters.find((p) => p.name === 'key').value;
}

Here admin() is your authenticated GraphQL Admin API helper that returns the data object; fs.openAsBlob needs Node 19.8 or later.

3. Start the bulk mutation

mutation RunImport($path: String!) {
  bulkOperationRunMutation(
    mutation: "mutation call($identifier: ProductSetIdentifiers, $input: ProductSetInput!) { productSet(identifier: $identifier, input: $input) { product { id handle } userErrors { field message code } } }",
    stagedUploadPath: $path,
    clientIdentifier: "catalogue-2026-10-18-chunk-0"
  ) {
    bulkOperation { id status }
    userErrors { field message }
  }
}

Store the returned operation ID against the chunk. The optional clientIdentifier gives you something human-readable to query by later. Do not modify or delete the staged file while the operation is running; Shopify reports an INTERNAL_SERVER_ERROR if you do.

4. Wait for completion: webhook first, polling as backup

Subscribe to the BULK_OPERATIONS_FINISH topic. It fires when an operation completes, fails or is cancelled, for both queries and mutations. Two details catch people out: status and error_code in the payload are lowercase (completed, not COMPLETED), and Shopify does not guarantee delivery. Keep a slow poll of bulkOperation(id:) running for any operation that has not reported back.

The webhook handler should do almost nothing: verify the HMAC, enqueue a “download results” job keyed by the operation ID, and return 200. Shopify retries deliveries, so the job must be idempotent. That queue design is the subject of Scaling Distributed Webhook Ingestion with Redis and BullMQ.

5. Reconcile the result file

When the operation completes, its url field points to a JSONL file in which each line is the response for one input line, tagged with __lineNumber. If the operation failed partway, partialDataUrl may hold what did finish.

import readline from 'node:readline';
import {Readable} from 'node:stream';

async function reconcile(resultUrl, onFailure) {
  const res = await fetch(resultUrl);
  const lines = readline.createInterface({input: Readable.fromWeb(res.body)});

  for await (const raw of lines) {
    const row = JSON.parse(raw);
    const errors = row.data?.productSet?.userErrors ?? [];
    if (row.errors?.length || errors.length) {
      // __lineNumber is zero-based and points back into your input chunk.
      await onFailure(row.__lineNumber, row.errors ?? errors);
    }
  }
}

Write failures to a retry file and feed that back through the same pipeline. A successful bulk operation with failed lines is normal; treat the operation status and the per-line results as two separate signals.

Ordering and dependencies

Shopify explicitly does not process the file serially. Two consequences:

• Never put the same product on two lines of one file, or in two chunks running concurrently. Deduplicate by identifier before writing.
• Anything that depends on a product existing (publishing to a channel by ID, adding to a manual collection with a separate mutation, setting inventory at a location) belongs in a second phase that starts after the first has completed and you have the GIDs from the result file.

Inventory deserves its own phase anyway. Stock changes far more often than product content, and mixing the two means a stock update requires resending the full product. Run inventory as a separate, smaller sync with the inventory mutations, and keep productSet for content. The wider question of which system owns stock is discussed in E-Commerce Engineering: Shopify Custom Apps & Scalable WooCommerce Systems.

Production checklist

• Pin the app to API 2026-01 or later if you want parallel bulk operations, and cap your queue at five per shop.
• Chunk files below 100MB with headroom, and deduplicate identifiers across concurrently running chunks.
• Send complete products to productSet; route partial updates elsewhere.
• Upsert by handle or a unique customId metafield so the first import does not need stored GIDs.
• Subscribe to bulk_operations/finish and keep a fallback poll.
• Download result files promptly (the URL expires after a week) and retry failed lines from a generated file.
• Run dependent work (inventory, publishing, collections) as a second phase.
• Do a dry run on a development store with a copy of the real feed: the bugs are in your data, not the API.

Once the catalogue is in, a headless storefront reading it needs its own caching discipline; see Storefront API caching in Hydrogen.

Questions people actually ask

How many bulk operations can run at the same time?

From API 2026-01, an app can run up to five bulk mutation operations and up to five bulk query operations per shop at once. On earlier API versions it is one of each type.

Does bulkOperationRunMutation count against the rate limit?

The request that starts the operation is not subject to the usual rate limits, and the mutations inside it run on Shopify’s side. The surrounding calls (staged upload, polling, follow-up queries) still spend normal budget.

Why did productSet delete my variants?

Because variants are a list field and productSet fully syncs list fields. Any variant not in the input is removed. Send every variant, or use productVariantsBulkUpdate for partial changes.

What if the bulk_operations/finish webhook never arrives?

Shopify does not guarantee delivery, so poll bulkOperation(id:) on a slow schedule for any operation that has not reported back within the time you would expect.