Skip to main content

Bulk Import Worker

Processes large data imports from JSON streams with event-based processing.

Included in Base Preset

This plugin is part of the base preset and loaded automatically. Using the base preset is strongly recommended, so explicit installation is usually not required.

Installation​

import '@unchainedshop/plugins/worker/bulk-import';

Features​

  • Streaming Processing: Handles large files without memory issues
  • JSON Stream Parsing: Parses JSON events from uploaded files
  • Event-Based: Processes import events one by one
  • File Adapter Integration: Works with any Unchained file storage adapter
  • Backpressure Handling: Automatic flow control for large datasets

Usage​

Upload a JSON file with import events:

{
  "events": [
    {
      "type": "PRODUCT",
      "operation": "CREATE",
      "payload": {
        "sku": "PRODUCT-001",
        "title": "Sample Product"
      }
    },
    {
      "type": "PRODUCT",
      "operation": "UPDATE",
      "payload": {
        "sku": "PRODUCT-002",
        "title": "Updated Product"
      }
    }
  ]
}

Supported Event Types​

  • PRODUCT - Create/update products
  • ASSORTMENT - Create/update assortments
  • FILTER - Create/update filters
  • ENROLLMENT - Create/update enrollments

Triggering Import​

From Uploaded File​

mutation CreateBulkImportWork {
  createWork(
    type: "BULK_IMPORT"
    input: {
      payloadId: "uploaded-file-id"
      createShouldUpsertIfIDExists: false
      updateShouldUpsertIfIDNotExists: false
      skipCacheInvalidation: false
    }
  ) {
    _id
    status
  }
}

From Direct Events​

mutation CreateBulkImportWork {
  createWork(
    type: "BULK_IMPORT"
    input: {
      events: [
        { "type": "PRODUCT", "operation": "CREATE", "payload": { "sku": "PRODUCT-001" } }
      ]
    }
  ) {
    _id
    status
  }
}

Input Parameters​

ParameterTypeDefaultDescription
payloadIdString-File ID of uploaded JSON stream
eventsArray-Direct array of import events (alternative to payloadId)
createShouldUpsertIfIDExistsBooleanfalseUpsert on CREATE if ID already exists
updateShouldUpsertIfIDNotExistsBooleanfalseUpsert on UPDATE if ID doesn't exist
skipCacheInvalidationBooleanfalseSkip cache invalidation after import

Adapter Details​

PropertyValue
Keyshop.unchained.worker-plugin.bulk-import
Sourceworker/bulk-import.ts