PIM Connector Framework
Import Data from any PIM
We have created a special API to upload large amounts of data, for example from a PIM or ERP system, to the Unchained Engine.
Bulk Import API
The Bulk API stores the data in the Unchained Work Queue before processing and enables some neat features:
- Cloud native: Allows dedicated background or worker instances to do the actual processing
- Transparent process: Also stores the result on the work item, making successful and failed imports queryable.
- Captures sync issues and reports those with a status e-mail to a central e-mail address (sync error reporting)
- Performance: Only downloads assets that are not downloaded yet and tries to optimize the actual write calls to the database by using MongoDB bulk operations, making sync processing fast
- Push-based: Immediate representation of changes
In some situations, it's propably wise to develop a sync microservice: You have a source system that
- generates "Pull-based" data feeds, or
- can not adopt to the JSON described below
Endpoint
There is two ways to send bulk import events, one is through GraphQL by adding a BULK_IMPORT work type with an input like this:
{
events: [...],
...options
}
The other way is to use the REST endpoint /bulk-import:
curl -X POST -H "Authorization: Bearer XXX" -H "content-type: application/json" --data-binary '{ "events": [] }' -f -v http://localhost:4010/bulk-import?optionA=valueA
Every event consists of a type, an operation and a payload.
Supported entity types:
- PRODUCT
- ASSORTMENT
- FILTER
- ENROLLMENT (coming soon)
- ORDER (coming soon)
- USER (coming soon)
- REVIEWS (coming soon)
Supported operations:
- CREATE
- UPDATE
- REMOVE
All Events follow this JSON Structure:
{
"entity": "ENTITY TYPE",
"operation": "OPERATION TYPE",
"payload": { ... }
}
Try to always send as many events at a time, so Unchained can optimize write operations. And be aware that because of the MongoDB Document Size limitation, you have to use the REST endpoint when you send JSON which is more than 16m of size. If you have a lot of categories and/or products (5K entities+) please use the REST endpoint.
Options:
createShouldUpsertIfIDExists
: In some situations, this can be helpful programming forgiving sync code. If you setcreateShouldUpsertIfIDExists
to true, CREATE operations will not fail if the entity with the payloadId already exists and the bulk importer instead tries to merge the new product with the existing one by using update methods.skipCacheInvalidation
: In some situations, this can be helpful skipping because it can add a lot of weight to the BULK_IMPORT operation and sometimes it's not needed for example availability syncs to have updated filter and assortment caches.
JSON Reference
Entity Type: Product
Set by Unchained:
- authorId,
- slug history
- _id, created & updated if not provided
Languages:
- The language code in "content" fields should match an existing language entity's isoCode in Unchained.
Status:
- You can only use ACTIVE or DRAFT. You have to use the remove operation to set it to DELETED.
{
"entity": "PRODUCT",
"operation": "CREATE",
"payload": {
"_id": "A",
"specification": {
"created": null,
"updated": null,
"sequence": "",
"tags": ["nice"],
"type": "SimpleProduct",
"published": "2020-01-01T00:00Z",
"commerce": {
"salesUnit": "ST",
"salesQuantityPerUnit": "1",
"defaultOrderQuantity": "6",
"pricing": [
{
"isTaxable": true,
"isNetPrice": true,
"countryCode": "CH",
"currencyCode": "CHF",
"amount": 10000,
"maxQuantity": null
}
]
},
"warehousing": {
"baseUnit": "ST",
"sku": "",
"dimensions": {
"weightInGram": 0,
"heightInMillimeters": 0,
"lengthInMillimeters": 0,
"widthInMillimeters": 0
}
},
"variationResolvers": [
{
"vector": {
"color": "red"
},
"productId": "B"
}
],
"plan": {
"billingInterval": "DAYS",
"billingIntervalCount": 1,
"usageCalculationType": "METERED",
"trialInterval": "DAYS",
"trialIntervalCount": 1
},
"bundleItems": [
{
"productId": "c",
"quantity": 1,
"configuration": [
{
"key": "greeting",
"value": "For my Darling"
}
]
}
],
"meta": {},
"content": {
"de": {
"created": null,
"updated": null,
"vendor": "Herstellername",
"brand": "Marke",
"title": "Produktname",
"slug": "produktname",
"subtitle": "Short description",
"description": "Long description",
"labels": ["Neu"]
}
}
},
"media": [
{
"_id": null,
"created": null,
"updated": null,
"asset": {
"_id": null,
"url": "https://www.story.one/media/images/poop-4108423_1920.width-1600.format-jpeg.jpg"
},
"tags": ["big"],
"meta": {},
"content": {
"de": {
"created": null,
"updated": null,
"title": "Produktname",
"subtitle": "Short description"
}
}
}
],
"variations": [
{
"_id": null,
"created": null,
"updated": null,
"key": "color",
"type": "COLOR",
"options": [
{
"value": "ff0000",
"content": {
"de": {
"created": null,
"updated": null,
"title": "Rot",
"subtitle": ""
}
}
}
],
"content": {
"de": {
"created": null,
"updated": null,
"title": "Farbe",
"subtitle": "Farbvariante"
}
}
}
]
}
}
Entity Type: Assortment
Set by Unchained:
- authorId,
- slug history
- _id, created & updated if not provided
{
"entity": "ASSORTMENT",
"operation": "CREATE",
"payload": {
"_id": "A",
"specification": {
"created": null,
"updated": null,
"sequence": "",
"isActive": true,
"isBase": false,
"isRoot": true,
"tags": ["food"],
"meta": {},
"content": {
"de": {
"created": null,
"updated": null,
"title": "Groceries",
"slug": "groceries",
"subtitle": "Short description",
"description": "Long description"
}
}
},
"products": [
{
"_id": null,
"created": null,
"updated": null,
"productId": "A",
"tags": ["big"],
"meta": {}
}
],
"children": [
{
"_id": null,
"created": null,
"updated": null,
"assortmentId": "assortment2",
"tags": [],
"meta": {}
}
],
"filters": [
{
"_id": null,
"created": null,
"updated": null,
"filterId": "filter",
"tags": [],
"meta": {}
}
],
"media": [
{
"_id": null,
"created": null,
"updated": null,
"asset": {
"_id": null,
"url": "https://www.story.one/media/images/poop-4108423_1920.width-1600.format-jpeg.jpg"
},
"tags": ["big"],
"meta": {},
"content": {
"de": {
"created": null,
"updated": null,
"title": "assortmentName",
"subtitle": "Short description"
}
}
}
]
}
}
Entity Type: Filter
Set by Unchained:
- authorId,
- _id, created & updated if not provided
{
"entity": "FILTER",
"operation": "CREATE",
"payload": {
"_id": null,
"specification": {
"created": null,
"updated": null,
"key": "size_cm",
"isActive": true,
"type": "SINGLE_CHOICE",
"options": [
{
"value": "10",
"content": {
"de": {
"created": null,
"updated": null,
"title": "10 cm",
"subtitle": ""
}
}
}
],
"meta": {},
"content": {
"de": {
"created": null,
"updated": null,
"title": "Size",
"subtitle": "Size of product in centimeters"
}
}
}
}
}
Entity Type: Review (coming soon)
{
"entity": "REVIEW",
"operation": "CREATE",
"payload": {
"_id": null,
"specification": {
"created": null,
"updated": null,
"productId": "product"
"authorId": "root",
"rating": 1,
"title": "What the?",
"review": "That product just sucks big times",
"votes": [
{
"timestamp": null,
"userId": "root",
"type": "UPVOTE",
"meta": {}
}
],
"meta": {}
}
}
}
}