Data Import Module

Bulk import master data, pricing, and inventory from Excel/CSV files via an asynchronous queue-based pipeline.

Architecture

HTTP upload → FileParserService (xlsx/csv → rows[])
           → DataImportService.createJob() → importJob DB row (status=pending)
           → BullMQ queue ("import-processing")
           → ImportProcessingProcessor (worker)
               → ImportTypeRegistry.get(importType)
               → handler.validate(row)
               → handler.handle(row)       → upsert/create entity
               → importRow DB rows (success/error per row)
           → importJob status=completed

Layers (Hexagonal Architecture)

Layer	Path	Responsibility
Domain	domain/import-repository.domain.ts	Repository port (IImportRepository), param types
Application	application/	DataImportService (job lifecycle), TemplateGeneratorService, handler registry, 17 import handlers
Infrastructure	infrastructure/	ImportRepository (Kysely adapter), FileParserService (Excel/CSV parsing), ImportProcessingProcessor (BullMQ worker)
Interface	interfaces/	DataImportController (HTTP endpoints), DTOs

Dependency Flow

Controller → DataImportService → IImportRepository (port)
                               → FileParserService
                               → BullMQ Queue
Processor  → IImportRepository (port)
           → ImportTypeRegistry → ImportTypeHandler (per type)

Supported Import Types (17)

Type	Handler	Upsert Key	Dependencies
attribute	AttributeImportHandler	name	Attributes
brand	BrandImportHandler	name	Brands
category	CategoryImportHandler	name	Categories
color	ColorImportHandler	name	Colors
size	SizeImportHandler	name	Sizes
style	StyleImportHandler	name	Styles, Brands
model	ModelImportHandler	name	Models, Brands
product	ProductImportHandler	sku	Products, Categories, Brands, Colors, Sizes, Styles, Models, Currencies, Taxes, Inventories
product_with_variants	ProductWithVariantsImportHandler	name+category (product), sku (variant)	Products, ProductVariants, Categories, Brands, Currencies, Taxes, Inventories
opening_inventory	OpeningInventoryImportHandler	sku+location	Products, Locations, Inventories, InventoryLedgers
supplier	SupplierImportHandler	supplierCode or email	Suppliers
customer	CustomerImportHandler	customerCode or email	Customers
location	LocationImportHandler	name	Locations, Addresses
employee	EmployeeImportHandler	employeeNumber or email	Employees, Locations, Users
price_list	PriceListImportHandler	name	BusinessCurrencies
price_list_scope	PriceListScopeImportHandler	priceListId+locationId+channel	PriceLists, Locations
price_list_item	PriceListItemImportHandler	priceListId+itemType+identifier	PriceLists, Products

API Endpoints

Method	Path	Auth	Description
POST	/data-import/upload	Bearer	Upload file and create import job
GET	/data-import/jobs	Bearer	List jobs (paginated, filterable)
GET	/data-import/jobs/:id	Bearer	Get job status
GET	/data-import/jobs/:id/errors	Bearer	Get error rows (JSON or CSV)
POST	/data-import/preview	Bearer	Preview first 20 mapped rows
GET	/data-import/templates/:importType	Bearer	Download Excel template

Key Design Decisions

1. Handler Registry Pattern — Handlers are registered via NestJS factory provider into a Map<string, ImportTypeHandler>. Adding a new type requires only a handler file, schema entry, and module registration.

2. Upsert Semantics — All handlers follow find-by-natural-key → update-if-exists → create-if-not. Re-imports are safe and idempotent.

3. Async Processing — BullMQ with 3 retries and exponential backoff. Row-level idempotency via rowNumber tracking prevents duplicate processing on retry.

4. Batch Persistence — Row results are flushed to DB every 300 rows to balance memory and write efficiency.

5. Repository Token DI — IMPORT_REPOSITORY Symbol token in domain layer; infrastructure provides the Kysely implementation. Both service and processor depend on the IImportRepository interface, not the concrete class.

Related Docs

• Adding New Import Types — Step-by-step developer guide
• User Guide — End-user documentation
• API cURL Examples — cURL commands for all endpoints
• Pricing Import — Price list import specifics