vitepress-template

Appearance

Order Data Synchronization ​

**Referenced Files in This Document** - [[order.go]](file/bi-api-leke/internal/service/consumer-handlers/order.go) - [[order.go]](file/bi-api-leke/internal/biz/usecase/order.go) - [[order.go]](file/bi-api-leke/internal/service/order.go) - [[order.go]](file/bi-api-leke/internal/data/message/order.go) - [[client.go]](file/bi-api-leke/common/leke-helper/taobao/client.go) - [[order.go]](file/bi-basic/internal/service/order.go) - [[order.go]](file/bi-basic/internal/biz/logic/order.go) - [[order.go]](file/bi-basic/api/order/v1/order.pb.go)

Table of Contents ​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion

Introduction ​

This document describes the order data synchronization system that integrates with the Taobao ecosystem via the Leke API gateway. It covers:

  • Incremental order fetching using the OrderIncrement API with time-based filtering and pagination
  • Full order retrieval for historical coverage
  • Order lifecycle management from creation to completion, including status tracking and modification detection
  • Data transformation pipeline from Taobao API responses to internal order models
  • Order status mapping, payment processing, shipping tracking, and refund integration
  • Synchronization scheduling, batch processing strategies, and conflict resolution
  • Error handling for partial failures, duplicate detection, and data consistency validation

Project Structure ​

The order synchronization spans two primary modules:

  • bi-api-leke: Orchestrates synchronization, Kafka messaging, Taobao API integration, and downstream batching
  • bi-basic: Provides internal order storage, status mapping, and query APIs

Diagram sources

Section sources

Core Components ​

  • UseCase layer: Builds Kafka messages for full and incremental order synchronization, controls concurrency, and batches requests
  • Consumer Handlers: Consumes Kafka messages, validates DataKey, calls Taobao APIs, parses responses, and emits combined order and after-sale messages
  • Taobao Client: Encapsulates API calls (OrderIncrement, GetOrder1, GetOrderInfo, OrderRefundsGet) with retries and logging
  • Message Types: Define payload structures for full and incremental order fetch tasks
  • Order Service and Logic: Expose internal order APIs and persist transformed order data with status mapping and filtering

Section sources

Architecture Overview ​

The system follows a producer-consumer pattern:

  • UseCase producers emit Kafka messages for full/incremental order fetches
  • Consumer Handlers validate DataKey, call Taobao APIs, transform data, and send combined order and after-sale messages
  • Internal services receive messages and persist to the order model

Diagram sources

Detailed Component Analysis ​

Incremental Order Fetching with OrderIncrement API ​

  • Time-based filtering: The UseCase constructs messages with start and end times for the current day; the Taobao client enforces a one-day window constraint
  • Pagination: Uses page number and page size to iterate through results; total results parsed to compute total pages
  • Concurrency control: Limits concurrent shop processing via a semaphore
  • DataKey verification: Validates DataKey against the seller nickname before API calls

Diagram sources

Section sources

Full Order Retrieval Workflow ​

  • Historical coverage: Retrieves orders for the last month using GetOrder1 with date range
  • Page calculation: Computes total pages from total results and emits per-page messages
  • Parallelization: Processes multiple shops concurrently with controlled concurrency

Diagram sources

Section sources

Data Transformation Pipeline ​

  • From Taobao API responses to internal models:
    • Incremental/full order lists parsed into intermediate trade structures
    • Child orders extracted and enriched with parent order info
    • Optional refunds fetched and mapped into after-sale models
    • Transformed into combined order messages and after-sale messages for downstream consumption

Diagram sources

Section sources

Order Lifecycle Management and Status Tracking ​

  • Creation to completion: Orders are created with payment time, platform, shop, and product identifiers; statuses tracked via platform-specific values
  • Modification detection: Incremental fetching relies on modified time windows; combined order messages carry order and sub-order IDs for updates
  • Status mapping: Platform values mapped to human-readable names in the order display service

Diagram sources

Section sources

Payment Processing, Shipping Tracking, and Refund Integration ​

  • Payment: Captured via payment amount and payment time from Taobao responses
  • Shipping: Logistics company and invoice number mapped into delivery fields
  • Refunds: Optional refund details fetched when present; emitted as after-sale order messages

Diagram sources

Section sources

Synchronization Schedule and Batch Processing ​

  • Scheduling: UseCase exposes RPC endpoints to trigger full and incremental synchronization asynchronously
  • Batch sizes: Fixed page sizes for API calls; configurable batch size for Kafka sends
  • Concurrency: Controlled via a fixed-size semaphore to avoid overload
  • Request logging: Each API call logged with request parameters, response parameters, and processing time

Diagram sources

Section sources

Conflict Resolution and Data Consistency Validation ​

  • Duplicate detection: Kafka message keys include task IDs; consumers validate DataKey and skip invalid entries
  • Partial failures: Errors recorded in request logs; consumers continue processing remaining items
  • Consistency: Combined order messages include order/sub-order IDs and timestamps; after-sale messages link to orders

Section sources

Dependency Analysis ​

  • bi-api-leke depends on:
    • Taobao client for API calls
    • Kafka producer for emitting messages
    • gRPC clients for downstream services
  • bi-basic depends on:
    • Order logic for persistence and queries
    • Database abstraction for dynamic table routing

Diagram sources

Section sources

Performance Considerations ​

  • Concurrency limits: UseCase employs a semaphore to cap concurrent shop processing
  • Retry strategy: Taobao client uses retryable HTTP client with timeouts
  • Pagination sizing: Fixed page sizes balance throughput and latency
  • Logging overhead: Request logs capture timing and errors for diagnostics

[No sources needed since this section provides general guidance]

Troubleshooting Guide ​

Common issues and resolutions:

  • DataKey validation failure: Verify seller nicknames and DataKey correctness before API calls
  • API rate limits/throttling: Implement backoff and retry; monitor request logs for repeated failures
  • Partial failures during batch processing: Inspect request logs and reprocess failed pages
  • Duplicate messages: Ensure Kafka message keys are unique and consumer deduplicates by DataKey/task ID
  • Time window constraints: Incremental fetch must respect one-day window; adjust start/end times accordingly

Section sources

Conclusion ​

The order synchronization system provides robust, time-based incremental and historical order ingestion from Taobao via the Leke API gateway. It transforms raw API responses into unified internal order models, supports refund integration, and persists data through a scalable consumer-driven pipeline. With built-in validation, logging, and concurrency controls, it ensures reliability and maintainability for ongoing order lifecycle management.