External System Integrations

**Referenced Files in This Document** - [[main.go]](file/bi-basic/cmd/server/main.go) - [[External_gRPC_Calls.md]](file/bi-basic/docs/external-grpc-calls.md) - [[gRPC_Call_Principle.md]](file/bi-basic/docs/grpc-call-principle.md) - [[shop_auth.go]](file/bi-basic/internal/biz/logic/shop-auth.go) - [[consumer.go]](file/bi-basic/internal/kafka/consumer/consumer.go) - [[producer.go]](file/bi-basic/internal/kafka/producer/producer.go) - [[config.go]](file/bi-basic/internal/kafka/config/config.go) - [[consumer.go]](file/bi-common/mq/kafkax/consumer.go) - [[producer.go]](file/bi-common/mq/kafkax/producer.go) - [[retry.go]](file/bi-common/mq/kafkax/retry.go) - [[client.go]](file/bi-common/registry/nacos/client.go) - [[auth.go]](file/bi-api-jushuitan/common/sdkhelper/auth.go)

Table of Contents

1. Introduction
2. Project Structure
3. Core Components
4. Architecture Overview
5. Detailed Component Analysis
6. Dependency Analysis
7. Performance Considerations
8. Troubleshooting Guide
9. Conclusion

Introduction

This document explains how bi-basic integrates with external systems and services, focusing on:

• Kafka event processing architecture and message consumption patterns
• Real-time data synchronization mechanisms
• gRPC client implementations for external API calls, including authentication, error handling, and retry strategies
• Integrations with Leke and Jushuitan external platforms
• Event-driven architecture patterns, message serialization, and distributed system communication protocols used in the foundational layer

Project Structure

The integration surface spans three primary areas:

• Kafka messaging: producers and consumers for order, refund, and product events
• gRPC clients: invoking bi-api-leke for external platform data
• External SDKs: Jushuitan OAuth and token exchange flows

Diagram sources

• [main.go]
• [consumer.go]
• [producer.go]
• [shop_auth.go]
• [config.go]
• [consumer.go]
• [client.go]

Section sources

• [main.go]
• [config.go]

Core Components

• Kafka producers and consumers: emit and process order-combined, after-sale, and goods-sku events with batching and concurrency
• gRPC client to bi-api-leke: resolves service via Nacos discovery and calls remote methods with timeouts and error handling
• External platform integrations:
  • Leke (Taobao): gRPC calls to fetch user info and product details
  • Jushuitan: OAuth authorization URL building and token exchange via HTTP client

Section sources

• [producer.go]
• [consumer.go]
• [shop_auth.go]
• [auth.go]

Architecture Overview

The system follows an event-driven pattern:

• Producers serialize domain events to JSON and publish to Kafka topics
• Consumers subscribe to topics, process batches concurrently, and commit offsets upon successful handling
• gRPC clients integrate with bi-api-leke for external data, leveraging Nacos-based service discovery

Diagram sources

• [main.go]
• [producer.go]
• [consumer.go]
• [shop_auth.go]
• [client.go]

Detailed Component Analysis

Kafka Event Processing Architecture

• Topics and models:
  • order-combined-events, after-sale-order-events, goods-sku-events
  • Structs for OrderCombined, AfterSaleOrder, GoodsWithSkuList, and KafkaData define serialization contracts
• Producer capabilities:
  • Single and batch sends with JSON serialization
  • Raw byte sends for specialized payloads
  • Async writes with configurable batch sizes and timeouts
• Consumer capabilities:
  • Multi-topic support with separate consumer groups per topic
  • Worker pools with channel-based dispatch and dedicated offset commit goroutines
  • Batch collection bounded by minimum message count and maximum bytes
  • Graceful shutdown and health checks

Diagram sources

• [consumer.go]
• [consumer.go]
• [consumer.go]

Section sources

• [config.go]
• [config.go]
• [producer.go]
• [producer.go]
• [consumer.go]
• [consumer.go]
• [consumer.go]
• [consumer.go]

gRPC Client to bi-api-leke

• Service discovery: Kratos with Nacos registry resolves bi-api-leke.grpc
• Authentication: Uses insecure dial with discovery endpoint and configured timeout
• Error handling: Validates discovery presence, wraps connection and RPC errors
• Timeouts: Separate connection and RPC timeouts tailored to endpoint latency
• Business usage: Calls TaobaoGetUserInfo during shop authorization flows

Diagram sources

• [shop_auth.go]
• [gRPC_Call_Principle.md]
• [External_gRPC_Calls.md]

Section sources

• [shop_auth.go]
• [gRPC_Call_Principle.md]
• [External_gRPC_Calls.md]

External Platform Integrations

Jushuitan OAuth and Token Exchange

• Builds authorization URLs with signed parameters
• Exchanges authorization code for access tokens and refresh tokens
• Supports initial token retrieval for self-service scenarios
• Encodes form parameters and signs requests

Diagram sources

• [auth.go]
• [auth.go]

Section sources

• [auth.go]

Leke (Taobao) Integration

• bi-basic invokes bi-api-leke via gRPC to:
  • Fetch Taobao user info for shop authorization
  • Sync product details and SKU lists on demand
• Internal flows:
  • On shop authorization creation, call TaobaoGetUserInfo and persist shop records
  • Manual sync triggers TaobaoItemSellerGoodsDetails for SKU updates

Diagram sources

• [shop_auth.go]
• [External_gRPC_Calls.md]

Section sources

• [External_gRPC_Calls.md]
• [shop_auth.go]

Message Serialization and Protocols

• Kafka producers serialize payloads to JSON for most topics
• Producers also support raw byte sends for specialized use cases
• Consumers receive messages and process batches; offsets are committed after successful handling
• Optional retry/backoff strategies are available via shared utilities

Diagram sources

• [config.go]
• [config.go]

Section sources

• [producer.go]
• [producer.go]
• [consumer.go]
• [retry.go]

Dependency Analysis

• Application bootstrap initializes Nacos-based service discovery and starts Kafka consumers
• Kafka utilities provide robust consumer/producer abstractions with health checks and graceful shutdown
• gRPC clients depend on Kratos discovery and Nacos for service resolution
• External SDKs encapsulate platform-specific authentication flows

Diagram sources

• [main.go]
• [consumer.go]
• [producer.go]
• [client.go]

Section sources

• [main.go]
• [client.go]

Performance Considerations

• Kafka batching: Tune BatchMinMessages and BatchMaxBytes to balance throughput and latency
• Worker pools: Adjust Workers to match CPU and IO capacity; ensure backpressure via buffered channels
• Offsets: Manual commits reduce duplication risk but require reliable error handling
• gRPC timeouts: Separate connection and RPC timeouts prevent resource starvation
• TLS: Enable TLS in KafkaConfig when required by cluster policy

Troubleshooting Guide

• Kafka connectivity:
  • Use Ping methods on producer/consumer to validate broker reachability
  • Verify Brokers list and topic existence
• Consumer lag and rebalancing:
  • Monitor Lag and ReaderStats; adjust SessionTimeout and HeartbeatInterval
  • Ensure dedicated consumer groups per topic to avoid cross-consumption
• gRPC failures:
  • Confirm Nacos registration of bi-api-leke.grpc
  • Validate discovery endpoint and network policies
  • Inspect error wrapping for connection vs RPC errors
• Jushuitan OAuth:
  • Ensure correct app credentials and signed parameters
  • Respect code expiration windows and token refresh cadence

Section sources

• [consumer.go]
• [producer.go]
• [shop_auth.go]
• [auth.go]

Conclusion

bi-basic implements a robust, event-driven integration layer:

• Kafka producers and consumers enable scalable, asynchronous processing of orders, refunds, and product data
• gRPC clients integrate with bi-api-leke using Kratos/Nacos for service discovery and timeouts
• External platform integrations (Leke and Jushuitan) are encapsulated with clear authentication and error handling patterns
• The architecture supports concurrency, reliability, and operational visibility through standardized utilities and logging