vitepress-template

Appearance

Integration Patterns and External Systems ​

**Referenced Files in This Document** - [[main.go]](file/bi-api-leke/cmd/bi-api-leke/main.go) - [[main.go]](file/bi-api-jushuitan/cmd/bi-api-jushuitan/main.go) - [[client.go]](file/bi-api-leke/common/leke-helper/taobao/client.go) - [[client.go]](file/bi-api-jushuitan/common/sdkhelper/client.go) - [[application-dev.yaml]](file/bi-api-leke/configs/application-dev.yaml) - [[application-dev.yaml]](file/bi-api-jushuitan/configs/application-dev.yaml) - [[root.go]](file/bi-cron/cmd/root.go) - [[main.go]](file/bi-cron/main.go) - [[main.py]](file/bi-chat-asktable/app/main.py) - [[nacos-config.yaml]](file/bi-chat-asktable/configs/nacos-config.yaml) - [[server.py]](file/mcp-server-starrocks/src/mcp-server-starrocks/server.py) - [[pyproject.toml]](file/mcp-server-starrocks/pyproject.toml) - [[client.go]](file/bi-api-leke/internal/data/client/client.go) - [[client.go]](file/bi-api-jushuitan/internal/data/client/client.go)

Table of Contents ​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction ​

This document explains integration patterns and external system connectivity across the BI platform. It covers:

  • Adapter-style integrations with external systems (Leke API, Jushuitan ERP, Taobao advertising platforms)
  • AskTable NL2SQL integration and Model Context Protocol (MCP) implementation for AI agent database access
  • Security measures for external integrations (authentication, authorization, data sanitization)
  • Webhook implementations, polling mechanisms, and real-time data synchronization strategies
  • Error handling, retry policies, and circuit breaker patterns for external dependencies
  • Configuration management for multiple environments and tenant-specific integration settings
  • Sandboxing approaches for AI agent access to sensitive data sources

Project Structure ​

The integration surface spans multiple services:

  • bi-api-leke: Adapts to Taobao ecosystem APIs and streams data via Kafka
  • bi-api-jushuitan: Adapts to Jushuitan ERP SDK and rate-limits requests
  • bi-chat-asktable: Exposes AskTable integration with row-level access control
  • mcp-server-starrocks: Implements MCP for secure AI agent access to StarRocks
  • bi-cron: Orchestrates scheduled tasks and polling for external systems

Diagram sources

Section sources

Core Components ​

  • Adapter clients for external systems:
    • Leke/Taobao: HTTP client with retry and logging
    • Jushuitan: SDK-style client with signing and structured responses
  • Message-driven ingestion:
    • Kafka consumers for real-time event processing
  • Configuration via Nacos with environment-specific data IDs
  • Scheduled polling via bi-cron
  • AskTable integration with row-level access control
  • MCP server for AI agents to query StarRocks securely

Section sources

Architecture Overview ​

The system integrates external systems through adapters and asynchronous messaging, while exposing controlled access to analytics data for AI agents.

Diagram sources

Detailed Component Analysis ​

Leke API Adapter (Taobao Ecosystem) ​

  • Purpose: Integrate with Taobao advertising platform APIs via an HTTP adapter with retry and logging.
  • Authentication: Partner ID and signature included in requests.
  • Error handling: Logs failures and returns structured responses.
  • Retry policy: Built-in retry via HTTP client wrapper.

Diagram sources

Section sources

Jushuitan ERP Adapter ​

  • Purpose: Integrate with Jushuitan ERP via SDK-style client with standardized signing and response handling.
  • Authentication: AppKey, AccessToken, and MD5 signature computed from sorted parameters.
  • Rate limiting: Implemented via concurrency and token bucket limiters.
  • Error handling: Structured APIError and detailed logging.

Diagram sources

Section sources

AskTable NL2SQL Integration ​

  • Purpose: Provide AskTable integration with row-level access control and Nacos-backed configuration.
  • Configuration: Externalized via Nacos data ID with AskTable API credentials and datasource metadata.
  • Access control: Enforced at the service boundary to restrict data exposure.

Diagram sources

Section sources

MCP (Model Context Protocol) Implementation for AI Agent Database Access ​

  • Purpose: Securely expose StarRocks capabilities to AI agents via MCP with tools for queries, schema introspection, and plotting.
  • Transport modes: stdio, SSE, HTTP with CORS middleware for web frontends.
  • Safety: Expression validation for plotting, connection health checks, and structured tool outputs.

Diagram sources

Section sources

Real-Time Data Synchronization and Polling ​

  • Kafka consumers: Both Leke and Jushuitan services start consumer managers configured from Nacos-backed bootstrap configuration.
  • Polling: bi-cron orchestrates scheduled tasks for periodic synchronization with external systems.

Diagram sources

Section sources

Dependency Analysis ​

  • Configuration dependencies:
    • Nacos data IDs for common and service-specific configurations
    • Environment-specific YAML files for dev/test/prod
  • Service wiring:
    • Wire provider sets for Kafka, gRPC, and streamload clients
  • External dependencies:
    • HTTP clients with retry
    • Kafka consumer manager
    • StarRocks via MCP

Diagram sources

Section sources

Performance Considerations ​

  • Asynchronous ingestion: Kafka decouples producers from consumers, enabling backpressure handling and horizontal scaling.
  • Retry and timeouts: HTTP clients configured with retry and bounded timeouts reduce transient failure impact.
  • Rate limiting: Jushuitan adapter employs concurrency and token bucket limiters to respect upstream quotas.
  • Streaming: Streamload clients minimize round trips for bulk data ingestion.

[No sources needed since this section provides general guidance]

Troubleshooting Guide ​

  • External API errors:
    • Log structured responses and status codes; inspect APIError for Jushuitan and HTTP client wrappers for Leke.
  • Authentication failures:
    • Verify partner/signature for Taobao and AppKey/AccessToken/MD5 signature for Jushuitan.
  • Configuration drift:
    • Confirm Nacos data IDs and namespaces; ensure environment-specific YAML is loaded.
  • Connectivity issues:
    • Check Kafka consumer manager initialization and broker connectivity.
  • MCP tool failures:
    • Review expression validation for plotting and connection health checks.

Section sources

Conclusion ​

The platform employs robust adapter patterns for external integrations, asynchronous ingestion via Kafka, and controlled access for AI agents. Security is addressed through authentication, authorization boundaries, and data sanitization. Operational reliability is ensured by retry policies, rate limiting, and health monitoring. Configuration is centralized via Nacos with environment-specific overrides. These patterns enable scalable, maintainable, and secure integration with external systems.

[No sources needed since this section summarizes without analyzing specific files]

Appendices ​

Security Measures for External Integrations ​

  • Authentication:
    • Taobao: Partner ID and signature appended to requests
    • Jushuitan: AppKey, AccessToken, and MD5 signature computed from sorted parameters
  • Authorization:
    • AskTable integration enforces row-level access control at the service boundary
  • Data sanitization:
    • Expression validation for plotting to prevent arbitrary code execution
    • Structured logging with sensitive fields redacted

Section sources

Configuration Management Across Environments ​

  • Nacos data IDs:
    • bi-common.yaml and service-specific YAMLs (e.g., bi-api-leke.yaml, bi-api-jushuitan.yaml)
  • Environment selection:
    • Dev/test/prod via environment flag and corresponding YAML files
  • AskTable configuration:
    • Externalized via Nacos with API URL, API key, bot ID, and datasource ID

Section sources

Sandbox Approach for AI Agent Access ​

  • MCP server runs in-process with:
    • Connection health checks
    • CORS-enabled HTTP transport for web frontends
    • Structured tool outputs and optional chart generation
  • Plotting safety:
    • AST-based validation of expressions to prevent arbitrary code execution

Section sources