vitepress-template

Appearance

Technology Stack ​

**Referenced Files in This Document** - [[bi-analysis/go.mod]](file/bi-analysis/go.mod) - [[bi-analysis/cmd/bi-analysis/main.go]](file/bi-analysis/cmd/bi-analysis/main.go) - [[bi-analysis/Dockerfile]](file/bi-analysis/dockerfile) - [[bi-common/go.mod]](file/bi-common/go.mod) - [[bi-common/registry/nacos/client.go]](file/bi-common/registry/nacos/client.go) - [[bi-common/mq/kafkax/consumer.go]](file/bi-common/mq/kafkax/consumer.go) - [[bi-common/database/starrocks/streamload/client.go]](file/bi-common/database/starrocks/streamload/client.go) - [[bi-common/observability/logger/logger.go]](file/bi-common/observability/logger/logger.go) - [[bi-proto/go.mod]](file/bi-proto/go.mod) - [[ui-web/package.json]](file/ui-web/package.json) - [[ui-web/next.config.ts]](file/ui-web/next.config.ts) - [[bi-chat/requirements.txt]](file/bi-chat/requirements.txt) - [[bi-chat/bi_chat/main.py]](file/bi-chat/bi-chat/main.py) - [[bi-chat/Dockerfile]](file/bi-chat/dockerfile)

Table of Contents ​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion

Introduction ​

This document describes the complete technology ecosystem of the BI Analysis Platform. It covers backend frameworks (Go Kratos, Protocol Buffers, gRPC), frontend technologies (Next.js 16, React, TypeScript), AI components (AgentScope, FastAPI, Milvus), and infrastructure (StarRocks OLAP, Apache Kafka, Nacos, APISIX). It also explains the rationale behind technology choices, version compatibility requirements, integration patterns, dependency management strategies, development tooling, deployment technologies, and cross-cutting concerns such as observability, monitoring, and security.

Project Structure ​

The platform is organized as a multi-service monorepo with:

  • Go microservices using Kratos framework and Protocol Buffers/gRPC
  • Python AI service built on FastAPI and AgentScope
  • Frontend applications using Next.js 16 with TypeScript
  • Shared libraries for common concerns (logging, metrics, registry, Kafka, StarRocks StreamLoad)
  • Infrastructure integrations (Nacos, Kafka, StarRocks, APISIX)

Diagram sources

Section sources

Core Components ​

  • Backend microservices: Built with Go Kratos, exposing gRPC and HTTP endpoints, configured via Nacos, and instrumented with OpenTelemetry logging.
  • Protocol and API definitions: Managed centrally in bi-proto using Protocol Buffers and validated with protoc-gen-validate.
  • Event streaming: Implemented with Apache Kafka via a reusable consumer library in bi-common.
  • Data ingestion: StarRocks StreamLoad client in bi-common supports CSV/JSON ingestion with retry and redirect handling.
  • AI assistant service: FastAPI-based chat service powered by AgentScope, with optional multi-agent collaboration and Milvus vector storage.
  • Frontend: Next.js 16 application with TypeScript, Arco Design, and configurable API proxying to backend services.

Section sources

Architecture Overview ​

The platform follows a distributed microservices architecture:

  • Services communicate via gRPC and HTTP, with Protocol Buffers for schema evolution.
  • Nacos provides service registration/discovery and centralized configuration.
  • Kafka handles asynchronous event streaming across services.
  • StarRocks serves as the analytical data store with StreamLoad for batch ingestion.
  • APISIX acts as the API gateway routing traffic to services.
  • Observability is implemented through structured logging and OpenTelemetry.

Diagram sources

Detailed Component Analysis ​

Backend Microservices (Go Kratos + gRPC + Protobuf) ​

  • Framework and transport: Kratos v2 with HTTP and gRPC servers; automaxprocs for CPU tuning.
  • Protobuf and gRPC: Defined in bi-proto; validated with protoc-gen-validate; gateway support via grpc-gateway.
  • Configuration: Loaded from Nacos at runtime; supports dev/test/prod environments.
  • Observability: Structured logging via bi-common logger; OpenTelemetry SDK available in dependencies.
  • Deployment: Multi-stage Docker build with Alpine base, timezone configured, and exposed ports for gRPC and HTTP.

Diagram sources

Section sources

Protocol Buffers and gRPC ​

  • Centralized protobuf definitions under bi-proto.
  • Validation via protoc-gen-validate; generated code consumed by services.
  • Gateway support for REST-to-gRPC translation.

Section sources

Frontend (Next.js 16 + React + TypeScript) ​

  • Next.js 16 with React 19 and TypeScript.
  • Environment-aware configuration and API proxying to backend services.
  • UI libraries include Arco Design, ECharts, AG Grid, and others.

Diagram sources

Section sources

AI Assistant Service (FastAPI + AgentScope + Milvus) ​

  • FastAPI service with AgentScope multi-agent orchestration.
  • Supports legacy and modern chat protocols; includes session lifecycle management and periodic cleanup.
  • Dependencies include pymilvus for vector storage, SQLAlchemy/asyncpg for databases, Redis, and Pydantic settings.

Diagram sources

Section sources

Infrastructure Integrations ​

Nacos (Service Discovery + Configuration) ​

  • Provides config center and naming client APIs.
  • Supports environment-specific config loading and change listeners.

Section sources

Apache Kafka (Event Streaming) ​

  • Robust consumer with SASL/TLS support, partition modes, batching, and graceful shutdown.
  • Used for decoupled event processing across services.

Section sources

StarRocks (OLAP + StreamLoad) ​

  • HTTP-based StreamLoad client supporting CSV/JSON, retries, and BE proxy redirection.
  • Designed for high-throughput batch data ingestion.

Section sources

APISIX (API Gateway) ​

  • Used for routing and load balancing to backend services.
  • Configuration artifacts present in the repository for gateway setup.

Section sources

Observability, Monitoring, and Security ​

  • Logging: Structured logging with file rotation and optional Aliyun log forwarding; configurable levels and formats.
  • Metrics: Prometheus client included in dependencies.
  • Tracing: OpenTelemetry SDK and contrib packages available.
  • Security: JWT utilities and RBAC helpers in bi-common; TLS/SASL support in Kafka; basic auth in StarRocks StreamLoad.

Section sources

Dependency Analysis ​

  • Go services depend on bi-common and bi-proto via replace directives for local development.
  • bi-common aggregates Kratos, gRPC, MySQL, Kafka, Redis, Prometheus, OpenTelemetry, and YAML parsing.
  • Frontend depends on Next.js 16, React 19, TypeScript, and UI libraries.
  • AI service depends on FastAPI, AgentScope, Milvus, and related Python packages.

Diagram sources

Section sources

Performance Considerations ​

  • Use Kratos HTTP/gRPC servers with automaxprocs for optimal CPU utilization.
  • Kafka consumers support batching and partitioned consumption; tune MinBytes/MaxBytes and CommitInterval for throughput.
  • StarRocks StreamLoad supports retries and BE proxy redirection; configure MaxRetry and Timeout appropriately.
  • Frontend builds with standalone output and optimized package imports to reduce bundle size.

[No sources needed since this section provides general guidance]

Troubleshooting Guide ​

  • Logging: Verify log level and output configuration; check file rotation settings and Aliyun forwarding.
  • Kafka: Confirm broker connectivity, SASL/TLS settings, and consumer group/partition configuration; monitor lag and stats.
  • StarRocks: Validate credentials, host/port, and BE proxy settings; inspect StreamLoad response codes.
  • Nacos: Ensure namespace/group/dataId are correct; confirm client credentials and network reachability.
  • Frontend: Check NEXT_PUBLIC_APP_ENV and proxy rewrites; confirm backend service availability.

Section sources

Conclusion ​

The BI Analysis Platform leverages a cohesive technology stack: Go Kratos microservices with Protocol Buffers/gRPC, a centralized frontend built on Next.js 16, an AI assistant powered by FastAPI and AgentScope, and robust infrastructure integrations including StarRocks, Kafka, Nacos, and APISIX. The design emphasizes modularity, observability, and scalability, with clear separation of concerns and shared libraries for common capabilities.