Analytics Engine Overview

**Referenced Files in This Document** - [[main.go]](file/bi-analysis/cmd/bi-analysis/main.go) - [[wire.go]](file/bi-analysis/cmd/bi-analysis/wire.go) - [[README.md]](file/bi-analysis/readme.md) - [[conf.proto]](file/bi-analysis/internal/conf/conf.proto) - [[data.go]](file/bi-analysis/internal/data/data.go) - [[server.go]](file/bi-analysis/internal/server/server.go) - [[http.go]](file/bi-analysis/internal/server/http.go) - [[grpc.go]](file/bi-analysis/internal/server/grpc.go) - [[biz.go]](file/bi-analysis/internal/biz/biz.go) - [[service.go]](file/bi-analysis/internal/service/service.go) - [[query.go]](file/bi-analysis/internal/biz/query.go) - [[sql_builder.go]](file/bi-analysis/internal/biz/sql-builder.go) - [[table.go]](file/bi-analysis/internal/biz/table.go) - [[template.go]](file/bi-analysis/internal/biz/template.go) - [[query_repo.go]](file/bi-analysis/internal/data/query.go) - [[application-dev.yaml]](file/bi-analysis/configs/application-dev.yaml)

Table of Contents

1. Introduction
2. Project Structure
3. Core Components
4. Architecture Overview
5. Detailed Component Analysis
6. Dependency Analysis
7. Performance Considerations
8. Troubleshooting Guide
9. Conclusion

Introduction

The bi-analysis analytics engine is the central processing component of the BI platform. It orchestrates analytical queries, manages dimensional and metric configurations, and exposes both HTTP and gRPC APIs for clients and other services. Built on the Kratos framework and integrated with bi-common modules, it provides unified logging, configuration via Nacos, distributed tracing, and standardized API responses. Its primary responsibilities include:

• Accepting analytical requests with rows, columns, metrics, filters, and pagination
• Resolving field and dimension metadata
• Building optimized SQL via a sophisticated CTE-based builder
• Executing queries against StarRocks/MySQL and returning formatted results
• Managing custom tables and templates for analysts
• Enforcing remote authentication via bi-tenant and integrating with frontend dashboards

Project Structure

The bi-analysis service follows a layered architecture with clear separation of concerns:

• cmd/bi-analysis: Application entrypoint and dependency injection wiring
• internal/biz: Business logic layer encapsulating use cases and domain entities
• internal/data: Data access layer with repositories and persistence utilities
• internal/server: HTTP/gRPC server configuration and middlewares
• internal/service: Service layer exposing business capabilities to clients
• configs: Environment-specific Nacos configuration sources
• api: Protocol buffers defining HTTP/gRPC contracts

Diagram sources

• [main.go]
• [wire.go]
• [http.go]
• [grpc.go]
• [server.go]
• [service.go]
• [biz.go]
• [data.go]
• [query.go]
• [sql_builder.go]
• [table.go]
• [template.go]
• [query_repo.go]
• [conf.proto]
• [application-dev.yaml]

Section sources

• [README.md]
• [main.go]
• [wire.go]

Core Components

• Application bootstrap and DI wiring:
  • Entry initializes Kratos app, loads Nacos configuration, sets up logging and snowflake ID generator, creates service registry, and wires components via Wire.
• Server configuration:
  • HTTP server registers both admin and client services, applies tracing, request ID, logging, recovery, and remote auth middleware.
  • gRPC server registers shared services and injects user context from metadata.
• Data layer:
  • Initializes database and optional Redis clients, provides transaction support, and exposes GetDB with transaction-awareness.
• Business logic:
  • Query use case parses requests, resolves fields/dimensions, loads formula dependencies, builds SQL, executes, and formats results.
  • Table use case manages custom table creation, physical DDL, and DIY data CRUD.
  • Template use case handles saved templates and field visibility configurations.
• Service layer:
  • Exposes HTTP/gRPC handlers wrapping use cases for clients and internal consumers.

Section sources

• [main.go]
• [wire.go]
• [http.go]
• [grpc.go]
• [data.go]
• [query.go]
• [table.go]
• [template.go]

Architecture Overview

The analytics engine integrates tightly with the broader BI ecosystem:

• Frontend dashboards call HTTP endpoints for previewing queries and retrieving SQL.
• Internal services communicate via gRPC using shared protobuf definitions.
• Authentication is enforced remotely by bi-tenant; HTTP middleware validates tokens and injects user context.
• Data retrieval leverages bi-basic for shop dictionaries and StarRocks/MySQL for analytics.

Diagram sources

• [http.go]
• [grpc.go]
• [server.go]
• [data.go]

Detailed Component Analysis

Initialization and Dependency Injection

• Bootstrap flow:
  • Parses environment flag, loads Nacos config source, scans Bootstrap config, initializes logger and snowflake, constructs registry, and invokes Wire to assemble the Kratos app.
• Wire assembly:
  • Provider sets from server, data, biz, and service packages are composed into a single app with HTTP and gRPC servers.

Diagram sources

• [main.go]
• [wire.go]

Section sources

• [main.go]
• [wire.go]
• [conf.proto]
• [application-dev.yaml]

HTTP Server and Middleware Pipeline

• Middleware stack:
  • Tracing, RequestID, structured logging, recovery, and remote auth middleware.
• Route registration:
  • Admin services under anls-m/v1 and client services under anls/v1.
  • Specialized handlers for array-returning endpoints and empty-string fixes for repeated fields.

Diagram sources

• [http.go]
• [server.go]

Section sources

• [http.go]
• [server.go]

gRPC Server and Metadata Propagation

• Middleware pipeline:
  • Tracing, metadata propagation, user context extraction, request ID, logging, recovery.
• Service registration:
  • Uses shared protobuf definitions to expose template and field dictionary services.

Diagram sources

• [grpc.go]

Section sources

• [grpc.go]

Data Access and Transactions

• Initialization:
  • Sets up database client and optional Redis client; warns on Redis failure.
• Transaction support:
  • Provides Transaction interface and InTx wrapper to propagate a transaction handle via context.
• Repository pattern:
  • Query repository executes raw SQL, maps rows to maps, and resolves fields/dimensions.
  • Checks table existence with Redis caching and database fallback.

Diagram sources

• [data.go]
• [query_repo.go]

Section sources

• [data.go]
• [query_repo.go]

Business Logic: Query Execution

• Request parsing:
  • Supports rows, columns, metrics, runtime field configs, filters, pagination, ranking, and summation modes.
• Formula dependency resolution:
  • Recursively loads referenced fields for formula expressions.
• Shop resolution:
  • Detects order/finance-related fields and fetches shop lists from bi-basic; validates order table existence.
• SQL building:
  • Categorizes fields (base/formula/diy/order), generates CTEs, and constructs optimized queries with date series and ranking.

Diagram sources

• [query.go]
• [sql_builder.go]

Section sources

• [query.go]
• [query.go]
• [sql_builder.go]
• [sql_builder.go]

Business Logic: Custom Tables and Templates

• Table use case:
  • Validates dimensions, prevents duplicate combinations, generates physical table names, creates DDL, and persists configuration.
  • Supports DIY data upsert and paginated listing.
• Template use case:
  • Manages template creation/update/deletion, name uniqueness checks, sorting updates, and cascading cleanup of field visibility configs.

Diagram sources

• [table.go]

Section sources

• [table.go]
• [table.go]
• [template.go]
• [template.go]

Dependency Analysis

• Internal dependencies:
  • Server depends on service layer; service layer depends on biz; biz depends on data; data depends on bi-common components (gormx, redisx, snowflake).
• External integrations:
  • Nacos for configuration and service registry
  • bi-basic for shop dictionaries
  • bi-tenant for remote token verification
• Configuration sources:
  • Environment-specific YAML files define Nacos connection and data IDs.

Diagram sources

• [main.go]
• [wire.go]
• [server.go]
• [data.go]
• [biz.go]
• [service.go]
• [http.go]
• [grpc.go]
• [query_repo.go]
• [query.go]
• [sql_builder.go]
• [table.go]
• [template.go]

Section sources

• [main.go]
• [wire.go]
• [application-dev.yaml]

Performance Considerations

• SQL generation:
  • Categorization of fields and targeted CTE construction reduce complexity; date-series handling ensures sparse time-series completeness.
• Caching:
  • Redis used for table existence checks to avoid frequent information_schema queries.
• Pagination and ranking:
  • Dense rank and window functions enable efficient top-N and ranking scenarios.
• Observability:
  • Structured logging, tracing, and recovery middleware aid in diagnosing slow queries and failures.
• Scalability:
  • Stateless HTTP/gRPC servers scale horizontally; centralized Nacos configuration supports environment-driven tuning.

[No sources needed since this section provides general guidance]

Troubleshooting Guide

• Authentication failures:
  • Remote auth middleware returns unauthorized when token is missing or invalid; verify bi-tenant connectivity and token validity.
• Empty repeated strings:
  • HTTP handlers preprocess JSON to convert empty strings to null for proto compatibility.
• Table existence checks:
  • Redis cache accelerates repeated checks; fallback to information_schema ensures correctness.
• Logging and tracing:
  • Enable structured logs and trace correlation IDs to locate bottlenecks.

Section sources

• [server.go]
• [http.go]
• [query_repo.go]

Conclusion

The bi-analysis analytics engine serves as the analytical backbone of the BI platform. Through a clean layered architecture, robust DI wiring, and strong integration with Nacos, bi-basic, and bi-tenant, it delivers secure, scalable, and observable analytics capabilities. Its SQL builder and repository patterns enable flexible query composition, while its service and middleware layers ensure consistent behavior across HTTP and gRPC clients.