vitepress-template

Appearance

Service Communication Protocols ​

**Referenced Files in This Document** - [[Makefile]](file/bi-proto/makefile) - [[go.mod]](file/bi-proto/go.mod) - [[shop.proto]](file/bi-proto/basic/v1/shop.proto) - [[category.proto]](file/bi-proto/basic/v1/category.proto) - [[client.go]](file/bi-common/grpcx/client.go) - [[response.go]](file/bi-common/apitypes/response.go) - [[error.go]](file/bi-common/apitypes/error.go) - [[config.go]](file/bi-common/mq/kafkax/config.go) - [[producer.go]](file/bi-common/mq/kafkax/producer.go) - [[consumer.go]](file/bi-common/mq/kafkax/consumer.go) - [[kafka_http.pb.go]](file/bi-proto/kafka/v1/kafka-http.pb.go) - [[kafka_grpc.pb.go]](file/bi-proto/kafka/v1/kafka-grpc.pb.go) - [[kafka.proto]](file/bi-proto/kafka/v1/kafka.proto) - [[kafka_default.yaml]](file/bi-common/mq/kafkax/kafkax-default.yaml) - [[services.yaml]](file/bi-intra/apisix/services.yaml) - [[apisix-manager.sh]](file/bi-intra/apisix/apisix-manager.sh) - [[README.md]](file/bi-intra/apisix/readme.md)

Table of Contents ​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction ​

This document explains the service communication protocols across the system with a contract-first approach using Protocol Buffers defined in bi-proto. It covers:

  • Contract-first design with gRPC and HTTP bindings
  • Service definitions, message schemas, and versioning strategies
  • gRPC implementation patterns (unary calls, keepalive, retries)
  • REST API integration via APISIX gateway (HTTP method mapping, routing, response formatting)
  • Bidirectional communication patterns, event streaming via Kafka, and asynchronous messaging
  • Authentication and authorization patterns, rate limiting, and circuit breakers
  • Error propagation across network boundaries and performance optimization techniques

Project Structure ​

The repository organizes communication artifacts into three primary areas:

  • bi-proto: Protocol Buffer contracts and generated bindings (gRPC, HTTP, OpenAPI, validation)
  • bi-common: Shared transport utilities (gRPC client, HTTP response/error types, Kafka client)
  • bi-intra: APISIX gateway configuration for REST exposure of gRPC services

Diagram sources

Section sources

Core Components ​

  • Protocol generation pipeline: Protobuf contracts generate Go stubs for gRPC, HTTP, OpenAPI, and validation.
  • gRPC client: Centralized client builder with keepalive, metadata propagation, tracing, recovery, and retry policies.
  • HTTP response/error model: Standardized response envelope and business error mapping.
  • Kafka client: Producer/consumer abstractions with batching, compression, SASL/TLS, and graceful shutdown.
  • APISIX gateway: REST-to-gRPC mapping with routing and response formatting.

Section sources

Architecture Overview ​

The system follows a contract-first design:

  • Define .proto contracts in bi-proto
  • Generate gRPC and HTTP bindings plus OpenAPI schemas
  • Expose gRPC services internally and map them to REST via APISIX
  • Use Kafka for event streaming and asynchronous messaging
  • Apply shared middleware for logging, tracing, and error propagation

Diagram sources

Detailed Component Analysis ​

Protocol Buffers and Contract-First Design ​

  • Generation targets: Go (protocol), gRPC, HTTP, OpenAPI, validation.
  • Validation plugin ensures schema-level constraints.
  • OpenAPI artifacts enable REST documentation and SDK generation.

Diagram sources

Section sources

gRPC Implementation Patterns ​

  • Client configuration supports keepalive, metadata propagation, tracing, recovery, and custom gRPC options.
  • Default retry policy configured via ServiceConfig for transient failures.
  • Example service definitions include dictionary retrieval RPCs.

Diagram sources

Section sources

Unary gRPC Calls and Keepalive ​

  • Clients configure keepalive parameters to maintain long-lived connections.
  • Default middlewares include recovery and tracing; optional logging middleware can be appended.
  • ServiceConfig enables automatic retries for transient errors.

Diagram sources

Section sources

Streaming Operations and Event Streaming via Kafka ​

  • Kafka producer supports batching, compression, async writes, and JSON helpers.
  • Kafka consumer supports single-partition and consumer-group modes, graceful shutdown, and lag monitoring.
  • Default configurations embedded via YAML.

Diagram sources

Section sources

REST API Integration via APISIX Gateway ​

  • APISIX routes HTTP requests to gRPC backends using HTTP bindings generated from .proto files.
  • Response formatting uses a unified envelope for HTTP APIs.
  • Management scripts and service definitions support lifecycle operations.

Diagram sources

Section sources

Error Propagation Across Network Boundaries ​

  • Business errors carry a structured code in metadata while preserving semantic HTTP status mapping.
  • HTTP responses use a unified envelope with code/message/data fields.
  • Middleware ensures consistent error handling and propagation.

Diagram sources

Section sources

Authentication and Authorization Patterns ​

  • Metadata propagation allows forwarding identity and permissions across service boundaries.
  • gRPC clients propagate selected metadata prefixes automatically.
  • APISIX can enforce authentication and authorization policies at the edge.

Diagram sources

Section sources

Rate Limiting and Circuit Breakers ​

  • APISIX supports rate limiting and circuit breaking at the gateway level.
  • gRPC clients can leverage retry policies and keepalive to improve resilience.

Section sources

Dependency Analysis ​

The bi-proto module depends on Kratos, gRPC, and validation libraries. Generated artifacts are consumed by services and shared utilities.

Diagram sources

Section sources

Performance Considerations ​

  • Enable keepalive on gRPC clients to reduce connection churn.
  • Use batched writes and compression in Kafka producers for throughput.
  • Configure appropriate timeouts and backoff for retries.
  • Prefer consumer groups for horizontal scaling and partition balancing.
  • Use APISIX caching and load balancing for REST traffic.

[No sources needed since this section provides general guidance]

Troubleshooting Guide ​

  • gRPC connectivity: Verify endpoint, keepalive, and dial options; confirm service discovery configuration.
  • Kafka health: Use Ping methods on producer/consumer; check broker connectivity and SASL/TLS settings.
  • HTTP responses: Ensure unified envelope formatting and business error metadata extraction.
  • APISIX routing: Confirm HTTP-to-gRPC mapping and route definitions.

Section sources

Conclusion ​

The system employs a robust contract-first strategy with Protocol Buffers, standardized HTTP envelopes, and shared transport utilities. gRPC clients integrate keepalive and retries, APISIX exposes REST endpoints mapped from gRPC, and Kafka provides resilient event streaming. Together, these components enable scalable, observable, and maintainable service communication across the platform.

[No sources needed since this section summarizes without analyzing specific files]

Appendices ​

Service Definitions and Message Schemas ​

  • Dictionary services for shops and categories demonstrate unary RPC patterns with map-based responses.

Section sources

Kafka Protocol and Bindings ​

  • Kafka protocol includes service definitions and HTTP bindings for event streaming.

Section sources