Deployment and Operations
Table of Contents
- Introduction
- Project Structure
- Core Components
- Architecture Overview
- Detailed Component Analysis
- Dependency Analysis
- Performance Considerations
- Troubleshooting Guide
- Conclusion
- Appendices
Introduction
This document provides comprehensive deployment and operations guidance for the BI Analysis Platform. It covers containerization with Docker, Kubernetes deployment manifests, and operational procedures centered around service discovery via Nacos, API gateway configuration with APISIX, and load balancing strategies. It also documents production deployment procedures, environment management, configuration drift prevention, monitoring and observability, logging, alerting, scaling and capacity planning, performance tuning, disaster recovery, backup strategies, and maintenance workflows.
Project Structure
The platform consists of multiple microservices packaged as Docker images and deployed via Kubernetes manifests. Each service exposes HTTP and/or gRPC endpoints and integrates with Nacos for configuration and service discovery. APISIX serves as the API gateway to route traffic to services and enforce security policies.
Diagram sources
Section sources
Core Components
- Containerization with Docker: Each service includes a Dockerfile that builds a minimal runtime image, copies configuration files, sets timezone, and exposes appropriate ports.
- Kubernetes Deployment: Deployments define imagePullSecrets, environment variables, resource requests/limits, and container ports aligned with service configuration.
- Service Exposure: Services expose HTTP and gRPC ports for each service, with NodePort in development/test and APISIX as ingress in production.
- Configuration Management: Nacos is used for centralized configuration and service discovery, with typed configuration structures and client helpers.
- API Gateway: APISIX manages routing, authentication, rate limiting, and WAF protection across services.
Section sources
Architecture Overview
The platform follows a service mesh-like pattern behind APISIX:
- Clients reach the gateway via a cloud load balancer.
- APISIX routes requests to service endpoints discovered via Nacos or configured upstreams.
- Services communicate internally using HTTP/gRPC and share configuration via Nacos.
Diagram sources
Detailed Component Analysis
Containerization Strategies with Docker
- Multi-stage builds: Builder stage installs Go dependencies and compiles binaries; final stage packages runtime-only artifacts.
- Private module support: Uses .netrc and GOPROXY configuration during build; credentials are not carried into the final image.
- Runtime configuration: Copies configs into the image and sets timezone; exposes ports per service.
- Example services: bi-analysis, bi-api-jushuitan, bi-api-leke, bi-basic, bi-chat.
Diagram sources
Section sources
Kubernetes Deployment Manifests
- Image pull secrets: Ensures private registry access.
- Environment injection: DEPLOY_ENV and NACOS_* credentials injected from ConfigMap and Secret.
- Ports: Align with service configuration (HTTP and gRPC).
- Resources: Requests/limits defined per service.
Diagram sources
Section sources
Service Discovery and Configuration Management with Nacos
- Typed configuration model: Structured server/client/config/auth/advanced fields with defaults and validation.
- Client helpers: Separate config and naming clients; supports publish/listen/get/delete operations.
- Application integration: Services read DEPLOY_ENV and NACOS_* credentials to initialize Nacos client and fetch configuration.
Diagram sources
Section sources
API Gateway Setup with APISIX
- Centralized routing: Services are mapped by route_prefix and host; some services use Nacos service names while others use explicit upstream URLs.
- Security: JWT authentication, API key, rate limiting, and WAF rules configured centrally.
- External exposure: Cloud Load Balancer annotation example for HTTPS-only with bandwidth and certificate configuration.
Diagram sources
Section sources
Load Balancing Strategies
- Internal: Kubernetes Services distribute traffic to pods.
- External: Cloud Load Balancer forwards to APISIX Gateway; annotations configure HTTPS-only mode, bandwidth, and certificate.
- Recommendations: Use health checks and pod anti-affinity to spread traffic; enable connection timeouts and retries at APISIX.
Section sources
Production Deployment Procedures
- Pre-deploy checklist:
- Confirm image tags and imagePullSecrets.
- Verify ConfigMaps and Secrets (DEPLOY_ENV, NACOS_*).
- Validate APISIX route definitions and security settings.
- Deployment steps:
- Apply Nacos configuration files and secrets.
- Deploy services via kubectl apply.
- Verify readiness and logs.
- Post-deploy verification:
- Test routing via APISIX endpoints.
- Confirm service discovery and configuration updates.
Section sources
Environment Management and Configuration Drift Prevention
- Environments: DEPLOY_ENV drives configuration selection (dev/test/prod).
- Secrets: NACOS_* credentials injected via Kubernetes Secrets.
- Drift prevention:
- Store all configuration in Nacos with versioning.
- Use APISIX admin API keys and immutable route definitions.
- Pin image digests and avoid “latest” tags.
- Track changes via GitOps or declarative manifests.
Section sources
Monitoring and Observability
- Metrics: Enable Prometheus scraping on services and APISIX; export metrics from APISIX global rules.
- Logs: Standard out/in container logs; centralize with Fluent Bit/Fluentd; include correlation IDs.
- Alerts: Define SLOs for latency and error rates; alert on rate limit hits and WAF rejections.
[No sources needed since this section provides general guidance]
Logging Strategies and Alerting Mechanisms
- Logging: Capture request traces, errors, and audit events; tag with tenant/service.
- Alerting: Threshold-based alerts for error rate, latency P95/P99, and gateway anomalies.
[No sources needed since this section provides general guidance]
Scaling Approaches and Capacity Planning
- Horizontal scaling: Increase replicas in deployments; ensure adequate CPU/memory requests/limits.
- Auto-scaling: Consider HPA based on CPU/utilization or custom metrics.
- Capacity planning: Estimate peak RPS per service; size APISIX workers and upstream pools accordingly.
Section sources
Performance Tuning for Production
- APISIX: Tune upstream timeouts, retries, and thread pool sizes; enable gzip/compression.
- Services: Optimize GC and concurrency; reduce allocations; monitor hotspots.
- Network: Minimize round trips; enable keep-alive; colocate services near databases.
[No sources needed since this section provides general guidance]
Disaster Recovery and Backup Strategies
- Backups: Regular snapshots of persistent volumes and Nacos configuration backups.
- DR: Multi-zone clusters with geo-redundant storage; failover DNS or LB policies.
- Maintenance windows: Schedule rolling updates and drain traffic before upgrades.
[No sources needed since this section provides general guidance]
Maintenance Workflows
- Patching: Rolling updates with zero downtime; validate with canary traffic.
- Rotation: Rotate secrets and API keys; update Nacos credentials and APISIX admin keys.
- Cleanup: Archive old images, prune unused ConfigMaps/Secrets.
[No sources needed since this section provides general guidance]
Dependency Analysis
The following diagram shows key dependencies among components:
Diagram sources
Section sources
Performance Considerations
- Resource sizing: Start with conservative requests/limits; adjust based on CPU and memory utilization.
- APISIX tuning: Configure upstream timeouts, retries, and connection pooling; enable compression.
- Service-level: Optimize hot paths; reduce GC pressure; leverage connection reuse.
[No sources needed since this section provides general guidance]
Troubleshooting Guide
- Service not reachable:
- Verify Service selectors and ports match Deployment labels and container ports.
- Check APISIX route_prefix and host configuration.
- Authentication failures:
- Confirm JWT secret and API key settings in APISIX.
- Validate whitelist entries for public endpoints.
- Configuration not applied:
- Ensure Nacos credentials and namespace/group are correct.
- Confirm data IDs and content are published and watched.
Section sources
Conclusion
This guide outlines a robust, production-ready deployment and operations model for the BI Analysis Platform. By leveraging Docker for containerization, Kubernetes for orchestration, Nacos for configuration and service discovery, and APISIX for secure routing, the platform achieves scalability, reliability, and maintainability. Adhering to environment management practices, monitoring, and disaster recovery procedures ensures smooth operations across environments.
Appendices
- Cron Jobs: Background tasks orchestrated via Kubernetes Jobs; configure schedule and resource limits appropriately.
Section sources