vitepress-template

Appearance

Utilities & Helpers ​

**Referenced Files in This Document** - [[parser.go]](file/bi-common/utils/envutil/parser.go) - [[README.md]](file/bi-common/utils/envutil/readme.md) - [[converter.go]](file/bi-common/utils/converter/converter.go) - [[builder.go]](file/bi-common/utils/querybuilder/builder.go) - [[validate.go]](file/bi-common/utils/validation/validate.go) - [[README.md]](file/bi-common/utils/validation/readme.md) - [[endpoint.go]](file/bi-common/utils/netutil/endpoint.go) - [[snowflake.go]](file/bi-common/utils/snowflake/snowflake.go) - [[convert.go]](file/bi-common/utils/strutil/convert.go) - [[README.md]](file/bi-common/utils/strutil/readme.md) - [[jwt.go]](file/bi-common/utils/jwt/jwt.go) - [[config.go]](file/bi-common/utils/fileutil/config.go)

Table of Contents ​

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion

Introduction ​

This document describes the utility libraries and helper functions across the platform, focusing on:

  • Environment variable parsing and configuration loading
  • Data conversion utilities and collection transformations
  • SQL query building helpers for GORM
  • Validation helpers for robust configuration and input checks
  • Network utilities and ID generation strategies
  • String manipulation utilities It also covers usage patterns, integration points with core services, performance characteristics, thread safety, and best practices.

Project Structure ​

The utilities live under the bi-common module’s utils directory, organized by domain:

  • envutil: environment variable parsing and defaults
  • fileutil: configuration file load/save and nested config extraction
  • converter: generic slice/map conversions and helpers
  • querybuilder: fluent SQL/GORM query builder
  • validation: common validation functions
  • netutil: network endpoint helpers
  • snowflake: distributed ID generator
  • strutil: string and URL encoding helpers
  • jwt: token generation and parsing

Diagram sources

Section sources

Core Components ​

  • Environment variable parsing: robust typed getters, required-value getters, existence checks, and fallbacks across multiple keys.
  • Data conversion utilities: generic map/slice transforms, grouping, filtering, deduplication, and pointer helpers.
  • SQL query builder: fluent builder for GORM with conditions, ordering, pagination, and safe LIKE/IN/BETWEEN clauses.
  • Validation helpers: network, time, and general-purpose validators with clear error messages.
  • Network utilities: endpoint host replacement for local development scenarios.
  • ID generation: Snowflake-compatible ID generator with clock-backward tolerance and initialization from config/env/IP.
  • String utilities: zero-allocation integer-to-string and URL encoding helpers.
  • JWT utilities: token generation and parsing with configurable expiration and refresh threshold.

Section sources

Architecture Overview ​

The utilities are designed as independent, composable packages with minimal external dependencies. They integrate with core services via:

  • Environment-driven configuration via envutil
  • Robust validation via validation
  • Safe SQL construction via querybuilder
  • Data transformation via converter
  • Endpoint resolution via netutil
  • Stable IDs via snowflake
  • String formatting via strutil
  • Token handling via jwt

Diagram sources

Detailed Component Analysis ​

Environment Variable Parsing (envutil) ​

  • Provides typed getters for string, int, int64, uint, uint64, bool, float64, and time.Duration with defaults.
  • Required-value getters panic on missing values.
  • Existence checking and multi-key fallback support for backward compatibility.
  • Zero-dependency, high-performance, and suitable for hot paths.

Usage patterns:

  • Load configuration from environment with sensible defaults.
  • Enforce required values early during startup.
  • Support legacy environment variable names via fallback.

Integration examples:

  • Redis configuration loading uses envutil extensively for all connection parameters.
  • Configuration loaders can delegate environment parsing to envutil.

Best practices:

  • Prefer MustGet for truly required values.
  • Provide explicit defaults for optional values.
  • Use GetEnvOr for gradual migration of environment variable names.

Section sources

Data Conversion Utilities (converter) ​

  • Generic helpers for transforming slices and maps, including ToSlice, ToMap, ToMapByKey, GroupBy, Filter, Find, Contains, Unique, and Pluck.
  • Time formatting helpers and pointer/value helpers (Ptr, Val, ValOr).
  • Zero-allocation time formatting and safe pointer dereferencing.

Usage patterns:

  • Convert domain models to DTOs using ToSlice.
  • Build lookup maps via ToMapByKey.
  • Deduplicate collections with Unique.
  • Safely handle pointer fields with Val/ValOr.

Integration examples:

  • Business logic converts persisted models to business structs using ToSlice.
  • Query conditions use converter.Val to safely extract optional filter values.

Thread safety:

  • Stateless functions; safe for concurrent use.

Section sources

SQL Query Builder (querybuilder) ​

  • Fluent builder for GORM queries with Where, Eq, Ne, Gt, Gte, Lt, Lte, Like, LikeLeft, LikeRight, In, NotIn, IsNull, IsNotNull, Between, OrderBy, Page.
  • Supports conditional clauses via WhereIf/EqIf/GtIf/etc.
  • Pagination helpers and offset/limit extraction.

Usage patterns:

  • Build dynamic filters from request parameters.
  • Apply optional conditions only when values are present.
  • Paginate results consistently across services.

Integration examples:

  • System services construct queries using New() and chain WhereIf/OrderBy/Page.

Thread safety:

  • Builder holds state; avoid sharing across goroutines. Use per-request instances.

Section sources

Validation Helpers (validation) ​

  • Validates ports, hosts, URLs, timeouts, IP addresses, non-empty strings, positive integers, and numeric ranges.
  • Returns descriptive errors for easy diagnostics.

Usage patterns:

  • Validate configuration objects before use.
  • Validate API inputs and request parameters.
  • Replace ad-hoc validation with centralized helpers.

Integration examples:

  • Services call ValidateHostPort/ValidateURL/ValidateTimeout in preconditions.

Thread safety:

  • Stateless functions; safe for concurrent use.

Section sources

Network Utilities (netutil) ​

  • Replace endpoint host with EXTERNAL_IP for local development and VPN scenarios.
  • Parses scheme/host:port and reconstructs endpoint with new host.

Usage patterns:

  • Override service endpoints during local testing.
  • Ensure consistent endpoint formatting across services.

Thread safety:

  • Stateless function; safe for concurrent use.

Section sources

ID Generation (snowflake) ​

  • Distributed ID generator compatible with Twitter Snowflake.
  • Clock-backward tolerance with graceful waits or errors.
  • Initialization from configuration, environment variables, or IP-derived worker ID.
  • Optional integration with Nacos for dynamic worker assignments.

Usage patterns:

  • Generate unique IDs for entities across services.
  • Initialize once at startup with Init/InitFromConf/InitFromConfWithNacos.
  • Parse IDs to extract timestamps and metadata.

Thread safety:

  • Node uses a mutex internally; safe for concurrent use after initialization.

Section sources

String Utilities (strutil) ​

  • Zero-allocation integer-to-string conversion (Itoa).
  • URL encoding for special characters.
  • Optimized for high-frequency use cases.

Usage patterns:

  • Construct DSNs and HTTP parameters efficiently.
  • Encode credentials and identifiers for URLs.

Thread safety:

  • Stateless functions; safe for concurrent use.

Section sources

JWT Utilities (jwt) ​

  • Generate access and refresh tokens with configurable expiration and issuer.
  • Parse tokens and validate claims.
  • Determine whether a token needs refreshing based on threshold.

Usage patterns:

  • Issue tokens after successful authentication.
  • Validate incoming requests and refresh tokens proactively.

Thread safety:

  • Stateless; safe for concurrent use.

Section sources

File Utilities (fileutil) ​

  • Load configuration from YAML/JSON files with automatic format detection.
  • Nested configuration extraction by root key.
  • Serialize configurations back to files.
  • File existence checks and parsing helpers.

Usage patterns:

  • Load service configuration from mounted volumes or config maps.
  • Support both flat and nested configuration structures.

Thread safety:

  • Stateless functions; safe for concurrent use.

Section sources

Dependency Analysis ​

  • Minimal external dependencies: only Go standard library and GORM for querybuilder.
  • Internal coupling is low; utilities are cohesive by domain.
  • No circular dependencies observed among the utility packages.

Diagram sources

Section sources

Performance Considerations ​

  • envutil: extremely fast environment reads with zero allocations for most paths.
  • strutil.Itoa: significantly faster than standard library with zero allocations.
  • querybuilder: builds GORM queries incrementally; keep builder instances per request to avoid contention.
  • snowflake: thread-safe with internal locking; initialize once and reuse.
  • validation: constant-time checks with minimal allocations.
  • fileutil: YAML/JSON parsing overhead depends on file size; consider caching parsed configs.

[No sources needed since this section provides general guidance]

Troubleshooting Guide ​

Common issues and resolutions:

  • Missing required environment variables: Use MustGet variants to fail fast with clear panic messages.
  • Invalid configuration values: Apply validation helpers before use to catch errors early.
  • Query builder misuse: Avoid sharing builder instances across goroutines; create per-request builders.
  • Snowflake initialization: Ensure SNOWFLAKE_WORKER_ID/DATACENTER_ID are set appropriately or rely on IP-based derivation.
  • Endpoint host mismatch: Set EXTERNAL_IP to override host in local development.
  • JWT token errors: Check secret, issuer, and expiration; use NeedRefresh to manage token lifecycle.

Section sources

Conclusion ​

These utilities provide a consistent, efficient, and safe foundation for configuration, data transformation, query building, validation, networking, ID generation, string manipulation, and token handling. By adopting these helpers, services remain maintainable, performant, and resilient across environments and workloads.