CtrlK
BlogDocsLog inGet started
Tessl Logo
SkillPublic

subsystem-summary-of-util

read this skill for a token-efficient summary of the util subsystem

Quality

70%

Does it follow best practices?

Run evals on this skill

Adds up to 20 points to the overall score

View guide
SecuritybySnyk

Passed

No findings from the security scan

Fix and improve this skill with Tessl

tessl review fix ./.claude/skills/subsystem-summary-of-util/SKILL.md
SKILL.md
Quality
Evals
Security

Util Subsystem — Technical Summary

The src/util/ directory provides foundational infrastructure used across all of stellar-core. It covers timing, scheduling, logging, numeric safety, filesystem operations, serialization, data structures, threading, and various small helpers.


1. Virtual Clock & Timers (Timer.h, Timer.cpp)

VirtualClock

Central timing facility with two modes: REAL_TIME (wall clock) and VIRTUAL_TIME (simulated, for tests). Owns an asio::io_context for IO dispatch and a Scheduler for main-thread action scheduling.

  • Key types: time_point (steady clock), system_time_point (wall/calendar time), duration.
  • crank(bool block): The main event-loop step. Dispatches pending timers, polls IO (with exponential priority biasing under overload), runs scheduled actions, and transfers items from the thread-safe pending queue to the scheduler. In VIRTUAL_TIME mode, advances time to the next event if idle.
  • postAction(): Thread-safe submission point for deferred work callbacks. Uses a mutex-protected pending queue; wakes the main thread via asio::post when enqueueing into an empty queue.
  • setCurrentVirtualTime(): Advance virtual time forward (monotonic; asserts forward progress).
  • sleep_for(): Real sleep in REAL_TIME, virtual time advance in VIRTUAL_TIME.
  • shouldYield(): In REAL_TIME, returns true after 500ms has elapsed (time-slice limit). Always returns true in VIRTUAL_TIME.
  • Time conversion helpers: to_time_t, from_time_t, systemPointToTm, tmToSystemPoint, isoStringToTm, tmToISOString, systemPointToISOString.

VirtualTimer

Timer coupled to a VirtualClock that uses virtual time. Supports expires_at(), expires_from_now(), async_wait(), cancel(). Typical usage for all delayed / periodic operations in core.

VirtualClockEvent

Internal event representation stored in the clock's priority queue. Carries a callback, trigger time, sequence number, and cancelled state.


2. Scheduler (Scheduler.h, Scheduler.cpp)

A multi-queue fair scheduler implementing a variant of the LAS (Least Attained Service) / FB (Foreground-Background) algorithm.

Design

  • Multiple named ActionQueues each track cumulative runtime (mTotalService).
  • On each runOne(), the queue with the lowest accumulated service time runs its next action.
  • A latency window (default 5s) serves three roles: (1) credit cap to prevent bursty monopolization, (2) overload detection threshold, (3) idle queue expiration.
  • Droppable actions are shed under overload (queue time exceeds latency window).

ActionQueue (private inner class)

  • Stores a deque of Element{Action, enqueueTime}.
  • Tracks mTotalService, mLastService, idle list membership.
  • tryTrim(): drops droppable actions that exceed the latency window.
  • runNext(): runs the front action and updates total service (floored at minTotalService).

Scheduler public interface

  • enqueue(name, action, type): Creates or reactivates a queue and appends the action.
  • runOne(): Dequeues from the lowest-service queue, trims stale idle queues.
  • getOverloadedDuration(): Reports how long any queue has been overloaded.
  • currentActionType(): Returns the ActionType of the currently executing action.
  • shutdown(): Clears all queues.

3. Logging (Logging.h, Logging.cpp, SpdlogTweaks.h, LogPartitions.def)

Logging class

Static utility managing spdlog-based logging with partitioned loggers.

  • Partitions: Defined in LogPartitions.def via X-macro (LOG_PARTITION(name)), yielding ~15 named partitions (e.g., Bucket, Herder, Ledger, Overlay, Tx, etc.). Each partition has its own spdlog logger with independent log level.
  • init(): Creates sinks (stderr + optional file), registers all partition loggers, configures flush behavior (every 1s, immediate on error). Also initializes Rust-side logging.
  • setLogLevel(level, partition): Per-partition or global level setting.
  • setLoggingToFile(filename): Supports {datetime}-formatted filenames. Opens in append mode (O_APPEND) for durability with external log rotation.
  • setFmt(peerID): Sets the log pattern including a peer identifier.
  • rotate(): Log rotation support.

Log macros

  • CLOG_TRACE/DEBUG/INFO/WARNING/ERROR/FATAL(partition, fmt, ...): Partition-aware logging with compile-time format string checking via FMT_STRING.
  • LOG_TRACE/DEBUG/INFO/WARNING/ERROR/FATAL(logger, fmt, ...): Logger-handle-based logging.
  • LOG_CHECK: Guards log calls with should_log() check to avoid formatting overhead.

CoutLogger

Fallback logger when spdlog is not available. Writes to std::cout.

LogLevel enum

LVL_FATAL(0) through LVL_TRACE(5).

format_as overloads

Provides fmt-compatible formatting for XDR enum types and std::filesystem::path.


4. Numeric Utilities (numeric.h, numeric.cpp, numeric128.h)

Safe arithmetic for financial calculations where overflow matters.

64-bit operations (numeric.h)

  • bigDivide(result, A, B, C, rounding): Computes A*B/C safely when A*B would overflow int64_t. Returns success/failure bool.
  • bigDivideOrThrow(A, B, C, rounding): Throwing variant.
  • bigDivideUnsigned(): Unsigned variant.
  • bigSquareRoot(a, b): Integer square root of a*b, ROUND_DOWN only.
  • saturatingMultiply(a, b): Caps at INT64_MAX on overflow (non-negative inputs).
  • saturatingAdd(a, b): Unsigned, caps at type max.
  • isRepresentableAsInt64(double): Checks safe double→int64 conversion.
  • doubleToClampedUint32(double): Clamps double to uint32 range.
  • Rounding enum: ROUND_DOWN, ROUND_UP.

128-bit operations (numeric128.h)

  • bigMultiply(a, b) / bigMultiplyUnsigned(a, b): Returns uint128_t.
  • bigDivide128() / bigDivideUnsigned128(): Divide 128-bit by 64-bit.
  • hugeDivide(result, a, B, C, rounding): Computes a*B/C when C < INT32_MAX * INT64_MAX.

5. Filesystem Utilities (Fs.h, Fs.cpp, FileSystemException.h, TmpDir.h)

fs namespace

  • File locking: lockFile() / unlockFile() using flock (POSIX) or CreateFile (Win32).
  • Durable I/O: flushFileChanges() (fsync), durableRename() (rename + dir fsync), openFileToWrite().
  • Directory ops: exists(), deltree(), mkdir(), mkpath() (recursive), findfiles().
  • Path construction: hexStr(), hexDir(), baseName(), remoteName(), remoteDir() — used for history archive paths.
  • Handle counting: getMaxHandles(), getOpenHandleCount() — Linux reads /proc/self/fd.
  • bufsz(): Returns 256KB (AWS EBS IOP alignment).

FileSystemException

Extends std::runtime_error with failWith() and failWithErrno() static helpers that log before throwing.

TmpDir / TmpDirManager

RAII temporary directory management. TmpDirManager creates and cleans a root directory; TmpDir creates prefixed subdirectories that are deleted on destruction.


6. XDR Stream I/O (XDRStream.h)

XDRInputFileStream

Reads XDR-framed objects from a file one at a time.

  • readOne(T& out, SHA256* hasher): Reads 4-byte big-endian size header, then that many bytes, unmarshals into out. Optional hash accumulation.
  • readPage(T& out, key, pageSize): Reads a page-sized chunk and scans for a specific LedgerKey. Used for indexed bucket lookups.
  • Supports seek(), pos(), size limits.

OutputFileStream

Low-level durable write stream using ASIO buffered writes (POSIX) or FILE* (Win32).

  • writeBytes(), flush(), close() with optional fsync on close.
  • Cross-platform via fs::stream_t / fs::native_handle_t.

XDROutputFileStream

Extends OutputFileStream with XDR framing.

  • writeOne(T, hasher, bytesPut): Encodes 4-byte size header + XDR payload. Optional hash and byte counting.
  • durableWriteOne(): writeOne + flush + fsync for crash safety.

7. Data Structures

RandomEvictionCache<K, V> (RandomEvictionCache.h)

Fixed-size cache with random-2-choices eviction (not LRU). More robust under pathological access patterns.

  • Stores entries in unordered_map with stable pointers in a vector for random access.
  • Tracks generation counter for last-access ordering.
  • put(), get(), maybeGet(), exists(), erase_if(), clear().
  • Maintains hit/miss/insert/update/evict counters.

BitSet (BitSet.h)

Value-semantic C++ wrapper around a C bitset (cbitset.h). Uses inline storage for small sets (≤64 bits) to avoid heap allocation.

  • Full set operations: union (|), intersection (&), difference (-), symmetric difference.
  • Counting variants: unionCount(), intersectionCount(), etc.
  • Iteration: nextSet(i) for scanning set bits.
  • Hash function support via BitSet::HashFunction.

UnorderedMap<K, V> / UnorderedSet<K> (UnorderedMap.h, UnorderedSet.h)

Type aliases for std::unordered_map / std::unordered_set using RandHasher to prevent hash-flooding attacks.

BinaryFuseFilter<T> (BinaryFuseFilter.h, BinaryFuseFilter.cpp)

Probabilistic membership-test filter (like Bloom filter but more space-efficient). Wraps binary_fuse_t library. Supports 8/16/32-bit widths with corresponding false-positive rates (1/256, 1/65536, 1/4B). Serializable via cereal to SerializedBinaryFuseFilter XDR type. Used for efficient key-membership checks in bucket indexes.

TarjanSCCCalculator (TarjanSCCCalculator.h)

Tarjan's algorithm for computing strongly connected components of a directed graph. Uses BitSet for SCC representation. Used in quorum analysis (SCP).


8. Threading Utilities

ThreadAnnotations.h

Comprehensive Clang thread-safety annotation macros (GUARDED_BY, REQUIRES, ACQUIRE, RELEASE, TRY_ACQUIRE, etc.). Provides annotated mutex wrappers:

  • Mutex: Wraps std::mutex with annotations.
  • SharedMutex: Wraps std::shared_mutex with Lock()/LockShared().
  • RecursiveMutex: Wraps std::recursive_mutex.
  • MutexLocker / RecursiveMutexLocker / SharedLockExclusive / SharedLockShared: RAII lock guards with annotations.

GlobalChecks.h / GlobalChecks.cpp

  • threadIsMain(): Checks if current thread is the main thread (captured at static init).
  • releaseAssert(e): Assert that is NOT compiled out by NDEBUG. Prints backtrace and aborts.
  • releaseAssertOrThrow(e): Like releaseAssert but throws runtime_error instead of aborting.
  • dbgAssert(e): Debug-only assert.
  • LockGuard / RecursiveLockGuard: Aliases that adapt between Tracy-instrumented and plain lock guards.

Thread.h / Thread.cpp

  • runCurrentThreadWithLowPriority() / runCurrentThreadWithMediumPriority(): OS-level thread priority adjustment.
  • futureIsReady(std::future|std::shared_future): Non-blocking readiness check.

JitterInjection.h / JitterInjection.cpp

Testing framework for probabilistic thread-delay injection (enabled via BUILD_THREAD_JITTER).

  • JitterInjector::injectDelay(probability, minUsec, maxUsec): Probabilistically sleeps to expose race conditions.
  • JITTER_INJECT_DELAY(), JITTER_YIELD() macros: Compile to no-ops in production.
  • Reproducible via seeded RNG tied to test seed.

NonCopyable.h

Base structs: NonCopyable, NonMovable, NonMovableOrCopyable — disable copy/move semantics via deleted special members.


9. Metrics & Performance Monitoring

MetricsRegistry (MetricsRegistry.h, MetricsRegistry.cpp)

Extends medida::MetricsRegistry with SimpleTimer support. NewSimpleTimer() registers lightweight timers. syncSimpleTimerStats() syncs max values (called on /metrics endpoint).

SimpleTimer / SimpleTimerContext (SimpleTimer.h, SimpleTimer.cpp)

Lightweight replacement for medida::Timer using counters (sum, count, max). Avoids histogram overhead. Uses medida::Counter internally for metric export. TimeScope() returns an RAII context for automatic timing.

MetricResetter (MetricResetter.h, MetricResetter.cpp)

Implements medida::MetricProcessor to reset all metric types (Counter, Meter, Histogram, Timer, Buckets).

LogSlowExecution (LogSlowExecution.h, LogSlowExecution.cpp)

RAII guard that logs a warning if a scope takes longer than a threshold (default 1 second). Two modes:

  • AUTOMATIC_RAII: Logs on destructor.
  • MANUAL: Caller checks checkElapsedTime().
  • RateLimitedLog: Subclass with zero threshold for rate-limited logging.

10. Type Utilities (types.h, types.cpp)

Core type definitions and helpers for XDR/Stellar types.

  • Blob: std::vector<unsigned char>.
  • LedgerKeySet: std::set<LedgerKey, LedgerEntryIdCmp>.
  • LedgerEntryKey(LedgerEntry): Extracts the key from an entry.
  • Asset helpers: assetToString(), getIssuer(), isIssuer(), assetCodeToStr(), strToAssetCode(), isAssetValid(), compareAsset().
  • getBucketLedgerKey(): Extracts LedgerKey from BucketEntry or HotArchiveBucketEntry.
  • addBalance(balance, delta, max): Safe balance addition with overflow check.
  • isZero(uint256), lessThanXored(): Hash/XOR comparison helpers.
  • unsignedToSigned(): Safe conversion with overflow check.
  • formatSize(): Human-readable size formatting.
  • iequals(): Case-insensitive string comparison.
  • Price comparison operators: >=, >, ==.
  • ASCII helpers: isAsciiAlphaNumeric(), isAsciiNonControl(), toAsciiLower() — locale-independent.
  • roundDown(v, m): Round down to largest multiple of power-of-2.

11. Resource Tracking (TxResource.h, TxResource.cpp)

Resource

Multi-dimensional resource vector for transaction resource accounting (operations, instructions, byte sizes, ledger entries). Supports:

  • 1-element (classic), 2-element (classic with bytes), or 7-element (Soroban) resource tuples.
  • Arithmetic: +=, -=, +, -, with overflow-safe bigDivideOrThrow.
  • Comparisons: <=, ==, >, anyLessThan(), anyGreater().
  • subtractNonNegative(), limitTo(), saturatedMultiplyByDouble().
  • Resource::Type enum: OPERATIONS, INSTRUCTIONS, TX_BYTE_SIZE, DISK_READ_BYTES, WRITE_BYTES, READ_LEDGER_ENTRIES, WRITE_LEDGER_ENTRIES.

12. XDR Query Engine (xdrquery/)

A mini query language for filtering XDR objects by field values.

Components

  • XDRFieldResolver.h: Template-based XDR field traversal using xdr_traits. Resolves dotted field paths (e.g., "data.account.balance") to values. Handles unions, enums, public keys (string representation), opaque arrays (hex).
  • XDRQueryEval.h: Expression evaluation engine. EvalNode hierarchy: LiteralNode, ColumnNode, BoolOpNode (AND/OR/NOT), ComparisonOpNode (==, !=, <, >, <=, >=). DynamicXDRGetter interface for runtime field access.
  • XDRQuery.h: TypedDynamicXDRGetterResolver<T> connects XDR types to the query engine. matchXDRQuery() function parses and evaluates queries against XDR objects.
  • XDRQueryError.h: Exception type for query parsing/evaluation errors.

13. Serialization Helpers

XDRCereal.h / XDRCereal.cpp

Custom cereal_override functions for human-readable JSON serialization of XDR types. Overrides for: public keys (StrKey), assets (code+issuer), enums (string names), opaque data (hex), optional pointers (null handling), SCAddress, MuxedAccount, ConfigUpgradeSetKey.

  • xdrToCerealString(t, name, compact): Serialize any XDR type to JSON string.

BufferedAsioCerealOutputArchive.h

Custom cereal output archive that writes through OutputFileStream (supporting fsync) instead of std::ofstream.

XDROperators.h

Imports xdr::operator== and xdr::operator< into the stellar namespace.


14. Hashing Utilities

RandHasher<T> (RandHasher.h)

Hash functor that XORs the output of std::hash<T> with a random mixer value (initialized once at startup). Prevents hash-flooding DoS attacks on hash tables.

HashOfHash (HashOfHash.h, HashOfHash.cpp)

Specialization of std::hash<stellar::uint256> for use in hash maps keyed by transaction/ledger hashes.


15. Protocol Version Utilities (ProtocolVersion.h, ProtocolVersion.cpp)

ProtocolVersion enum

Enumerates all protocol versions V_0 through V_26.

Comparison functions

  • protocolVersionIsBefore(version, beforeVersion): Strictly less than.
  • protocolVersionStartsFrom(version, fromVersion): Greater than or equal.
  • protocolVersionEquals(version, equalsVersion): Exact match.

Notable constants

  • SOROBAN_PROTOCOL_VERSION = V_20
  • PARALLEL_SOROBAN_PHASE_PROTOCOL_VERSION = V_23
  • AUTO_RESTORE_PROTOCOL_VERSION = V_23

16. Other Utilities

StatusManager (StatusManager.h)

Manages status messages by category (HISTORY_CATCHUP, HISTORY_PUBLISH, NTP, REQUIRES_UPGRADES). Used for the /info JSON endpoint.

SecretValue (SecretValue.h)

Wrapper around std::string to prevent accidental logging of sensitive values (DB passwords, secret keys).

Decoder.h

Base32/Base64 encoding and decoding helpers (encode_b32, encode_b64, decode_b32, decode_b64).

Math.h / Math.cpp

Random number utilities: rand_fraction(), rand_flip(), rand_uniform(), rand_element(), k_means(), closest_cluster(), exponentialBackoff(). initializeAllGlobalState() initializes all PRNG and hash seeds. reinitializeAllGlobalStateWithSeed() (test-only) resets for reproducibility.

MetaUtils.h / MetaUtils.cpp

normalizeMeta(): Sorts order-agnostic fields in TransactionMeta / LedgerCloseMeta for deterministic comparison/hashing.

DebugMetaUtils.h / DebugMetaUtils.cpp

Manages debug meta-data files (XDR segments of ledger metadata). Handles file naming, path construction, listing, and cleanup based on configurable segment sizes (256 ledgers).

Backtrace.h / Backtrace.cpp

printCurrentBacktrace(): Prints stack trace via the Rust bridge.

must_use.h

MUST_USE macro: Maps to __attribute__((warn_unused_result)) where supported.

Algorithm.h

split() function template: Splits a vector into a map of vectors keyed by an extractor function.

asio.h

Correctly configures and includes ASIO headers with proper preprocessor definitions (ASIO_SEPARATE_COMPILATION, ASIO_STANDALONE). Must be included before other system headers.

TcmallocConfig.cpp

Tcmalloc configuration (if tcmalloc is the allocator).


Key Data Flows

  1. Main event loop: VirtualClock::crank() → dispatches timers → polls ASIO IO → runs Scheduler::runOne() → transfers pending actions from thread-safe queue to scheduler.

  2. Cross-thread work submission: Any thread → VirtualClock::postAction() (mutex-protected queue) → main thread dequeues into Scheduler → fair-scheduled execution.

  3. XDR file I/O: Write path: XDROutputFileStream::writeOne() → 4-byte size header + XDR marshal → buffered write → optional fsync. Read path: XDRInputFileStream::readOne() → read size header → read payload → XDR unmarshal.

  4. Metrics flow: Code uses medida::Timer/SimpleTimer → MetricsRegistry collects → syncSimpleTimerStats() synchronizes max tracking → exported via /metrics endpoint.

  5. Resource accounting: Resource vectors created per-transaction → arithmetic operations check against ledger limits → anyGreater()/<= comparisons for admission control.

Repository

stellar/stellar-core

Path

.claude/skills/subsystem-summary-of-util/SKILL.md

Last updated
First committed

Is this your skill?

If you maintain this skill, you can claim it as your own. Once claimed, you can manage eval scenarios, bundle related skills, attach documentation or rules, and ensure cross-agent compatibility.