Caching in ypricemagic

Overview

ypricemagic uses multiple caching strategies to maximize performance, minimize redundant RPC/database calls, and provide fast access to frequently used data. This document describes the two main types of disk caches and the various in-memory LRU cache mechanisms used throughout the codebase.

Disk Caches

Local SQL Cache

  • Backend: Pony ORM with SQLite (default, configurable via environment variables)

  • Location: ~/.ypricemagic/ypricemagic.sqlite (can be overridden with YPRICEMAGIC_SQLITE_PATH)

  • Purpose: Persistent storage for blockchain data, price data, and cache metadata.

  • What is cached: - Blockchain data: chains, blocks, addresses, contracts, tokens, prices - Cache metadata: TraceCacheInfo, LogCacheInfo (for logs/traces)

  • Implementation: Entities defined in y/_db/entities.py, setup in y/_db/config.py

  • Usage: Used for both persistent data and as a cache for expensive lookups.

Integration with eth-portfolio

If [eth-portfolio](https://github.com/BobTheBuidler/eth-portfolio) is installed in your environment, ypricemagic will automatically use eth_portfolio’s extended database schema and utility functions for persistent storage and lookups. This means:

  • Both ypricemagic and eth_portfolio will share the same persistent SQL database (typically SQLite).

  • Cached blockchain, price, and portfolio data will be unified and accessible to both packages.

  • Utility functions such as get_block and get_token will use the eth_portfolio versions if available, ensuring consistent and extended data access.

  • No configuration is required: the integration is seamless and automatic if eth_portfolio is present.

Joblib Disk Cache

  • Backend: joblib.Memory

  • Location: cache/{chain.id}

  • Purpose: Disk-based function result caching for expensive or slow-to-compute results.

  • Usage: Decorate functions with @memory.cache to persist results to disk.

  • Example: Used in y/contracts.py and other modules.

Database Environment Variables

The following environment variables control how ypricemagic connects to and configures its database:

  • YPRICEMAGIC_DB_PROVIDER (str, default: “sqlite”): Database backend to use (e.g., “sqlite”, “postgresql”).

  • YPRICEMAGIC_SQLITE_PATH (str, default: ~/.ypricemagic/ypricemagic.sqlite): Path to the SQLite database file. Set this to use a custom or temporary database file (e.g., for testing, CI, or multiple environments).

  • YPRICEMAGIC_DB_HOST (str, default: “”): Host address for the database if not using sqlite.

  • YPRICEMAGIC_DB_PORT (str, default: “”): Port for the database.

  • YPRICEMAGIC_DB_USER (str, default: “”): Username for the database.

  • YPRICEMAGIC_DB_PASSWORD (str, default: “”): Password for the database.

  • YPRICEMAGIC_DB_DATABASE (str, default: “ypricemagic”): Database name.

Set these variables in your environment to control which database backend is used and how ypricemagic connects to it.

In-Memory LRU Caches

Synchronous LRU Caches

  • functools.lru_cache: Used for pure in-memory caching of function results (e.g., token metadata, block lookups).

  • cachetools.cached/TTLCache: Used for time-limited in-memory caching, especially for DB query results.

  • db_session_cached: Combines lru_cache with DB session/retry logic for DB reads.

Asynchronous LRU Caches

  • async_lru.alru_cache: Used for async LRU caching of coroutine results (e.g., async DB/API calls).

  • a_sync.a_sync: Used for both sync and async function caching, with options for TTL and maxsize.

  • a_sync_ttl_cache: Preconfigured async decorator with TTL, used for async memory caching.

What is LRU-cached

  • Token metadata (symbols, decimals, names, buckets)

  • Contract ABIs and deploy blocks

  • Block/timestamp lookups

  • Price data

  • Log topics and hashes

  • Results of expensive on-chain or DB queries

Cache Control

  • Many functions accept skip_cache parameters to force fresh computation.

  • TTL and maxsize are configurable for many caches.

  • Some caches are per-chain (partitioned by chain ID).

Summary Table

Cache Type

Library/Decorator

Example Usage

What is Cached

Sync/Async

Local SQL

Pony ORM (SQLite)

y/_db/entities.py

Blockchain data, cache meta

Sync

Joblib Disk

joblib.Memory

y/utils/cache.py, y/contracts.py

Function results

Sync

LRU (in-memory)

functools.lru_cache

y/_db/utils/, y/contracts.py

Function results

Sync

LRU (in-memory)

cachetools.cached/TTL

y/_db/utils/token.py

DB query results

Sync

LRU (in-memory)

async_lru.alru_cache

y/prices/utils/ypriceapi.py

Async function results

Async

LRU (in-memory)

a_sync.a_sync

y/utils/cache.py, y/prices/yearn.py

Sync/async function results

Sync/Async

LRU (in-memory)

db_session_cached

y/_db/decorators.py

DB reads

Sync

References

  • y/utils/cache.py

  • y/_db/entities.py

  • y/_db/decorators.py

  • y/_db/utils/

  • y/contracts.py

  • y/prices/utils/ypriceapi.py