Storage

Overview

Seerflow uses Protocol-based storage interfaces. Two relational backends and three entity-graph backends are swappable via a few config lines:

storage:
  backend: sqlite             # sqlite | postgresql
  graph_backend: igraph       # igraph | falkordb | postgres_age

The four core Protocols — LogStore, AlertStore, ModelStore, EntityStore — are implemented by both relational backends, so application code is unchanged when you move from SQLite to PostgreSQL.

SQLite (default)

Zero-config storage with WAL mode for concurrent reads during writes.

Features

• Auto-created on first run at ~/.local/share/seerflow/seerflow.db (respects $XDG_DATA_HOME and $SEERFLOW_DATA_DIR)
• WAL mode + tuned PRAGMAs (64 MiB cache, 256 MiB mmap)
• FTS5 full-text search on log messages
• Batch writes through WriteBuffer (1000 events or 100 ms)
• ~30K events/sec batched write throughput on commodity SSDs

Schema

Created automatically on first run by the migration runner (forward-only):

• events — indexed columns + msgpack BLOB for the full event
• entity_events — junction table for entity-to-event queries, with the event_ts column denormalized so the planner can drive from the junction index
• alerts — with a dedup_key UNIQUE index
• model_state — key/value store for ML model checkpoints
• templates — Drain3 template metadata (seen count, first/last seen)

Configuration

storage:
  backend: sqlite
  # data_dir: ~/.local/share/seerflow   # default
  # sqlite_path: /custom/path/seerflow.db

PostgreSQL (production)

Production-scale backend, available since v0.5.0. Uses asyncpg with a tunable connection pool.

Install

uv sync --extra postgres
# or
pip install 'seerflow[postgres]'

Configure

storage:
  backend: postgresql
  postgresql_url: ${SEERFLOW_PG_URL}            # required
  postgresql_pool_min_size: 2                   # asyncpg pool floor (>= 1)
  postgresql_pool_max_size: 10                  # asyncpg pool ceiling (>= min)
  postgresql_command_timeout_s: 30.0            # per-query timeout (s)

Features

• Same LogStore / AlertStore / ModelStore / EntityStore Protocols as SQLite
• Forward-only schema migrations under seerflow.storage.postgres_migrations
• Index layout mirrors SQLite for query parity (including the denormalized junction index)
• Batched writes via the same WriteBuffer interface
• Dedicated tables for alert state (_postgres_alerts.py) and Sigma rule state (_postgres_sigma_state.py)

Migrating SQLite → PostgreSQL

A runbook ships in the repo as STORAGE_MIGRATION.md. The high-level flow is:

1. Stand up an empty PostgreSQL 14+ database.
2. Point seerflow.yaml at the new DSN with backend: postgresql.
3. Replay historical events from a SQLite export via seerflow export events --output events.ndjson and seerflow import events.ndjson.

Entity-graph backends

Set with storage.graph_backend:

Backend	When to use	Install	DSN key
igraph (default)	Single-process, in-memory, fastest	(none)	—
falkordb	External, shared graph; Cypher queries	uv sync --extra graph-falkordb	falkordb_url
postgres_age	Co-locate the graph with the relational store	uv sync --extra graph-postgres-age	reuses postgresql_url

postgres_age requires the Apache AGE extension to be installed on the server (CREATE EXTENSION age). On managed Postgres services, confirm AGE is in the provider’s allowlist before relying on it.

To move an existing entity graph between backends, use the migration CLI:

# Dry-run a migration
seerflow graph migrate --from igraph --to falkordb --dry-run

# Real migration (will refuse to overwrite a non-empty destination unless --wipe-destination is passed)
seerflow graph migrate --from igraph --to falkordb

# Downgrade back to the in-process default
seerflow graph migrate --from falkordb --to igraph