Overall System Architecture

Source: Notion | Last edited: 2025-12-03 | ID: 29b2d2dc-3ef...

QuantOS · End-to-End System Architecture (Master Map)

flowchart LR
  %% ===== LAYER: SOURCES =====
  subgraph SRC["Data Sources"]
    MKT["Markets<br/>(trades • quotes • orderbooks • bars)"]
    REF["Reference<br/>(fundamentals • corp actions • calendars)"]
    ALT["Alt Data<br/>(news • social • RSS)"]
    ONC["On-chain<br/>(RPC • indexers)"]
  end

  %% ===== LAYER: STREAMING =====
  subgraph STRM["Streaming & Contracts"]
    RP["Redpanda / Kafka"]
    SR["Schema Registry<br/>(Avro/Protobuf)"]
    SPC["Stream Processing<br/>(Materialize / Flink)"]
  end

  %% ===== LAYER: LAKEHOUSE & ANALYTICS =====
  subgraph LAKE["Lakehouse & Analytics"]
    S3["Object Store<br/>(S3 / MinIO)"]
    ICB["Iceberg Catalog<br/>(Glue / Nessie)"]
    CH["OLAP / Timeseries<br/>(ClickHouse)"]
    SQL["SQL Access<br/>(Trino / Athena / DuckDB)"]
    OBJ["Raw Objects<br/>(docs • JSON • media)"]
    VEC["Vector DB<br/>(pgvector / Qdrant)"]
  end

  %% ===== LAYER: FEATURE =====
  subgraph FEAT["Feature Layer"]
    OFF["Offline Feature Store<br/>(Iceberg tables)"]
    ONL["Online Feature Store<br/>(Feast → Redis / ClickHouse)"]
  end

  %% ===== LAYER: STRATEGY & COMPILER =====
  subgraph STRAT["Strategy & Compiler"]
    DSL["Strategy DSL"]
    CMP["Compiler → DAG<br/>(JSON/YAML)"]
    REG["Registry<br/>(plugins • models • strategies)"]
    ART["Artifacts Store<br/>(S3 / OCI registry)"]
  end

  %% ===== LAYER: ORCHESTRATION =====
  subgraph ORCH["Orchestration"]
    Q["Job Queue<br/>(Kafka topics / Argo Events)"]
    WF["Workflow Engine<br/>(Argo / Airflow)"]
    TMP["Job Templates<br/>(K8s CRDs • RayJobs • Helm)"]
    POOLS["Compute Pools<br/>(Ray • Spark • GPU/CPU • Spot)"]
    SCL["Autoscaling<br/>(KEDA • Cluster Autoscaler)"]
    SCH["Scheduling Policy<br/>(cost • latency • locality)"]
  end

  %% ===== LAYER: EXECUTION =====
  subgraph EXEC["Execution Layer"]
    INT["Intent Normalizer<br/>(schema • validation)"]
    RSK["Pre-Trade Risk<br/>(limits • exposure • sanity)"]
    OMS["Order Manager<br/>(state • retry • ttl)"]
    RTR["Smart Router<br/>(venue • order type • slippage)"]
    ADP["Venue Adapters<br/>(Binance • OKX • IBKR • FIX)"]
    SIM["Simulation Gateway<br/>(paper • shadow)"]
    AUD["Audit Journal<br/>(immutable events → CH)"]
    POS["Positions / PnL<br/>(real-time state)"]
    CPL["Compliance / OPA<br/>(geo • KYC • throttle)"]
  end

  %% ===== LAYER: ACCESS =====
  subgraph API["Access & Serving"]
    FLT["Arrow Flight / FlightSQL"]
    GRP["gRPC / REST / WebSocket"]
  end

  %% ===== LAYER: GOVERNANCE & OBS =====
  subgraph GOV["Governance • Quality • Observability"]
    OMD["OpenMetadata<br/>(catalog)"]
    OLG["OpenLineage<br/>(lineage)"]
    DQ["Data Quality<br/>(Great Expectations)"]
    OBS["Observability<br/>(Prom • Grafana • Loki • Tempo)"]
    SEC["Secrets & Keys<br/>(Vault • KMS • External Secrets)"]
    PRM["Promotion Gates<br/>(staging → canary → prod)"]
  end

  %% ===== LAYER: CONSUMERS =====
  subgraph CONS["Consumers"]
    AG["AI Agents"]
    RUN["QuantOS Runner"]
    NB["Research Notebooks"]
    UI["Ops / Admin UI"]
  end

  %% ===== FLOWS =====
  %% Sources → Streaming
  MKT --> RP
  REF --> RP
  ALT --> RP
  ONC --> RP
  RP --> SR
  RP --> SPC

  %% Streaming → Lakehouse / OLAP / Objects / Vector
  RP --> S3
  SPC --> S3
  S3 --> ICB
  S3 --> CH
  ALT --> OBJ
  OBJ --> VEC

  %% Lakehouse/OLAP → Feature
  S3 --> OFF
  CH --> OFF
  OFF --> ONL

  %% Registry & Artifacts
  REG --- ART

  %% Strategy → Compiler → Orchestration
  DSL --> CMP
  REG --> DSL
  CMP --> Q
  Q --> WF
  WF --> TMP
  TMP --> POOLS
  WF --> SCL
  WF --> SCH

  %% Orchestration outputs
  POOLS --> ART
  POOLS --> OLG
  POOLS --> OBS
  POOLS --> OFF

  %% Feature / Access to Consumers
  OFF --> FLT
  ONL --> FLT
  CH --> FLT
  FLT --> RUN
  FLT --> AG
  GRP --> RUN
  GRP --> AG
  SQL --> NB

  %% Runner / Agents → Execution
  RUN --> INT
  AG --> INT
  INT --> RSK --> OMS --> RTR --> ADP
  RTR --> SIM
  ADP --> POS
  ADP --> AUD
  RSK --> CPL
  AUD --> CH

  %% Governance & Security hooks
  S3 --> OMD
  RP --> OMD
  OFF --> OMD
  CH --> OMD
  OLG --- OMD
  DQ --- OMD
  SEC --> EXEC
  SEC --> ORCH
  PRM --> EXEC
  PRM --> ORCH

  %% Admin UI
  UI --> GOV
  UI --> ORCH
  UI --> EXEC

What this map shows (at a glance)

Data path: Sources → Streaming → Lakehouse/OLAP → Feature (offline/online)
Strategy path: DSL → Compiler → Orchestration (Argo/Ray) → Artifacts/Features
Execution path: Runner/Agents → Intents → Risk/OMS/Router → Venues → Audit/Positions
Access path: Arrow Flight / gRPC for high-throughput, low-latency serving to Agents/Runner
Controls: Governance (catalog, lineage, quality), Observability, Secrets, Promotion gates

ASCII fallback (for docs that don’t render Mermaid)

[Sources] → [Redpanda/Kafka + Schema] → [S3+Iceberg | ClickHouse | Objects+Vector]
                   ↓                                  ↓
               [Stream Proc]                   [Offline Features] → [Online Features]
                                                     ↓                    ↓
[DSL] → [Compiler→DAG] → [Queue] → [Workflow(Argo)] → [Compute Pools(Ray/Spark)] → [Artifacts/Reports]

[Runner/Agents] → [Intents] → [Risk] → [OMS] → [Router] → [Venue Adapters] → [Positions/PnL]
                                              ↘ [Simulation]       ↘ [Audit→ClickHouse]

[Access] Arrow Flight / gRPC  → Agents / Runner / Notebooks
[Governance] Metadata • Lineage • Quality • Secrets • Promotion • Observability

If you want, I can also split this master map into four smaller diagrams (Data/Feature, Strategy/Compiler, Orchestration, Execution) for slide decks, each with “You are here” highlights.