<aside>
🧩 파이프라인 순서대로 각 모듈을 그룹별 서브페이지로 정리했습니다. 본 페이지는 전체 데이터 흐름 + 핵심 패턴 / 알고리즘 일람. · 원본: docs/architecture/
</aside>
Scheduler → Fetcher → Parser → Validate → (planned) Enrich/Embed/Cluster. ProcessingLock · CircuitBreaker · Claim Check · retry ZSET 의 위치를 표기.
flowchart TD
SCH["📡 Scheduler<br/>seed entries"]
PUB["📤 Publisher<br/>chained jobs"]
KH[("🟢 crawl.high")]:::topic
KN[("🟡 crawl.normal")]:::topic
KL[("⚪ crawl.low")]:::topic
SCH -->|CrawlJob| KH & KN & KL
PUB -.->|chained| KH
PM["👷 PoolManager<br/>3-tier consumers"]:::stage
KH & KN & KL --> PM
LOCK[/"🔐 ProcessingLock(Redis)<br/>+ CircuitBreaker"/]:::guard
PM --> LOCK
REG["🗂 Handler Registry<br/>crawler_name → Handler"]:::stage
LOCK --> REG
CH["⛓ ChainHandler<br/>goquery → chromedp"]:::stage
REG --> CH
RC[("📦 raw_contents<br/>Claim Check")]:::store
CH -->|raw HTML| RC
CH -->|RawContentRef| KFE[("🟦 fetched")]:::topic
PW["🎫 ParserWorker<br/>Claim Check"]:::stage
KFE --> PW
RC -.->|GetByID| PW
RULE["🧩 rule.Parser<br/>DB-driven SelectorMap"]:::stage
PW --> RULE
LG["🤖 llmgen<br/>(async fallback)"]:::stage
RULE -.->|ErrNoRule| LG
LG -.->|INSERT enabled=false| RULE
PW -->|list page| PUB
PW -->|article<br/>Content| KNZ[("🟪 normalized")]:::topic
PW -.->|cleanup| RC
VW["✔ Validator<br/>news / community"]:::stage
KNZ --> VW
VW -->|pass| KVA[("🟩 validated")]:::topic
VW -->|fail| DLQ[("🔴 dlq")]:::topic
EN["🔮 Enrich<br/>(planned)"]:::plan
EM["🔮 Embed<br/>(planned)"]:::plan
CLS["🔮 Cluster<br/>(planned)"]:::plan
KVA -.-> EN -.-> EM -.-> CLS
RS[("⏲ retry ZSET<br/>Redis")]:::store
PM -.->|"retry (이슈 #82)"| RS
RS -.-> KH
classDef stage fill:#dbe7ff,stroke:#1a73e8,color:#000
classDef topic fill:#fff4cc,stroke:#b8860b,color:#000
classDef store fill:#d1f4d1,stroke:#2ea44f,color:#000
classDef guard fill:#ffe7d6,stroke:#d97706,color:#000
classDef plan fill:#f0f0f0,stroke:#999,color:#666,stroke-dasharray:3 3
각 그룹은 하위 페이지에 정리됩니다 — 아래 page block 을 참조 (페이지 생성 후 자동 등장).
패턴 사용 위치
------------------------- -------------------------------------------------------------
Stage interface fetcher / parser / validate (cmd/issuetracker wiring)
Claim Check fetcher → parser (raw_contents 테이블, 이슈 #134)
Chain of Responsibility Fetcher chain (goquery → chromedp), LLM chain (provider 순회)
Registry handler.Registry (crawler_name → Handler)
Circuit Breaker fetcher/worker (per-source 실패 트래킹)
Token Bucket fetcher/rate_limiter (per-IP)
3-tier Priority Queues crawl.{high,normal,low} 단일 consumer group
Delayed Retry Queue fetcher/worker.RetryScheduler (Redis ZSET, 이슈 #82)
At-Least-Once 모든 consumer (FetchMessage → 처리 → CommitMessages)
Repository + Service internal/storage 3-layer
3-table Transaction contents + content_bodies + content_meta 단일 INSERT
Adapter classifier (gRPC + HTTP dual-protocol)
Strategy (Provider) pkg/llm — Gemini / OpenAI / Claude 교체
Policy (LLM routing) fixed / cheapest / hybrid / latency
Optimistic Concurrency refiner UpdatePathPattern (guard)
TTL Cache + Regex Cache parser/rule.Resolver (host → rules 5min, regex 1회 compile)
Distributed Lock locks (Redis SETNX + Noop fallback)
Stage Prefix ProcessingKey(stage, url) — fetcher/parser/validator 독립
Atomic Dedup IngestionLock (Publisher enqueue 직전)
Graceful Degrade Redis/LLM 부재 시 Noop fallback (fail-soft)
Fail-Fast DB · Scheduler · Validate · ParsingRules seed
Lock-free Optional Dep Scheduler atomic.Pointer (Gate/Throttler)
RAII (Resource Cleanup) chromedp graceful_timeout (tab leak 방지)