МАТЧАСТЬ / MONITORING · SLO · INCIDENTS / DOCUMENT 42 / 03.09.2026

Monitoring, SLA и Incident Management

Как понять, что «Матчасть» действительно работает, заметить деградацию раньше пользователя, не утонуть в алертах и честно общаться во время инцидентов. Документ задаёт наблюдаемость metrics/logs/traces, black-box monitoring, внутренние SLI/SLO, error budgets, alert routing, backup freshness, Search/AI provider health, incident severity, роли, runbooks, status page, postmortems и критерии будущего коммерческого SLA.

User-first SLOизмеряем то, что чувствует пользователь, а не только «контейнер запущен»

External black-boxвнешний probe обязателен: монитор на том же сервере не заметит смерть сервера

Actionable alertsстраница/звонок только когда человеку нужно действовать сейчас

Internal SLO firstдо стабильной статистики не продаём договорное «99.9% SLA»

1. Главное решение

На MVP вводим внутренние SLO, но не публичный договорный SLA. Сервис single-host, recovery automation ещё только проверяется, а реальная статистика доступности отсутствует. Поэтому сначала собираем 60–90 дней данных, проводим restore/incident drills и только затем решаем, какое SLA можно обещать коммерчески.

OBSERVE metrics + logs + traces + black-box ↓ DEFINE USER SLIs ↓ INTERNAL SLO ↓ ERROR BUDGET ↓ ACTIONABLE ALERT ↓ INCIDENT RESPONSE ↓ POSTMORTEM ↓ RELIABILITY WORK ↓ ONLY AFTER EVIDENCE: possible contractual SLA.

2. SLI, SLO и SLA — разные вещи

Термин	Что значит для Mathchast
SLI	Фактически измеряемый показатель: доля успешных запросов, p95 latency, freshness отчётов и т. п.
SLO	Внутренняя цель над SLI, например 99.5% успешных public read events за 30 дней.
SLA	Коммерческое/договорное обязательство с условиями, исключениями и возможными последствиями.

3. Google SRE principle

Google SRE рекомендует определять SLO через поведение, важное пользователю, и отдельно предупреждает, что 100% availability обычно и нереалистична, и нежелательна. Error budget — допустимая доля «плохих» событий внутри выбранного периода.

4. Что пользователь «Матчасти» реально считает доступностью

Reader: открывается public article/company/research Client: может войти видит свой workspace может создать/сохранить draft видит корректные reports Agency: не видит чужой client data может переключать workspace Operator: publication/payment state не теряется Async: jobs eventually complete within promised freshness.

5. Не считать uptime контейнера SLO

Контейнер может быть healthy, а статья отдавать 500, login loop, database queries зависать или reports быть stale неделю.

6. Service map

PUBLIC EXPERIENCE web + API + Postgres + nginx AUTH EXPERIENCE Authentik + app session WORKSPACE API + Postgres + Redis for async ASYNC CONTROL PLANE Redis/BullMQ + workers + scheduler EVIDENCE STORAGE MinIO EXTERNAL OBSERVATION Google/Yandex/AI providers RECOVERY backup repository + WAL archive.

7. Надёжность по пользовательским классам

Class A: public reads Class B: authenticated interactive actions Class C: async jobs Class D: external-data freshness Class E: recovery capability.

8. Initial SLO philosophy

Targets below are launch hypotheses, not contractual commitments. Tighten only after measured real traffic and incident history.

9. Public availability SLI

GOOD: HTTP public request returns expected 2xx/3xx within latency threshold BAD: 5xx timeout broken application response EXCLUDED carefully: obvious abusive probes planned maintenance only if policy says so.

10. Proposed public availability SLO

99.5% over rolling 30 days as first internal target for core public reading paths.

99.5% leaves roughly 3 h 36 min of bad time in a 30-day month if modeled as pure downtime. Это значительно реалистичнее для single-host launch, чем маркетинговое 99.9%.

11. Why not 99.9 immediately

99.9% monthly budget: ≈43 minutes One: host reboot DB recovery network incident bad deployment can consume most of it. Before HA: not honest target.

12. Interactive workspace SLI

Good event: authenticated command/query succeeds within target latency Examples: load workspace save draft submit moderation create order open report.

13. Proposed workspace SLO

99.0% successful eligible interactive requests over 30 days initially.

14. Why lower than public read

Workspace writes touch more dependencies and have lower early traffic. Low-volume systems also make percentage alerting noisy, a problem Google SRE specifically highlights for low-traffic services.

15. Latency SLI

Public cached/read: p95 target < 1.5s server response for core HTML endpoint Interactive API: p95 target < 1.0s for normal DB-backed commands/queries Exclude: AI/PDF/external provider jobs because async.

Starting product targets; real browser Core Web Vitals are tracked separately from server/API latency.

16. No one latency target for everything

Generate PDF and save a title field are not the same workload. SLOs should reflect workload classes.

17. Async completion SLI

Metric: age of oldest eligible job and completion time by job class Classes: publication precheck health check AI run Search sync report email.

18. P0 async targets

Job	Initial objective
Core side effects	95% < 5 min
Publication Health	95% < 15 min after trigger
Transactional email	95% enqueued/sent-to-provider < 5 min
Report render	95% < 15 min after data snapshot ready
Scheduled AI/Search	within planned monitoring window; freshness-based rather than seconds

19. External freshness SLI

For each provider/data source: last_success_at expected_next_at freshness_state FRESH DELAYED STALE ERROR.

20. Provider outage is not always Mathchast outage

Если Gemini API недоступен, но dashboard честно показывает delayed state и publishing работает, это degradation конкретного data source, а не 100% downtime платформы.

21. Data correctness is an SLI too

Examples: wrong client report wrong credit balance 0 instead of unknown cross-tenant response stale metric marked fresh Some correctness failures: more severe than downtime.

22. Security/privacy is not reduced to availability

99.99% uptime with cross-client data leak is not reliable service.

23. Reliability dimensions

availability latency correctness freshness durability isolation recoverability.

24. Recovery SLI

backup age WAL archive age restore drill result measured RPO measured RTO This is: production reliability data, not just ops housekeeping.

25. Recovery targets from Doc 41

Initial hypotheses: Postgres RPO ≤15 min core RTO ≤4 h They become achieved SLOs only: after repeated restore drills.

26. Error budget

SLO 99.5% → error budget 0.5% Use budget to decide: Can we ship risky change? Need reliability sprint? Is incident trend worsening?

27. Error budget is not «permission to be broken»

It is a decision tool balancing reliability and change, not a target to intentionally consume.

28. Error budget policy P0

Budget healthy: normal releases >50% consumed early in window: review top causes >80% consumed: freeze non-critical risky releases until causes addressed 100% exhausted: reliability work takes priority except security/critical business changes.

Mathchast project policy inspired by SRE error-budget practice; thresholds are initial internal rules.

29. Separate budgets

Public read Workspace Async freshness Do not: hide API outage inside huge public-read traffic denominator.

30. Monitoring architecture

APPLICATION OTel SDK │ ▼ OpenTelemetry Collector ├─ metrics → Prometheus ├─ logs → Loki └─ traces → Tempo optional/P1 HOST node_exporter → Prometheus PUBLIC EXTERNAL blackbox probe → monitoring outside host VISUALIZE Grafana ALERT Prometheus rules → Alertmanager → Telegram/email/phone escalation.

31. Why OpenTelemetry

OpenTelemetry provides vendor-neutral traces, metrics and logs. Current documentation recommends Collector use alongside services because it can batch, retry, filter sensitive data and decouple application instrumentation from storage backends.

32. OTel Collector on same host

Single-host deployment can use one Collector gateway container. No need for multi-tier agent/gateway topology until there are several hosts.

33. Collector itself must be monitored

OpenTelemetry security guidance recommends monitoring Collector CPU, memory and throughput and protecting its resources. A dead telemetry collector should not silently make the system «look healthy».

34. Metrics backend

Prometheus. Suitable for app, host, queue and service metrics and for alert-rule evaluation.

35. Logs backend

Loki via native OTLP ingestion. Current Loki supports OpenTelemetry logs directly over OTLP HTTP from the OTel Collector.

36. Promtail is not recommended

Grafana documents Promtail as end-of-life since March 2, 2026. New Mathchast deployment should not introduce an EOL log shipper.

37. Traces backend

Tempo or another OTLP-compatible trace backend is P1 if traces provide diagnostic value. P0 can retain sampled traces in a lightweight backend or start with metrics+logs+request IDs.

38. Why not install every LGTM component immediately

Need first: metrics alerts logs black-box dashboards Trace storage: add when: cross-service/job debugging justifies operational cost.

39. Grafana role

dashboards Explore incident views SLO/error budget panels backup panels provider freshness alert visibility.

40. Alertmanager role

group deduplicate silence inhibit route notifications. Prometheus: evaluates rules.

41. Alert philosophy

Prometheus best practices recommend alerting primarily on user-visible symptoms, keeping alerting simple and avoiding pages where there is nothing actionable to do.

Every page alert must answer: «Что человек должен сделать прямо сейчас?»

42. Page vs ticket

PAGE: urgent human action within minutes TICKET / business-hours: important degradation can safely wait DASHBOARD: diagnostic only.

43. P0 on-call reality

Небольшой проект не должен притворяться 24×7 SRE-отделом. На MVP круглосуточное push/phone escalation — только для SEV1-like critical conditions. SEV2/3 can be business-hours unless contractual support later says otherwise.

44. External black-box monitoring is mandatory

Prometheus on the Mathchast host cannot alert if the entire host, ISP or power disappears.

45. External probe

Run outside production host: GET / GET /articles/canary GET /companies/canary GET /api/public/health-check Validate: DNS TLS HTTP content marker latency.

46. Prometheus blackbox exporter

The Prometheus Blackbox Exporter supports HTTP, HTTPS, DNS, TCP, ICMP and gRPC probes and exposes probe success/timing metrics. It is useful inside a separate monitoring location or secondary host.

47. Best topology

PRIMARY HOST Mathchast + internal monitoring SECOND LOCATION external blackbox backup checks status heartbeat Optional SaaS: independent third probe.

48. External status page

Status page should be hosted outside the main Mathchast failure domain. A status page that disappears together with production is weak incident communication.

49. Status page components

Public Website Client Workspace Publishing Authentication Reports AI Visibility Search Proof Billing Show: Operational Degraded Partial Outage Major Outage Maintenance.

50. Do not expose internal topology

Public status should describe user-facing components, not advertise DB hostnames, network ranges or sensitive incident detail.

51. Synthetic public checks

Homepage: content marker Article: title + source block Company: entity marker Login: OIDC start endpoint not full user password automation initially.

52. Synthetic authenticated check

P1: dedicated synthetic test account/workspace can validate login and read/write flows. Keep credentials isolated and actions clearly marked TEST.

53. Canary publication

Dedicated public: Mathchast system canary Used for: render canonical sitemap search health external probe Not: client content.

54. Host metrics

Prometheus Node Exporter exposes Linux hardware/kernel metrics including CPU, filesystem and network data.

CPU load RAM swap filesystem free inode free disk IO network boot time temperature if available.

55. Host critical alerts

disk nearly full OOM pressure swap thrash filesystem read-only sustained high IO wait time sync broken host reboot unexpected.

56. Disk alert tiers

WARNING: <20% free HIGH: <10% CRITICAL: forecast to full soon or <5% Use: absolute + percent + trend.

Starting thresholds; tune for actual WAL/object growth.

57. Forecast beats threshold

A 2 TB disk at 12% free can be safe for months or doomed in two hours. Dashboard should show growth rate/time-to-full when possible.

58. PostgreSQL monitoring

PostgreSQL 18 includes cumulative statistics views such as pg_stat_activity, pg_stat_database, pg_stat_archiver, pg_stat_wal, pg_stat_io and checkpointer statistics.

connections active/idle transactions deadlocks temp files DB size WAL archiver checkpoints IO locks long queries autovacuum indicators.

59. DB user-facing alerts

query error rate API DB timeout connection exhaustion replication/backup gap disk/WAL risk Prefer: symptom + diagnostic panel.

60. Long queries

Dashboard: top normalized SQL duration calls total time pg_stat_statements: query-level evidence.

61. Connection alert

Warning: pool waiting rises Critical: available DB connections near zero and API latency/errors affected.

62. Backup/WAL monitoring

last WAL archived archive failures last base backup backup repo reachable last restore drill backup size anomaly.

63. Critical backup alert

If WAL archive age exceeds the intended RPO window, that is a critical recovery-risk alert even when users currently see no outage.

64. Restore drill state

last_restore_test duration target_timestamp integrity_checks result RPO achieved RTO achieved.

65. Redis/BullMQ monitoring

Redis memory connections evictions persistence errors queue waiting active failed delayed oldest job age retry count.

66. Queue depth alone is not sufficient

100 tiny jobs can be fine; one 8-hour-old paid report can be serious. Track age and job class.

67. Queue alert examples

Core queue: oldest >5m → warning >15m → high Report: oldest >30m → warning AI monitoring: freshness window missed rather than raw queue count.

68. Dead/failed jobs

failed permanent: visible queue Alert: critical class or burst rate Provide: job ID reason runbook retry action.

69. Scheduler monitoring

last tick due schedules late schedules lock contention enqueue rate scheduler clock.

70. Scheduler dead alert

A scheduler can die while all APIs are green. Heartbeat + overdue schedule count required.

71. MinIO/object monitoring

availability disk/capacity upload failures private download failures backup replication/copy age versioning config check.

72. Auth monitoring

OIDC success/failure callback errors session creation MFA/provider health login latency unexpected auth failure burst.

73. Authentik own metrics/logs

Monitor Authentik as a dependency, but distinguish Mathchast app authorization failures from IdP login failures.

74. Payment monitoring

invoice creation failures webhook signature failure webhook backlog payment state mismatch duplicate/replay attempts credit grant failure.

75. Money correctness alert

Credit balance invariant violation is SEV-level even if website availability is 100%.

76. Billing reconciliation job

periodically compare: provider payment events orders payment rows credit ledger Mismatch: alert + hold automated correction.

77. Publication pipeline monitoring

draft save errors submission errors moderation queue age publish failures render failures cache invalidation failures sitemap/IndexNow side effects.

78. Publication health operational metrics

active paid URLs returning 200 unexpected noindex canonical mismatch broken assets schema render errors.

79. Mass publication incident detection

If 50 URLs fail simultaneously: one platform incident Not: 50 client notifications without grouping.

80. Search provider monitoring

Google: API success quota freshness URL Inspection backlog Yandex: API success quota freshness IndexNow: submission success/response not indexing guarantee.

81. AI provider monitoring

success rate latency provider errors rate limits cost no-answer/refusal parser error citation extraction error model/method change.

82. Provider change event

A model/collection-method change is both product methodology event and observability annotation, not necessarily infrastructure incident.

83. AI run coverage

scheduled observations completed valid provider failure parse failure excluded Coverage: client-visible quality metric and ops metric.

84. AI cost alert

daily provider spend workspace spend run cost unexpected token growth Alert: spend anomaly before budget shock.

85. Local GPU monitoring

GPU utilization VRAM temperature worker OOM job latency model load failures.

86. GPU alerting severity

Local GPU worker failure is usually degradation, not page-worthy outage, because core product continues by design.

87. Application Golden Signals

Google SRE's monitoring chapter popularizes four useful classes: latency, traffic, errors and saturation. Они дают хороший base dashboard for Mathchast API.

LATENCY request/job duration TRAFFIC requests/jobs ERRORS 5xx/domain failures SATURATION CPU/DB pool/queue/GPU/disk.

88. HTTP metrics labels

service route template method status class Avoid: raw URL user ID company name request ID as metric label.

89. Cardinality control

Prometheus labels like publication_id, prompt text or user_id can explode time-series cardinality. IDs belong in traces/logs, not broad metrics.

90. Log labels

Loki index labels: service environment severity component Keep high-cardinality: request_id workspace_id job_id as structured metadata/fields, not index labels.

Grafana's current OTel→Loki guidance also cautions against using high-cardinality resource attributes such as service instance IDs as default index labels for new systems.

91. Structured logs

timestamp level service event request_id trace_id job_id error_code duration_ms safe subject IDs No: raw secrets/private bodies.

92. Request correlation

Browser error → request_id → API log → trace → SQL/job → outbox event → worker job.

93. Trace sampling

P0/P1: sample normal traffic keep errors/high latency more often Never: trace payload sensitive data.

94. Client-specific incident debugging

workspace_id may exist: structured log field Access: ops only Dashboard metrics: aggregate.

95. Dashboards P0

1. Executive service health 2. Public/API 3. Host 4. PostgreSQL 5. Redis/Queues 6. Publishing 7. Search/AI providers 8. Backups/Recovery 9. Billing 10. Security/Auth summary.

96. Executive health dashboard

Public SLO Workspace SLO Current incidents Error budget p95 latency Oldest critical job DB/Redis state Backup freshness External probe Provider freshness.

97. Dashboard colors

Follow Doc 39: colors are semantic but not the only state carrier. Every red/yellow/green panel also has explicit text/status.

98. Alert naming

MathchastPublicHighErrorRate MathchastExternalProbeFailed PostgresWALArchiveStale CriticalQueueOldestJobTooOld PaymentLedgerInvariantFailed BackupRestoreTestFailed.

99. Every alert annotation

summary user impact service severity dashboard runbook started/for likely owner safe context.

100. No alert without runbook after stabilization

If an alert pages twice, document the first diagnostic/mitigation steps.

101. Alert ownership

Grafana alerting best-practice guidance emphasizes that alerts need an owner and system scope; otherwise they tend to be ignored.

owner: ops backend security billing scope: public database AI provider backup agency auth.

102. Group alerts by incident

Grafana notification policies and Alertmanager can group related alert instances, reducing floods when one root failure triggers many symptoms.

103. Inhibition example

PostgresDown firing Suppress: API DB timeout queue DB persist report query failure Still show: child signals in incident dashboard.

104. Host-down inhibition

ExternalHostDown: one urgent incident Do not send: web down api down postgres down redis down minio down as 5 independent pages.

105. Alert timing

Immediate-ish: confirmed external total outage Short sustained window: high API errors Longer: disk growth provider degradation backup freshness.

106. Avoid alert flapping

Use: multi-window evidence small slack grouping appropriate FOR window hysteresis where useful.

107. SLO burn-rate alerting

Google SRE recommends multi-window, multi-burn-rate alerting as a robust way to defend SLOs, rather than paging on every short spike.

108. Launch simplification

Mathchast traffic may initially be too low for sophisticated burn-rate alerting. Start with black-box outage + sustained error/latency + explicit business invariants, then add burn-rate alerts when request volume becomes statistically useful.

109. Low-traffic reality

Google SRE explicitly notes that a single failed request can produce absurd burn rates in low-traffic services. This is relevant to first-month Mathchast workspaces.

110. Low-traffic mitigation

external synthetic traffic aggregate meaningful route classes longer windows absolute failure counts direct invariant alerts sensible SLO target.

111. External synthetic traffic helps

For public reading paths, regular synthetic checks produce a much cleaner availability signal than waiting for organic users at 03:00.

112. Notification channels P0

SEV1: Telegram/phone push equivalent + email fallback SEV2: Telegram/ops channel + email SEV3: ticket/issue/daily review Security: private restricted channel.

113. Avoid alerting only to same host email

If outbound email service or host is part of incident, a second external channel is necessary for critical alerts.

114. Alert test

Monthly: send synthetic test alert verify: route delivery acknowledgment runbook link.

115. Monitoring the monitoring

Prometheus scrape freshness Alertmanager health Grafana availability OTel Collector health external probe heartbeat notification test clock sync.

116. Dead man's switch

External service should expect a periodic heartbeat from production/monitoring. Missing heartbeat detects failure of the whole alert pipeline.

117. Incident definition

Incident = unplanned event that materially impacts: users data integrity security/privacy billing correctness or recovery capability.

118. Not every alert is incident

Alert: possible condition Incident: human/classifier determines material event requiring coordination.

119. Severity model

Severity	Meaning	Examples
SEV1	Critical, broad or irreversible risk	public/core outage, confirmed data leak, data loss, payment corruption, DB unavailable
SEV2	Major partial degradation	workspace unavailable, publishing blocked, critical async backlog, backup RPO at risk
SEV3	Limited degradation	one provider stale, report delay, GPU worker down, non-core feature broken
SEV4	Minor defect / operational issue	cosmetic/admin inconvenience, isolated non-urgent bug

120. Severity based on impact, not root cause

«Redis crashed» не имеет фиксированного SEV. Если no user impact because recovery worked, severity lower than if it blocks all paid workflows.

121. SEV1 response target

Acknowledge: ≤10 min target Incident lead assigned: ≤15 min Status communication if public impact: ≤30 min Update cadence: ~30–60 min depending incident.

Internal launch targets, not contractual SLA.

122. SEV2 response target

Acknowledge: ≤30 min during covered hours Owner: ≤60 min Client/public communication: if meaningful impact or expected duration substantial.

123. SEV3

Business-hours handling ticket/runbook monitor no emergency call unless escalates.

124. Security severity can override availability severity

One private file exposed to the wrong tenant may be SEV1 security incident even if only one user was affected.

125. Incident roles

Incident Commander Technical Lead Communications Scribe Small team: one person may hold two roles, but IC should avoid doing every technical action if possible.

126. Incident Commander

sets severity coordinates chooses priorities assigns owners controls changes decides recovery/status tracks next update.

127. Technical Lead

diagnoses mitigates coordinates technical work reports facts to IC Does not: single-handedly own comms.

128. Scribe

timeline hypotheses actions decisions commands/changes links impact observations.

129. Incident channel/document

INC-2026-009 Severity Start Impact Commander Current hypothesis Actions Timeline Next update Links Resolution.

130. NIST SP 800-61 Rev.3

NIST finalized SP 800-61 Rev.3 in April 2025 and integrates incident response into the full CSF 2.0 risk-management lifecycle rather than treating response as an isolated emergency procedure.

131. Practical incident loop

DETECT ↓ TRIAGE ↓ CONTAIN / MITIGATE ↓ RECOVER ↓ VERIFY ↓ COMMUNICATE ↓ LEARN / IMPROVE.

132. Mitigation before root cause

During active impact, restoring safe service is usually more important than proving the final root cause.

133. Example

Bad deploy → 500s First: rollback Then: investigate exactly which code path failed.

134. Change freeze during incident

SEV1: only incident-related changes unless security emergency SEV2: avoid unrelated risky deploys.

135. Incident actions must be reversible

Prefer: rollback disable flag pause worker read-only mode reduce concurrency Avoid: random DB surgery without backup/audit.

136. Feature flags as mitigation

Doc 40 feature flags become reliability controls: disable broken AI module without taking publishing offline.

137. Read-only mode

If write integrity uncertain: disable orders/drafts writes keep safe public reads show operator notice until DB verified.

138. Provider degradation response

AI provider down: mark DELAYED pause/retry preserve quota notify affected reports if needed Do not: convert missing results to zero.

139. Backup incident response

WAL archive stale: SEV2/SEV1 risk depending duration Actions: stop risky deployments diagnose archive confirm base backup restore recovery capability record RPO gap.

140. Data-integrity incident

wrong credit/report/entity data: freeze affected writes identify scope preserve evidence correct through controlled job audit every change.

141. Cross-tenant incident

Immediate: revoke/fix access path preserve logs identify objects/users rotate sessions/keys if needed legal/privacy response affected-user communication postmortem.

142. Public communications principle

Say what users experience, what is being done, and when next update will come. Do not speculate about unconfirmed root cause.

143. Initial incident message

"Мы фиксируем ошибки при входе в клиентский кабинет. Публичные публикации доступны. Команда расследует проблему. Следующее обновление — до 14:30 МСК."

144. Bad incident message

"У нас небольшие технические работы, скоро всё заработает :)" Problems: hides impact no scope no next update may be false.

145. Resolution message

Resolved start/end user impact what restored whether data affected follow-up/postmortem if appropriate.

146. No premature «no data loss»

Do not make security/integrity claims until verified.

147. Maintenance

planned window scope expected impact start/end rollback condition Public notice: only if users affected.

148. Maintenance is not free unlimited exclusion

Internal SLO policy must define which planned maintenance is excluded. Otherwise every outage can be relabeled maintenance after the fact.

149. Deployment annotation

Every production release: timestamp release ID git SHA Grafana: vertical annotation Incident: see changes near onset.

150. Configuration-change annotation

DB config queue concurrency provider mode feature flag nginx auth Audit + monitoring annotation for high-impact changes.

151. Incident timeline sources

alerts deploys audit logs traces provider status operator actions customer reports.

152. Customer report as signal

If customer repeatedly discovers outages before monitoring, monitoring itself requires postmortem action.

153. Postmortem triggers

Any SEV1 material SEV2 data loss/corruption security/privacy incident manual discovery of major outage restore failure repeated same incident near miss with high potential impact.

154. Google SRE postmortem practice

Google SRE treats postmortems as a structured way to document impact, response, contributing causes and follow-up actions, and advocates blameless postmortems focused on improving systems rather than punishing individuals.

155. Postmortem template

Title / Incident ID Date Severity Summary User impact Duration Detection Timeline Root/contributing factors What went well What went poorly Where we got lucky Mitigation Recovery Corrective actions Owners/dates SLO/error-budget impact.

156. Root cause is often plural

Prefer contributing factors over forcing one simplistic «root cause».

157. Blameless ≠ accountability-free

Do not write: "developer forgot" Ask: Why could one action reach production? Why no test? Why no rollback guard? Why alert late?

158. Action items need owner/date

BAD: "Improve monitoring" GOOD: "Add external canary for /article and alert after 3 failed probes. Owner X, Sep 12."

159. Postmortem action classes

PREVENT DETECT MITIGATE RECOVER COMMUNICATE DOCUMENT.

160. Not every action must prevent recurrence

Sometimes faster detection/mitigation is much cheaper and more valuable than impossible total prevention.

161. Postmortem review

Draft: within 2–5 business days for material incident Review: owner/team Track actions: until closed.

Internal operating target.

162. Public postmortem?

For severe public incidents, publish a customer-safe summary when useful. Security investigations may require delayed/limited disclosure.

163. Internal postmortem repository

incident ID category service cause tags severity duration SLO impact actions repeat relation.

164. Incident taxonomy

DEPLOYMENT DATABASE HOST NETWORK AUTH QUEUE PROVIDER BILLING DATA_INTEGRITY SECURITY BACKUP STORAGE UNKNOWN.

165. Reliability backlog

Postmortem actions SLO burn recurring alerts capacity risk restore gaps toil → engineering priority.

166. Alert quality review

For every page: Was it actionable? Was severity right? Was duplicate? Did it fire early enough? Did it resolve correctly? Could it be a ticket instead?

167. Alert KPI

pages/month false/no-action pages duplicate pages MTTA alerts discovered before users manual-discovery incidents.

168. Do not optimize MTTD blindly

Faster alerts that create 100 false pages are worse. Precision and actionability matter.

169. Core incident metrics

MTTD: detect after impact MTTA: acknowledge MTTM: mitigate MTTR: restore Impact: bad requests/users/minutes Use: trends, not employee ranking.

170. No engineer leaderboard

Do not rank people by MTTR. It incentivizes hiding incidents and discourages escalation.

171. Reliability KPI

SLO attainment error budget SEV1/2 count repeat incidents backup restore success manual-discovery rate critical action closure.

172. Availability calculation

Prefer: good events / eligible events For low traffic: combine with synthetic checks Do not rely only: "server process uptime".

173. SLO windows

Operational: rolling 30d Trend: weekly/monthly Commercial future: calendar month if contract defines it.

174. SLO exclusions

Potential: client device/browser issue client's own DNS/domain third-party AI provider for provider-specific SLO scheduled maintenance if predeclared abusive traffic Need: explicit rules before measurement.

175. Do not exclude provider outage from everything

AI provider outage may be excluded from core publishing availability but included in AI Visibility freshness SLO.

176. Internal SLO matrix P0

Capability	Initial internal objective
Public core read availability	99.5% / rolling 30d
Workspace interactive success	99.0% / rolling 30d
Public core p95 server latency	<1.5s
Normal API p95	<1.0s
Core async side effects	95% <5m
Publication Health trigger	95% <15m
Postgres recovery target	RPO ≤15m, RTO ≤4h after tested

177. Why no AI freshness number in matrix

AI providers, tiers and client schedules differ. Each plan should define its expected run cadence and freshness window instead of one universal 5-minute target.

178. AI monitoring freshness example

Weekly plan: due Monday 08:00–12:00 FRESH: completed within window DELAYED: window missed <24h STALE: >24h after window Exact: plan/methodology specific.

179. Search data freshness

Search Console: source itself has lag Mathchast monitors: ingestion vs source-available data Do not alert: because today's final Google data doesn't exist yet.

180. Provider-aware freshness

Freshness SLI should know expected provider lag, not compare every source to real time.

181. Business invariant alerts

published ACTIVE URL → must 200 credit ledger → no negative invalid balance paid order → exactly one logical payment report → metrics snapshot exists workspace → client entity relation valid review verified → relation evidence exists.

182. Invariants can catch silent corruption

Some of the most important alerts do not come from CPU graphs.

183. Synthetic invariant job

Hourly/daily: sample/check critical invariants On failure: create incident candidate with affected IDs.

184. Data-quality monitoring

unknown encoded as zero denominator missing metric version mismatch prompt set mismatch provider method changed report stale Alert: data pipeline quality.

185. Publication canary after deploy

Deploy → smoke: homepage article company login save draft queue event health snapshot Fail: rollback/flag before broad use.

186. Deployment health window

Observe: 5xx p95 DB queue memory external probe for initial post-deploy window before declaring complete.

187. Release rollback trigger

new release correlates with: major error increase auth failure data invariant failure SLO fast burn → rollback first unless rollback unsafe.

188. Change failure tracking

deploys rollbacks incidents caused hotfixes Useful: release quality trend.

189. Capacity monitoring

CPU RAM disk DB size WAL/day queue throughput AI cost editorial queue provider quotas.

190. Editorial ops can create service degradation

Mathchast is not only software. If moderation SLA stretches from 2 days to 14 days because workload exceeds capacity, product reliability degrades even with perfect servers.

191. Human workflow SLI

submission → first editorial decision needs-change → re-review verification request → decision dispute → response Track: median/p90 by service tier.

192. Human SLO vs infra SLO

Keep them separate. «Website 99.5% available» does not imply «editor responds in 24h».

193. Future service-level commitments

Potential commercial SLA/SLE: platform uptime support response report delivery publishing review window Each: separately defined.

194. Do not promise publication approval SLA

Can promise review/response timing, not guaranteed editorial approval.

195. Future SLA prerequisites

≥90 days telemetry known incident rate restore drills off-host monitoring operational ownership support coverage defined exclusions legal contract review capacity model.

196. Single-host SLA ceiling

A single production host without automatic failover is structurally weak for aggressive contractual uptime. If enterprise requires 99.9–99.95+, architecture likely needs second failure domain and tested failover.

197. When to add second production node

Trigger: SLO routinely threatened by host maintenance/failure enterprise SLA demand traffic/revenue justifies recovery time insufficient Then: separate app nodes DB HA strategy external storage/Redis careful failover.

198. Do not add HA only for badge

HA adds replication/failover failure modes. It must be tested, not just diagrammed.

199. Status history

Public status: incident timeline resolved maintenance history Internal: full detailed incident record.

200. Customer notifications

Channels: status page email for major account impact in-app after recovery agency contacts No: send every short 30s blip to all clients.

201. Targeted communication

If: only AI Visibility affected Notify: affected monitoring customers Do not: email readers who only use public articles.

202. Agency incident communication

Agency sees: affected clients shared issue current status Avoid: 20 duplicate alerts for same platform incident.

203. Incident object data model

incident_id severity status started_at detected_at ack_at mitigated_at resolved_at services[] impact commander cause_category related_deploy public_status postmortem_id.

204. Incident event model

incident_event: timestamp actor type message evidence_link visibility: INTERNAL / PUBLIC_DRAFT.

205. Alert → incident linkage

alert instances: many incident: one Link: alert IDs dashboard snapshot logs/traces.

206. Postmortem data model

postmortem_id incident_id summary impact timeline factors lessons actions[] reviewed_at visibility.

207. Corrective action data

action class owner priority due issue/PR status closed_at verification.

208. Repeat incident detection

tag: DATABASE_DISK AUTH_CALLBACK PROVIDER_RATE_LIMIT New incident: show previous related postmortems and open actions.

209. Reliability review cadence

Weekly: alerts/incidents backup capacity Monthly: SLO/error budget provider health postmortem actions Quarterly: DR drill SLO target review major risk review.

210. Weekly ops review

SLO incidents pages disk DB queues backup restore provider costs open SEV actions capacity.

211. No dashboard without owner

Dashboards that nobody checks are decoration. Each operational dashboard has a purpose and review cadence.

212. Dashboard retention

Metrics: 15–90d high resolution longer aggregates as useful Logs: shorter, e.g. 14–30d launch security/audit separate Traces: short, sampled. Tune: storage/cost.

Initial retention hypotheses; privacy/security and actual disk consumption control final values.

213. Do not store raw logs forever

Logs can contain identifiers and consume huge storage. Audit records have a different retention policy from debug logs.

214. Metrics backup

Operational metrics can be rebuildable/non-critical compared with Postgres business data. Do not let monitoring backup complexity threaten core backups.

215. Incident evidence snapshot

For SEV1/2: save relevant: dashboard snapshot/export logs window deploy ID queries provider status within retention window.

216. Provider status pages

During external outage: check provider status but verify our own metrics too Provider says green: does not prove our account/quota works.

217. Third-party dependency dashboard

Authentik Email Payment Google Yandex AI providers Off-host backup State: up/degraded/stale last success latency/quota.

218. Provider circuit breaker

Repeated failure: reduce concurrency backoff pause low priority protect quotas keep critical jobs. Avoid: retry storm.

219. Retry storm alert

provider errors + job retries + queue growth + outbound rate Detect: before system overload.

220. Database incident runbook P0

1. confirm user impact 2. connection/storage health 3. check recent deploy/config 4. stop risky jobs 5. read-only if needed 6. rollback/restore decision 7. verify integrity 8. resume 9. incident/postmortem.

221. Disk-full runbook P0

1. identify growth 2. protect Postgres/WAL 3. stop noncritical generators 4. free safe cache/temp only 5. extend/migrate storage if needed 6. verify backup/archive 7. investigate root growth.

222. Queue backlog runbook

1. identify job class 2. worker healthy? 3. provider limited? 4. DB/Redis? 5. poison job? 6. adjust safe concurrency 7. prioritize paid/due jobs 8. never drop silently.

223. External probe failure runbook

1. compare multiple locations 2. DNS/TLS 3. nginx 4. host/network 5. public canary 6. status incident 7. do not reboot randomly without evidence.

224. Backup stale runbook

1. stop risky changes 2. check pg_stat_archiver/tool 3. repo reachability/space 4. re-establish WAL 5. take/verify backup 6. record RPO exposure 7. restore test if uncertainty.

225. Auth incident runbook

1. determine Authentik vs app 2. provider health 3. callback/config 4. session/signing keys 5. recent change 6. preserve existing sessions if safe 7. rollback/fix 8. test admin/client flows.

226. Security incidents use separate runbook

Do not paste sensitive forensic details into public incident channel/status page.

227. Alert runbook format

Meaning User impact Likely causes First 5 checks Safe mitigations Unsafe actions Dashboard Logs query Escalation Related incidents.

228. Runbooks in Git

Version runbooks with code/infra. Every alert links to stable rendered/internal path.

229. Emergency contacts

primary secondary hosting/network payment email legal/privacy backup access Stored: secure + offline copy.

230. Vendor support dependency

If enterprise support relies on vendor tickets, record provider support tier and expected response; do not promise client resolution faster than dependency reality without fallback.

231. Monitoring stack failure domain

Same-host: Prometheus/Grafana/Loki good for diagnosis Off-host: black-box + heartbeat good for total-host detection Need both.

232. External logs/metrics SaaS?

Optional future resilience: send selected critical metrics/alerts to external observability service. Not mandatory P0 if independent black-box and off-host alerts exist.

233. Monitoring storage pressure

Prometheus/Loki: own disk quotas retention alerts Never: monitoring fills disk and kills Postgres.

234. Separate volume/budget

At minimum enforce retention and disk quotas/alerts so observability cannot consume the whole NVMe.

235. Metrics cardinality budget

Review new metrics: label count expected cardinality retention cost Reject: prompt_id/user_id/publication_id as broad labels.

236. Log sampling

Do not sample: errors/security/audit Can reduce: successful debug/info noise at high scale.

237. Production log level

INFO structured WARN/ERROR DEBUG: temporary scoped feature/service with expiry.

238. Debug mode expiry

Temporary verbose logging must have an owner and removal time, especially because it can leak data and fill disk.

239. Client-facing reliability dashboard?

P2/Enterprise: historical uptime/support metrics can be exposed. MVP public status page is enough.

240. Internal vs external reliability

Internal: fine-grained SLOs error budgets provider/DB metrics External: status and relevant service history Do not: publish misleading vanity uptime without methodology.

241. SLA credits?

Do not introduce service credits before commercial SLA, billing rules and uptime measurement are legally/product reviewed.

242. Future SLA structure

service covered measurement source monthly target exclusions maintenance support hours incident communication credit remedy claim window force majeure/dependencies.

243. Uptime ≠ editorial turnaround

Contract should separate technical service availability from editorial production/support commitments.

244. Reliability feature prioritization

Priority: user impact error budget burn incident recurrence data risk recovery risk Not: most visually impressive Grafana panel.

245. P0 monitoring stack

Prometheus Node Exporter Postgres metrics exporter/instrumentation Redis/app metrics OpenTelemetry Collector Loki Grafana Alertmanager external black-box probe heartbeat/dead-man status page.

246. P0 application instrumentation

HTTP: requests/errors/latency Jobs: duration/result/queue DB: pool/query symptoms Business: publish/payment invariants Providers: success/latency/quota/freshness Backup: WAL/base/restore.

247. P0 alerts

External public outage high 5xx DB unavailable disk critical WAL backup stale queue critical age payment invariant cross-tenant/security signal scheduler dead backup restore failed auth broad failure.

248. P1

SLO burn-rate alerts trace backend synthetic authenticated flow advanced capacity forecast provider circuit dashboards public status automation incident bot/templates automated postmortem metadata dependency SLOs.

249. P2

multi-location probes secondary production node HA/failover monitoring commercial SLA enterprise support external observability copy advanced anomaly detection chaos/failover drills.

250. What NOT to do

Не делать	Почему
Обещать 99.9% SLA на single-host до данных	нет доказанной способности выполнить
Мониторить только контейнеры	не видит user experience
Хранить monitoring только на prod host	не видит total-host outage
Page на каждый warning	alert fatigue
Считать missing provider data нулём	ложные отчёты
Использовать high-cardinality IDs как metric labels	monitoring explosion
Скрывать инцидент generic «техработами»	trust loss
Закрывать postmortem без action owners	инцидент повторится
Судить сотрудников по MTTR	плохие incentives

251. Acceptance test: host disappears

Power/network loss Expected: internal stack disappears external probe detects dead-man triggers SEV1 notification arrives status page remains reachable.

252. Acceptance test: Postgres down

DB unavailable Expected: API readiness fails public external probe detects impact dependent alerts grouped/inhibited one incident runbook linked.

253. Acceptance test: provider down

AI provider 503 Expected: publishing remains available jobs backoff freshness becomes delayed client metric not zeroed SEV3/2 based on scope.

254. Acceptance test: scheduler dead

Scheduler container stops Expected: heartbeat fails overdue schedules rise alert fires API can remain healthy incident diagnosed before client report date.

255. Acceptance test: disk growth

Loki starts consuming disk fast Expected: forecast/threshold alert monitoring retention limits Postgres protected no disk-full surprise.

256. Acceptance test: backup stale

WAL archive stopped 20 min Expected: critical recovery-risk alert release caution/freeze repair RPO exposure recorded.

257. Acceptance test: false alert

One isolated 500 at low traffic Expected: does not wake operator unless business invariant/security indicates serious issue.

258. Acceptance test: deploy incident

Release increases 5xx Expected: deploy annotation visible alert rollback metrics recover incident linked to release.

259. Acceptance test: cross-tenant event

Authorization test fails or real leakage detected Expected: SEV1 security path containment audit preservation privacy/legal process not treated as ordinary 403 bug.

260. Acceptance test: monitoring stack dies

Prometheus/Alertmanager down while app remains online Expected: external dead-man detects monitoring failure itself and notification comes through independent path.

261. Launch checklist

[ ] external probes [ ] dead-man heartbeat [ ] Prometheus [ ] Grafana [ ] Loki / OTel logs [ ] Alertmanager [ ] host metrics [ ] DB metrics [ ] Redis/queue metrics [ ] app metrics [ ] provider freshness [ ] backup/WAL metrics [ ] public canary [ ] runbooks [ ] severity policy [ ] status page [ ] alert test [ ] restore drill [ ] incident template [ ] postmortem template.

262. First 30 days after launch

Week 1: fix missing signals remove noisy alerts Week 2: baseline latency/errors tune dashboards Week 3: first SLO report review queues/providers Week 4: alert test + restore test update runbooks.

263. 60-day reliability review

SLO actual incidents manual discoveries alert noise recovery test provider degradation capacity Decision: tighten/relax targets and prioritize reliability.

264. 90-day SLA decision

Questions: Can we prove uptime? How often host failed? RTO achieved? Support coverage? Enterprise demand? Any SLO exhaustion? Need second node? Only then: commercial SLA design.

265. Suggested first commercial posture

«Мы публикуем status history и внутренне измеряем доступность, но на стандартных тарифах не обещаем договорный процент uptime до накопления операционной статистики.»

266. Enterprise conversation

If client requires: 99.9/99.95 24×7 response strict RTO Treat as: architecture/support requirement with separate price/contract, not a checkbox.

267. Reliability as product trust

Mathchast sells evidence and transparency. Its own operations should follow the same philosophy: measured SLOs, visible status, explicit uncertainty and evidence-backed postmortems.

268. Main operating loop

MEASURE ↓ NOTICE ↓ RESPOND ↓ RECOVER ↓ EXPLAIN ↓ LEARN ↓ HARDEN.

269. Решение документа

Утвердить observability/SLO/incident baseline. Mathchast instruments app and workers with OpenTelemetry; Prometheus stores/evaluates metrics, Grafana visualizes, Loki receives logs via native OTLP, and Alertmanager groups/routes alerts. A probe and dead-man heartbeat outside the production host are mandatory because same-host monitoring cannot detect total host/network loss; public status communication must also live outside the primary failure domain. Initial internal SLOs are deliberately modest: approximately 99.5% rolling-30d core public read availability and 99.0% workspace interactive success, with p95 targets around 1.5s for core public server responses and 1.0s for ordinary synchronous API operations. Async services use job-age/freshness SLOs rather than synchronous latency, and provider outages create DELAYED/STALE states instead of false zero metrics. Core reliability dimensions include availability, latency, correctness, freshness, durability, isolation and recoverability. Backup/WAL age and restore-test results are first-class operational metrics; a missed RPO window pages even without user-visible downtime. Alerts follow Prometheus/SRE principles: page on actionable user pain or critical data/recovery risk, group dependent symptoms, and use business-hours tickets for non-urgent degradation. Incident severity is SEV1–SEV4, with security/data-integrity impact able to override pure availability impact. Material incidents use an Incident Commander, timeline, status updates and a blameless postmortem with owned corrective actions. Error budgets guide release/reliability priorities, but sophisticated burn-rate paging is introduced only when traffic is sufficient; low-traffic launch relies more heavily on external synthetic checks, absolute failures and invariants. No contractual uptime SLA is sold on MVP. A commercial SLA is reconsidered only after roughly 60–90 days of telemetry, repeated restore drills, known support coverage and evidence that single-host architecture can meet the proposed target—or after implementing a second failure domain if stricter availability is required.

270. Что этот документ разблокирует

Mathchast_42 Monitoring / SLA / Incidents → Mathchast_43 MVP Scope / Roadmap → финальная сборка всей системы.

Источники исследования

The exact SLO percentages, latency targets, response times, severity thresholds, retention periods, alert windows and commercial-SLA gates in this document are Mathchast launch hypotheses, not external guarantees. They should be recalibrated after 30/60/90 days of real telemetry and after measured restore/incident drills. The final contractual SLA, if introduced, requires legal review and may require architectural changes beyond the single-host MVP.