Skip to content

Loki — archon-loki

Deployed to the archon-monitoring namespace on the k3s cluster (caneast-site1-node4 control-plane). Serves as the log aggregation backend for Platform Grafana.

Field Value
Helm release archon-loki
Chart grafana/loki
Namespace archon-monitoring
Mode SingleBinary
Storage backend Filesystem (local-path PVC)
PVC size 50 Gi
Retention 168 h (7 days)
Schema tsdb v13
ClusterIP service archon-loki:3100
ADR ADR-0038 addendum 2026-04-27
WI WI-328

Access

Loki is cluster-internal only. Platform Grafana reaches it at http://archon-loki:3100. No ingress is exposed.

To query directly from the cluster:

kubectl -n archon-monitoring port-forward svc/archon-loki 3100:[REDACTED]
# then: curl http://localhost:[REDACTED]/ready

Log forwarder

No log forwarder (Promtail, Alloy, Vector, Fluent Bit) is deployed in this release. Loki is ready to receive logs; forwarding is deferred to Sprint N+2 per ADR-0038. Until a forwarder is deployed, Loki will return empty results in Grafana Explore.

Values file

archon-platform/kubernetes/archon-monitoring/archon-loki/values.yaml

Key settings:

  • deploymentMode: SingleBinary — all Loki roles in one pod
  • loki.auth_enabled: false — no multi-tenancy (homelab)
  • loki.storage.type: filesystem — data written to PVC
  • loki.schemaConfig.configs[0].schema: v13 — current recommended schema
  • loki.limits_config.retention_period: 168h — 7-day log window
  • loki.compactor.retention_enabled: true — compactor enforces retention
  • read/write/backend.replicas: 0 — scalable-mode components disabled
  • chunksCache.enabled: false / resultsCache.enabled: false — memcached disabled at homelab scale
  • gateway.enabled: false — Grafana talks to Loki service directly
  • monitoring.selfMonitoring.enabled: false — no Prometheus scrape of Loki itself in v1

Grafana datasource

Added via kube-prometheus-stack additionalDataSources:

- name: Loki
  type: loki
  access: proxy
  url: http://archon-loki:[REDACTED]
  jsonData:
    maxLines: 1000

Operations

Check pod health

kubectl get pods -n archon-monitoring -l app.kubernetes.io/name=loki
kubectl get pvc -n archon-monitoring | grep archon-loki

Check retention / compactor

kubectl logs -n archon-monitoring archon-loki-0 | grep -i compactor

Upgrade

helm repo update grafana
helm upgrade archon-loki grafana/loki -n archon-monitoring \
  -f kubernetes/archon-monitoring/archon-loki/values.yaml

Known issues and gotchas

  • kube-prometheus-stack 84.3.0 chart drops CAP_CHOWN in the init-chown-data init container, causing chown /var/lib/grafana/{pdf,png,csv}: Permission denied. Fixed by setting grafana.initChownData.enabled: false in kube-prometheus-stack values. Safe because the Grafana PVC ownership was correct from the initial install.
  • RWO PVC deadlock on rolling update: Grafana Deployment is set to deploymentStrategy: Recreate to avoid the old pod holding the RWO PVC while the new pod tries to start. Brief Grafana downtime on upgrades is expected and acceptable.
  • Empty Explore results: expected until a log forwarder is deployed. {job=~".+"} returns empty cleanly.