Part 31: kube-prometheus-stack
"Observability is non-optional. The kube-prometheus-stack is the standard. We install it with the same Helm chart everyone else uses."
Why
kube-prometheus-stack (formerly prometheus-operator) is the standard observability bundle for Kubernetes clusters. It packages:
- Prometheus operator (manages Prometheus instances via CRDs)
- A Prometheus instance preconfigured to scrape kube-apiserver, kube-state-metrics, node-exporter, and any
ServiceMonitor/PodMonitorCRDs the user creates - Alertmanager for routing alerts
- node-exporter as a DaemonSet for host metrics
- kube-state-metrics for Kubernetes object metrics
- Grafana with a curated set of dashboards
- Prometheus rules for the standard k8s alerts (NodeNotReady, PodCrashLooping, EtcdInsufficientMembers, etc.)
The thesis: K8s.Dsl ships KubePrometheusStackHelmReleaseContributor as the default observability stack. Each workload contributor adds ServiceMonitor CRDs to be scraped. The HomeLab event bus from the core also bridges into Prometheus via a small pushgateway sidecar.
The shape
[Injectable(ServiceLifetime.Singleton)]
public sealed class KubePrometheusStackHelmReleaseContributor : IHelmReleaseContributor
{
public string TargetCluster => "*";
public void Contribute(KubernetesBundle bundle)
{
bundle.HelmReleases.Add(new HelmReleaseSpec
{
Name = "kube-prometheus-stack",
Namespace = "monitoring",
Chart = "prometheus-community/kube-prometheus-stack",
Version = "65.5.0",
RepoUrl = "https://prometheus-community.github.io/helm-charts",
CreateNamespace = true,
Wait = true,
Timeout = TimeSpan.FromMinutes(15),
Values = new()
{
["grafana"] = new Dictionary<string, object?>
{
["enabled"] = true,
["adminPassword"] = "{{ secret:GRAFANA_ADMIN_PASSWORD }}",
["ingress"] = new Dictionary<string, object?>
{
["enabled"] = true,
["ingressClassName"] = "nginx",
["annotations"] = new Dictionary<string, object?>
{
["cert-manager.io/cluster-issuer"] = "homelab-ca"
},
["hosts"] = new[] { $"grafana.{_config.Acme.Tld}" },
["tls"] = new[]
{
new Dictionary<string, object?>
{
["secretName"] = "grafana-tls",
["hosts"] = new[] { $"grafana.{_config.Acme.Tld}" }
}
}
},
["persistence"] = new Dictionary<string, object?>
{
["enabled"] = true,
["size"] = "10Gi",
["storageClassName"] = "longhorn"
},
["sidecar"] = new Dictionary<string, object?>
{
["dashboards"] = new Dictionary<string, object?>
{
["enabled"] = true,
["searchNamespace"] = "ALL"
}
}
},
["prometheus"] = new Dictionary<string, object?>
{
["prometheusSpec"] = new Dictionary<string, object?>
{
["replicas"] = _config.K8s?.Topology == "k8s-ha" ? 2 : 1,
["retention"] = "30d",
["storageSpec"] = new Dictionary<string, object?>
{
["volumeClaimTemplate"] = new Dictionary<string, object?>
{
["spec"] = new Dictionary<string, object?>
{
["storageClassName"] = "longhorn",
["accessModes"] = new[] { "ReadWriteOnce" },
["resources"] = new Dictionary<string, object?>
{
["requests"] = new Dictionary<string, object?> { ["storage"] = "20Gi" }
}
}
}
},
["serviceMonitorSelectorNilUsesHelmValues"] = false,
["podMonitorSelectorNilUsesHelmValues"] = false
}
},
["alertmanager"] = new Dictionary<string, object?>
{
["alertmanagerSpec"] = new Dictionary<string, object?>
{
["replicas"] = _config.K8s?.Topology == "k8s-ha" ? 3 : 1
}
}
}
});
}
}The two serviceMonitorSelectorNilUsesHelmValues: false settings tell Prometheus to scrape every ServiceMonitor it can find, regardless of namespace. Without this, the operator only scrapes ServiceMonitors with a specific label, which is one of the most common "why doesn't my dashboard show data" gotchas.
Workload-side ServiceMonitors
Each workload contributor that wants to be scraped declares a ServiceMonitor:
public void Contribute(KubernetesBundle bundle)
{
// The Service exposes a /metrics endpoint
bundle.Services.Add(new ServiceManifest
{
Name = "acme-api-metrics",
Namespace = "acme-prod",
Selector = new() { ["app"] = "acme-api" },
Ports = new[] { new ServicePort { Name = "metrics", Port = 9090, TargetPort = 9090 } }
});
// The ServiceMonitor tells Prometheus to scrape it
bundle.CrdInstances.Add(new RawManifest
{
ApiVersion = "monitoring.coreos.com/v1",
Kind = "ServiceMonitor",
Metadata = new() { Name = "acme-api", Namespace = "acme-prod" },
Spec = new Dictionary<string, object?>
{
["selector"] = new Dictionary<string, object?>
{
["matchLabels"] = new Dictionary<string, object?> { ["app"] = "acme-api" }
},
["endpoints"] = new[]
{
new Dictionary<string, object?>
{
["port"] = "metrics",
["interval"] = "30s",
["path"] = "/metrics"
}
}
}
});
}After this, Prometheus picks up the acme-api service on its next scrape cycle and starts collecting metrics. The Grafana dashboards that the chart pre-installs include a "Pod / Container" view that automatically shows the new metrics without any dashboard editing.
Bridging HomeLab events into Prometheus
The HomeLab event bus from homelab-docker Part 09 publishes typed events from the pipeline. We want those events to show up in Prometheus too. The bridge is a small subscriber that increments counters on the pushgateway:
[Injectable(ServiceLifetime.Singleton)]
public sealed class K8sEventBusObservabilityBridge : IHomeLabEventSubscriber
{
private readonly IPrometheusPushGateway _pushgw;
private readonly Counter _clusterEvents;
private readonly Histogram _stageDuration;
public K8sEventBusObservabilityBridge(IPrometheusPushGateway pushgw)
{
_pushgw = pushgw;
_clusterEvents = Metrics.CreateCounter("homelab_k8s_cluster_events_total", "...", "type", "cluster");
_stageDuration = Metrics.CreateHistogram("homelab_k8s_stage_duration_seconds", "...", "stage", "cluster");
}
public void Subscribe(IHomeLabEventBus bus)
{
bus.Subscribe<ClusterCreated>((e, ct) =>
{
_clusterEvents.WithLabels("created", e.ClusterName).Inc();
return _pushgw.PushAsync(ct);
});
bus.Subscribe<ClusterNodeJoinCompleted>((e, ct) =>
{
_clusterEvents.WithLabels("node_join", e.NodeName).Inc();
return _pushgw.PushAsync(ct);
});
// ... etc
}
}The pushgateway is itself a small Helm release. Now every homelab k8s create, node add, upgrade, backup event shows up in Prometheus and can be queried alongside the rest of the cluster metrics. The dogfood loop closes for observability too.
What this gives you that hand-rolled Prometheus doesn't
A hand-rolled Prometheus is kubectl apply -f prometheus.yaml plus a separate Grafana plus a separate Alertmanager plus per-workload manual scrape configs. The kube-prometheus-stack chart bundles all of it, with sane defaults, with a curated dashboard set, with the operator that handles upgrades.
The bargain pays back the first time you load https://grafana.acme.lab and see the cluster's CPU, memory, network, disk, and per-pod metrics on day one with no dashboard work.