Skip to main content
Welcome. This site supports keyboard navigation and screen readers. Press ? at any time for keyboard shortcuts. Press [ to focus the sidebar, ] to focus the content. High-contrast themes are available via the toolbar.
serard@dev00:~/cv

Part 21: metrics-server and the Basics

"Without metrics-server, kubectl top does not work and HorizontalPodAutoscalers do not function. With it, both do."

Why

metrics-server is a small cluster-wide pod that scrapes the kubelet's /metrics/resource endpoint on every node and exposes the data via the Kubernetes Metrics API (metrics.k8s.io). Without it:

  • kubectl top pod returns "metrics-server not installed"
  • kubectl top node returns the same
  • HorizontalPodAutoscaler cannot read CPU/memory and cannot scale workloads
  • Several dashboards in kube-prometheus-stack (Part 31) show "no data"

kube-state-metrics is a different thing: it scrapes the Kubernetes API server and exposes metrics about Kubernetes objects themselves (deployments, pods, services, jobs, etc.) in Prometheus format. Without it, Prometheus has no way to know how many pods are unhealthy or how many deployments have stale ReplicaSets.

Both are mandatory for any cluster that wants observability. K8s.Dsl ships both as IHelmReleaseContributors.

The thesis: two small Helm releases. Both unconditional. Both small (~50 MB and ~25 MB image sizes). Both enable downstream observability without the user thinking about them.

The shape

[Injectable(ServiceLifetime.Singleton)]
public sealed class MetricsServerHelmReleaseContributor : IHelmReleaseContributor
{
    public string TargetCluster => "*";

    public void Contribute(KubernetesBundle bundle)
    {
        bundle.HelmReleases.Add(new HelmReleaseSpec
        {
            Name = "metrics-server",
            Namespace = "kube-system",
            Chart = "metrics-server/metrics-server",
            Version = "3.12.2",
            RepoUrl = "https://kubernetes-sigs.github.io/metrics-server/",
            Wait = true,
            Values = new()
            {
                ["replicas"] = _config.K8s?.Topology == "k8s-ha" ? 2 : 1,
                ["args"] = new[]
                {
                    "--cert-dir=/tmp",
                    "--secure-port=10250",
                    "--kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname",
                    "--kubelet-use-node-status-port",
                    "--metric-resolution=15s",
                    "--kubelet-insecure-tls"   // necessary because kubelets serve self-signed certs
                }
            }
        });
    }
}

[Injectable(ServiceLifetime.Singleton)]
public sealed class KubeStateMetricsHelmReleaseContributor : IHelmReleaseContributor
{
    public string TargetCluster => "*";

    public void Contribute(KubernetesBundle bundle)
    {
        bundle.HelmReleases.Add(new HelmReleaseSpec
        {
            Name = "kube-state-metrics",
            Namespace = "kube-system",
            Chart = "prometheus-community/kube-state-metrics",
            Version = "5.27.0",
            RepoUrl = "https://prometheus-community.github.io/helm-charts",
            Wait = true,
            Values = new()
            {
                ["replicas"] = 1,
                ["metricLabelsAllowlist"] = new[]
                {
                    "namespaces=[*]",
                    "pods=[app,app.kubernetes.io/name,app.kubernetes.io/instance]"
                }
            }
        });
    }
}

The --kubelet-insecure-tls flag is necessary because kubeadm (and k3s) generate self-signed kubelet certs by default. Production clusters that use cert-manager-issued kubelet certs can drop this flag, but for HomeLab K8s the simpler path is to accept the self-signed kubelet certs.

Why both, not just metrics-server

A common confusion: developers see "metrics-server" and "kube-state-metrics" and think they are the same thing. They are not. The data they expose is complementary:

Component Data type API exposed
metrics-server Resource usage (CPU, memory) per pod and node metrics.k8s.io (Kubernetes Metrics API)
kube-state-metrics Kubernetes object state (replicas, conditions, restart counts, age) Prometheus /metrics endpoint

You need both. metrics-server is for kubectl top and HPA. kube-state-metrics is for Prometheus and Grafana. Installing one without the other leaves you with half the data.

The HPA test

Once metrics-server is installed, HPAs work. A test workload that demonstrates the integration:

public void Contribute(KubernetesBundle bundle)
{
    bundle.Deployments.Add(new DeploymentManifest
    {
        Name = "load-test",
        Namespace = "default",
        Replicas = 1,
        PodSpec = new PodSpec
        {
            Containers = new[]
            {
                new Container
                {
                    Name = "load",
                    Image = "nginx:1.27",
                    Resources = new ResourceRequirements
                    {
                        Requests = new() { ["cpu"] = "100m", ["memory"] = "128Mi" },
                        Limits = new() { ["cpu"] = "500m", ["memory"] = "256Mi" }
                    }
                }
            }
        }
    });

    bundle.CrdInstances.Add(new RawManifest
    {
        ApiVersion = "autoscaling/v2",
        Kind = "HorizontalPodAutoscaler",
        Metadata = new() { Name = "load-test", Namespace = "default" },
        Spec = new Dictionary<string, object?>
        {
            ["scaleTargetRef"] = new Dictionary<string, object?>
            {
                ["apiVersion"] = "apps/v1",
                ["kind"] = "Deployment",
                ["name"] = "load-test"
            },
            ["minReplicas"] = 1,
            ["maxReplicas"] = 5,
            ["metrics"] = new[]
            {
                new Dictionary<string, object?>
                {
                    ["type"] = "Resource",
                    ["resource"] = new Dictionary<string, object?>
                    {
                        ["name"] = "cpu",
                        ["target"] = new Dictionary<string, object?>
                        {
                            ["type"] = "Utilization",
                            ["averageUtilization"] = 50
                        }
                    }
                }
            }
        }
    });
}

Apply this, run kubectl run -it --rm load-tester --image=busybox -- sh -c "while true; do wget -O- http://load-test; done" against the service, and watch the HPA scale from 1 to 3 to 5 over a few minutes. Without metrics-server, the HPA would sit at 1 forever and the test would silently fail.

What this gives you

Two small contributors. Both run unconditionally. Both enable major downstream features (HPA, top, dashboards). The cost is ~80 MB of RAM total across the cluster (metrics-server's deployment uses ~25 MB; kube-state-metrics uses ~30 MB; both have small overhead). The benefit is that every other observability piece in the rest of the series works — kube-prometheus-stack from Part 31, Grafana dashboards, autoscaling tests, capacity planning.

End of Act III

We have now built the cluster machinery: distribution choice, node image, kubeadm bootstrap, worker join, CNI, CSI, ingress, cert-manager, external-dns, metrics-server. By the end of Act III, homelab k8s create produces a fully working Kubernetes cluster with HTTPS, DNS, persistent storage, and the basic observability needed by everything that comes next. There are no toy components in the cluster — every choice (Cilium, Longhorn, nginx-ingress or Traefik, cert-manager, external-dns, metrics-server) is the same one a production cluster would use.

Act IV tightens this into the three concrete topologies we keep mentioning: k8s-single, k8s-multi, k8s-ha. Each one is a configured combination of the components from Act III, sized against the budgets from Part 02.

Cross-links

⬇ Download