Part 16: CNI — Flannel vs Calico vs Cilium
"Pick the CNI you run in production. There is no reason to pick a different one for dev."
Why
A Kubernetes cluster needs a CNI plugin to give pods IP addresses, route traffic between them, and enforce NetworkPolicies. The choice locks in capabilities: some CNIs do not enforce policies, some do not support eBPF, some do not support BGP for the underlay. Choosing the wrong one for dev means you cannot test the production policies in dev.
The thesis: K8s.Dsl ships three CNI contributors as IHelmReleaseContributor (or raw manifest contributor for Flannel, which has no Helm chart). The user picks one per cluster via k8s.cni. The pluggability is the same as elsewhere: a future plugin can ship a fourth contributor for Antrea or Kube-OVN.
The matrix
| CNI | Network policy | eBPF | BGP | Pod CIDR isolation | Best for |
|---|---|---|---|---|---|
| Flannel | No (use kube-router or net-policy add-on) | No | No | Yes | Smallest dev clusters |
| Calico | Yes (full NetworkPolicy + Calico CRDs) | Optional | Yes | Yes | Realistic dev, BGP testing |
| Cilium | Yes (NetworkPolicy + CiliumNetworkPolicy with L7) | Yes | Yes (via BGP CP) | Yes | Modern, observability-rich, defaults |
The K8s.Dsl default is Cilium. It is the new standard for new clusters in 2025+ and supports the broadest feature set. Flannel and Calico are kept as alternatives because some teams have production clusters on those and want dev to match.
Flannel
[Injectable(ServiceLifetime.Singleton)]
public sealed class FlannelCniContributor : IK8sManifestContributor
{
public string TargetCluster => "*";
public bool ShouldContribute() => _config.K8s?.Cni == "flannel";
public void Contribute(KubernetesBundle bundle)
{
if (!ShouldContribute()) return;
bundle.CrdInstances.Add(new RawManifest
{
ApiVersion = "v1",
Kind = "Namespace",
Metadata = new() { Name = "kube-flannel", Labels = new() { ["pod-security.kubernetes.io/enforce"] = "privileged" } }
});
// The full Flannel manifest is large; we ship it as an embedded resource
// and inject the pod-network-cidr from the cluster config
var flannelYaml = EmbeddedResources.LoadFlannelManifest();
var podCidr = _config.K8s?.Kubeadm?.PodSubnet ?? "10.244.0.0/16";
flannelYaml = flannelYaml.Replace("{{POD_CIDR}}", podCidr);
bundle.CrdInstances.AddRange(KubernetesYamlDeserializer.SplitDocuments(flannelYaml));
}
}Flannel is the simplest. ~600 lines of YAML for the daemonset, the configmap, the RBAC. We ship it as an embedded resource (downloaded once at K8s.Dsl build time, pinned to a specific version) and substitute the pod CIDR. No Helm chart, no operator, no CRDs beyond the standard ones.
Calico
[Injectable(ServiceLifetime.Singleton)]
public sealed class CalicoHelmReleaseContributor : IHelmReleaseContributor
{
public string TargetCluster => "*";
public bool ShouldContribute() => _config.K8s?.Cni == "calico";
public void Contribute(KubernetesBundle bundle)
{
if (!ShouldContribute()) return;
bundle.HelmReleases.Add(new HelmReleaseSpec
{
Name = "calico",
Namespace = "tigera-operator",
Chart = "projectcalico/tigera-operator",
Version = "v3.29.1",
RepoUrl = "https://docs.tigera.io/calico/charts",
CreateNamespace = true,
Wait = true,
Values = new()
{
["installation"] = new Dictionary<string, object?>
{
["kubernetesProvider"] = "",
["calicoNetwork"] = new Dictionary<string, object?>
{
["bgp"] = "Disabled", // BGP off by default; user enables explicitly
["ipPools"] = new[]
{
new Dictionary<string, object?>
{
["cidr"] = _config.K8s?.Kubeadm?.PodSubnet ?? "10.244.0.0/16",
["encapsulation"] = "VXLAN",
["natOutgoing"] = "Enabled"
}
}
}
}
}
});
}
}Calico is shipped as a Helm release via the Tigera operator. The operator manages the Calico components (calico-node, calico-kube-controllers) and exposes the configuration through the Installation CRD. The contributor configures the IP pool with the right pod CIDR.
Cilium
[Injectable(ServiceLifetime.Singleton)]
public sealed class CiliumHelmReleaseContributor : IHelmReleaseContributor
{
public string TargetCluster => "*";
public bool ShouldContribute() =>
_config.K8s?.Cni == "cilium" || _config.K8s?.Cni is null; // default
public void Contribute(KubernetesBundle bundle)
{
if (!ShouldContribute()) return;
var podCidr = _config.K8s?.Kubeadm?.PodSubnet ?? "10.244.0.0/16";
bundle.HelmReleases.Add(new HelmReleaseSpec
{
Name = "cilium",
Namespace = "kube-system",
Chart = "cilium/cilium",
Version = "1.16.4",
RepoUrl = "https://helm.cilium.io/",
Wait = true,
Timeout = TimeSpan.FromMinutes(10),
Values = new()
{
["ipam"] = new Dictionary<string, object?>
{
["mode"] = "kubernetes",
["operator"] = new Dictionary<string, object?>
{
["clusterPoolIPv4PodCIDRList"] = new[] { podCidr }
}
},
["kubeProxyReplacement"] = "true", // replace kube-proxy entirely
["k8sServiceHost"] = _config.K8s?.ApiVip ?? _config.K8s?.ApiAdvertiseAddress,
["k8sServicePort"] = 6443,
["hubble"] = new Dictionary<string, object?>
{
["enabled"] = true,
["relay"] = new Dictionary<string, object?> { ["enabled"] = true },
["ui"] = new Dictionary<string, object?> { ["enabled"] = true }
},
["operator"] = new Dictionary<string, object?>
{
["replicas"] = _config.K8s?.Topology == "k8s-ha" ? 2 : 1
}
}
});
}
}Cilium is the heaviest contributor (it has the most options) and the most powerful. We enable Hubble (the eBPF observability layer) by default because it is the killer feature compared to the other two. We replace kube-proxy entirely (kubeProxyReplacement: true) because Cilium's eBPF-based service routing is faster and gives us network observability we would not otherwise have.
NetworkPolicy support
The K8s.Dsl spec from Part 06 declared a NetworkPolicy concept with a constraint: it can only be generated when the active CNI supports policies. The constraint is:
public static ConstraintResult NetworkPolicyRequiresPolicyCapableCni(ConceptValidationContext ctx)
{
var cni = ctx.GetGlobalProperty("k8s.cni") as string ?? "cilium";
return cni switch
{
"flannel" => ConstraintResult.Failed("NetworkPolicy is declared but the active CNI 'flannel' does not enforce them. Use 'calico' or 'cilium', or add the 'kube-network-policies' add-on."),
"calico" or "cilium" => ConstraintResult.Satisfied(),
_ => ConstraintResult.Failed($"unknown CNI: {cni}")
};
}The constraint runs at validation time. A user who declares a [NetworkPolicy] on Flannel sees the error before the cluster is ever bootstrapped, with a clear path to fix it.
The default-deny pattern
The Namespace concept from Part 06 had a DefaultDeny flag that defaulted to true. When the active CNI is policy-capable, K8s.Dsl emits a default-deny NetworkPolicy for every namespace marked with DefaultDeny:
[Injectable(ServiceLifetime.Singleton)]
public sealed class DefaultDenyNetworkPolicyContributor : IK8sManifestContributor
{
public string TargetCluster => "*";
public void Contribute(KubernetesBundle bundle)
{
var cni = _config.K8s?.Cni ?? "cilium";
if (cni == "flannel") return; // no-op on flannel
foreach (var ns in bundle.Namespaces.Values.Where(n => n.DefaultDeny))
{
bundle.NetworkPolicies.Add(new NetworkPolicyManifest
{
Name = "default-deny",
Namespace = ns.Name,
PodSelector = new Dictionary<string, string>(), // match all
PolicyTypes = new[] { "Ingress", "Egress" }
});
}
}
}Every namespace starts with deny-all and explicit allow rules are added by the workload contributors. This catches misconfigured pods early — a workload that forgets to declare its egress policy cannot reach external services until the developer notices and adds the rule.
The test
[Fact]
public void cilium_is_the_default_when_cni_is_unspecified()
{
var bundle = new KubernetesBundle();
var contributor = new CiliumHelmReleaseContributor(
Options.Create(new HomeLabConfig { K8s = new() { Cni = null } }));
contributor.ShouldContribute().Should().BeTrue();
contributor.Contribute(bundle);
bundle.HelmReleases.Should().ContainSingle(h => h.Name == "cilium");
}
[Fact]
public void flannel_does_not_run_when_cni_is_calico()
{
var bundle = new KubernetesBundle();
var flannel = new FlannelCniContributor(Options.Create(new HomeLabConfig { K8s = new() { Cni = "calico" } }));
flannel.ShouldContribute().Should().BeFalse();
flannel.Contribute(bundle);
bundle.CrdInstances.Should().BeEmpty();
}
[Fact]
public void network_policy_constraint_fails_on_flannel()
{
var ctx = new ConceptValidationContext();
ctx.SetGlobalProperty("k8s.cni", "flannel");
var result = NetworkPolicyConstraints.NetworkPolicyRequiresPolicyCapableCni(ctx);
result.IsSatisfied.Should().BeFalse();
result.Message.Should().Contain("flannel");
result.Message.Should().Contain("calico");
result.Message.Should().Contain("cilium");
}
[Fact]
public void default_deny_contributor_emits_one_policy_per_default_deny_namespace()
{
var bundle = new KubernetesBundle();
bundle.Namespaces["acme-prod"] = new NamespaceManifest { Name = "acme-prod", DefaultDeny = true };
bundle.Namespaces["acme-public"] = new NamespaceManifest { Name = "acme-public", DefaultDeny = false };
var contributor = new DefaultDenyNetworkPolicyContributor(
Options.Create(new HomeLabConfig { K8s = new() { Cni = "cilium" } }));
contributor.Contribute(bundle);
bundle.NetworkPolicies.Should().ContainSingle(np => np.Namespace == "acme-prod");
bundle.NetworkPolicies.Should().NotContain(np => np.Namespace == "acme-public");
}What this gives you that picking-one-and-hoping doesn't
A typical kubeadm install picks Calico (or Flannel, depending on the docs page you read) and the user lives with it. If the production cluster runs Cilium, the dev cluster running Calico is a different cluster for the purposes of NetworkPolicy testing — Calico's CRDs are not Cilium's, and the L7 features simply do not exist.
A typed three-CNI plugin set with a constraint validator gives you, for the same surface area:
- One config field to pick the CNI
- Three contributors that produce the right install
- Default-deny NetworkPolicy for every namespace marked as such (where the CNI supports it)
- A constraint validator that fails fast when NetworkPolicy is declared on a non-policy CNI
- Plugin extensibility for a fourth CNI (Antrea, Kube-OVN, etc.)
The bargain pays back the first time you write a CiliumNetworkPolicy with an L7 HTTP rule and watch it work locally exactly the way it will work in production.