Firewalls
A firewall is a decision the kernel makes about every packet. Allow, drop, return to sender. The decision looks like a few lines of config. Behind it is connection state, rule ordering, hook points, and an entire family of silent failure modes. Watch one work.
Scenario
Rule / packet
What just happened
Nerd tip
conntrack is the invisible ceiling on a busy firewall. Every stateful rule you write relies on netfilter's connection-tracking table. The table size is net.netfilter.nf_conntrack_max, usually a multiple of nf_conntrack_buckets. On stock Linux that is commonly 65,536 or 262,144 entries. A busy edge firewall with NAT on a fleet of clients will fill it, and when it does, dmesg shows nf_conntrack: table full, dropping packet and new connections start getting silently dropped. Size it up, or lose half the nines on your SLO during peak.
Nerd tip
Asymmetric routing breaks stateful firewalls silently. When a SYN leaves through fw-A and the SYN-ACK comes back through fw-B, neither one has a complete flow record. Both mark the packets INVALID and drop them. Your application log shows connection timeouts. You never see a rule hit count go up. The fix is either symmetric routing (so the same firewall sees both halves), or flow-state sync between firewalls (VRRP with conntrack-tools conntrackd), or stateless ACLs on the slow path. Every real HA firewall ships with this problem considered.
Nerd tip
Implicit deny is the contract. A safe firewall config ends with -A FORWARD -j DROP or, in nftables, policy drop;. Any packet that fails to match an explicit allow gets dropped. You audit the config by reading the allow list; the deny is the baseline. The anti-pattern is permissive-default with selective blocks: two thousand rules later, nobody can tell you what is actually forbidden without running a simulator across the whole rule set. The NSA's 2019 firewall audit guide is blunt about this, and nothing about it has aged since.
Nerd tip
Shadowed rules are the silent killers. If rule 5 says accept tcp 0.0.0.0/0 -> 10.0.0.5:443 and rule 40 says drop src 203.0.113.0/24 -> 10.0.0.5:443, rule 40 never fires. Rule 5 already accepted everything it would have seen. Every mature firewall ecosystem has an analyzer: iptables-save | iptables-audit, nftables' own conflict detection, or purpose-built tools like AlgoSec. Order-independent policy languages (Calico, AWS security groups) sidestep the problem entirely by treating rules as a set, not a list.