CtrlK
BlogDocsLog inGet started
Tessl Logo

nitinjain999/platform-skills

Production-grade platform engineering handbook — Kubernetes, Terraform, Flux CD, GitHub Actions, AWS, and more.

67

Quality

84%

Does it follow best practices?

Impact

No eval scenarios have been run

SecuritybySnyk

Passed

No known issues

Overview
Quality
Evals
Security
Files

runtime-security.mdreferences/

Runtime Security Reference

Covers Falco — eBPF-based syscall monitoring for Kubernetes workloads. Falco is a CNCF project, open-source, and free to run. Complements supply chain security (pre-deployment controls) with in-cluster threat detection (post-deployment controls).


Architecture

Falco runs as a DaemonSet on every node. It uses eBPF to hook into the Linux kernel and observe syscalls made by all containers on that node — without modifying the containers or the images.

Kernel syscalls (open, exec, connect, …)
  → Falco eBPF probe (per node)
  → Falco engine (rules evaluation)
  → Alert output
      ├── stdout (default)
      ├── Falcosidekick (fan-out: Slack, PagerDuty, SNS, webhook)
      └── gRPC output (for programmatic consumers)

Driver options

DriverRequiresUse when
modern_ebpfLinux 5.8+ kernel (BTF enabled)EKS (AL2023), GKE (COS), recommended default
ebpf (classic)Kernel headers on nodeOlder nodes without BTF support
kmod (kernel module)Build toolchain + privilegedNever use on managed K8s — breaks on node OS upgrades

Always use eBPF on managed Kubernetes. The kernel module requires privileged access and breaks whenever the node OS is upgraded (e.g., EKS AMI rotation, GKE node auto-upgrade).


Installing Falco on EKS

# falco-values.yaml — EKS with modern_ebpf driver
driver:
  kind: modern_ebpf   # no kernel headers required; needs Linux 5.8+ with BTF

falco:
  grpc:
    enabled: true     # enables gRPC output for programmatic consumers
  grpc_output:
    enabled: true
  json_output: true   # structured output; required for Falcosidekick parsing
  priority: warning   # suppress DEBUG and INFO from default ruleset

resources:
  requests:
    cpu: 100m
    memory: 256Mi
  limits:
    memory: 512Mi     # cpu limit intentionally omitted — kernel event processing is bursty

tolerations:
  - effect: NoSchedule
    operator: Exists  # Falco must run on ALL nodes including tainted ones
  - effect: NoExecute
    operator: Exists

Bottlerocket nodes: set driver.kind: modern_ebpf — Bottlerocket does not ship kernel headers.

Fargate: Falco cannot run on Fargate. Fargate does not expose the node kernel to DaemonSets.


Installing Falco on GKE

GKE Container-Optimized OS (COS) requires modern_ebpf (BTF enabled by default). Standard Ubuntu nodes can use either driver.

driver:
  kind: modern_ebpf

tolerations:
  - effect: NoSchedule
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists

GKE Autopilot: DaemonSets are not schedulable on Autopilot — Falco is not supported.


Built-in Ruleset

The default Falco ruleset (/etc/falco/falco_rules.yaml) ships ~200 rules. Key ones to enable in production:

RulePriorityWhat it detects
Terminal shell in containerNOTICEInteractive shell spawned inside a running container
Privilege Escalation via SudoWARNINGsudo or su executed in a container
Write below etcERRORWrite to /etc inside a container
Contact K8S API Server From ContainerNOTICEContainer calling the K8s API directly
Unexpected outbound connection destinationNOTICEOutbound connection to unexpected IP/port
Launch Privileged ContainerWARNINGContainer started with --privileged

Tune noise by adding process names or image repositories to built-in allow-list macros rather than disabling rules entirely.


Writing Custom Rules

Rule structure

- rule: Unexpected binary executed in web container
  desc: A binary not in the known-good set was exec'd in the web-tier container
  condition: >
    spawned_process
    and container
    and container.image.repository = "ghcr.io/<org>/web"
    and not proc.name in (node, npm, sh, bash)
  output: >
    Unexpected binary in web container
    (user=%user.name image=%container.image.repository
     binary=%proc.name parent=%proc.pname cmdline=%proc.cmdline
     pod=%k8s.pod.name ns=%k8s.ns.name)
  priority: WARNING
  tags: [container, execution, custom]

Key condition fields

FieldTypeExample
spawned_processmacronew process was exec'd
containermacroevent is inside a container
proc.namestringnode, bash
proc.cmdlinestringfull command including args
container.image.repositorystringghcr.io/org/image
user.namestringunix username
fd.netmacroevent involves a network fd
evt.typestringexecve, open, connect

Lists and macros

- list: allowed_web_binaries
  items: [node, npm, sh]

- macro: web_container
  condition: container.image.repository = "ghcr.io/<org>/web"

- rule: Unexpected binary in web container
  condition: spawned_process and web_container and not proc.name in (allowed_web_binaries)
  output: >
    Unexpected binary in web container
    (user=%user.name binary=%proc.name cmdline=%proc.cmdline pod=%k8s.pod.name)
  priority: WARNING
  tags: [container, execution, custom]

Loading custom rules via Helm

customRules:
  custom-rules.yaml: |-
    - list: allowed_web_binaries
      items: [node, npm, sh]
    - rule: Unexpected binary in web container
      condition: spawned_process and container and not proc.name in (allowed_web_binaries)
      output: "Binary %proc.name in container %container.name"
      priority: WARNING

Testing rules

# Deploy the official event-generator to trigger known-bad syscall patterns
kubectl run event-generator \
  --image=falcosecurity/event-generator \
  --restart=Never \
  --rm -it -- run syscall

# Watch Falco logs on the same node
kubectl logs -n falco -l app.kubernetes.io/name=falco | grep WARNING

Falcosidekick: Alert Routing

Falcosidekick fans out Falco alerts to 50+ outputs. Enable alongside Falco:

helm upgrade --install falco falcosecurity/falco \
  --namespace falco \
  --create-namespace \
  --set falcosidekick.enabled=true \
  --set falcosidekick.webui.enabled=true \
  -f falco-values.yaml

Slack configuration

falcosidekick:
  config:
    slack:
      webhookurl: "https://hooks.slack.com/services/<token>"
      minimumpriority: warning   # suppress DEBUG and INFO
      messageformat: >
        :rotating_light: *{{ .Rule }}* ({{ .Priority }})
        Container: `{{ index .OutputFields "container.name" }}`
        Image: `{{ index .OutputFields "container.image.repository" }}`
        Cmdline: `{{ index .OutputFields "proc.cmdline" }}`
        Pod: `{{ index .OutputFields "k8s.pod.name" }}` in `{{ index .OutputFields "k8s.ns.name" }}`

Output types (selection)

OutputConfig keyUse case
Slackslack.webhookurlTeam alert channel
Webhookwebhook.addressCustom handler or SIEM
AWS SNSaws.sns.topicarnAWS-native alert pipeline
PagerDutypagerduty.routingkeyOn-call escalation

Bridging Falco → Kyverno

Falco detects threats at runtime. Kyverno can prevent re-admission of flagged workloads.

Pattern

  1. Falco alert fires (e.g., shell in container)
  2. Alert webhook handler labels the Deployment: security.platform/falco-alert: critical
  3. Kyverno ValidatingPolicy blocks CREATE/UPDATE on Deployments with that label
  4. Ops team reviews, remediates, removes the label
apiVersion: policies.kyverno.io/v1
kind: ValidatingPolicy
metadata:
  name: block-falco-flagged-workloads
spec:
  validationActions: [Deny]
  matchConstraints:
    resourceRules:
    - apiGroups: ["apps"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["deployments"]
  validations:
  - expression: >
      !has(object.metadata.labels) ||
      !('security.platform/falco-alert' in object.metadata.labels) ||
      object.metadata.labels['security.platform/falco-alert'] != 'critical'
    message: >
      This Deployment is flagged by a Falco critical alert. Remediate and remove
      the security.platform/falco-alert label before redeploying.

See examples/runtime-security/falco-kyverno-bridge.yaml for the full policy with annotations.


Resource Sizing

ComponentCPU requestMemory requestMemory limit
Falco DaemonSet100m256Mi512Mi
Falcosidekick Deployment100m128Mi256Mi
Falcosidekick UI Deployment50m64Mi128Mi

Do not set CPU limits on Falco. Kernel event processing is bursty; a CPU limit causes throttling and missed events.


Troubleshooting

SymptomLikely causeFix
Rule not firingWrong condition field or event typeTest with falco-event-generator; check Falco startup logs for rule load errors
Rule loaded but no alertEvent not reaching rule conditionRun event-generator; check falco.priority threshold
High CPU on Falco podToo many low-priority eventsAdd conditions to narrow scope; raise falco.priority to warning
Missed eventsCPU throttlingRemove limits.cpu from Falco DaemonSet
Falcosidekick not receivingFalco gRPC not enabledSet falco.grpc.enabled: true and falco.grpc_output.enabled: true
DaemonSet not on tainted nodeMissing tolerationAdd tolerations: [{effect: NoSchedule, operator: Exists}]
No events on GKEWrong driver for COSSet driver.kind: modern_ebpf
failed to load moduleUsing kmod driver on managed K8sSwitch to driver.kind: modern_ebpf
Custom rules not loadedWrong mount pathRules load from /etc/falco/rules.d/, not /etc/falco/

BEFORE_AFTER.md

CHANGELOG.md

CODE_OF_CONDUCT.md

COMMANDS.md

CONTRIBUTING.md

EDITOR_INTEGRATIONS.md

GETTING_STARTED.md

HOW_IT_WORKS.md

install.sh

INSTALLATION.md

LAUNCH.md

PROMPTS.md

QUICKSTART.md

README.md

renovate.json

SECURITY.md

SKILL.md

tessl.json

tile.json