How to Supercharge Your Falco

Read the article to learn all the details on how to supercharge your Falco

Roman Myronov

2/11/2022 6 min read

Falco is an open-source cloud-based solution focused on runtime monitoring of anomalous application behavior and changes to configuration data in your Kubernetes environment. It is also capable of detecting cyber intrusions and data thefts. Falco was created in 2016 by Sysdig, an open-source company, and they later donated this project to Cloud Native Computing Foundation (CNCF). It is credited as the first run-time security project to join the CNCF as an “incubating” project. CNCF projects have three levels of maturity in ascending order: sandbox, incubating, and graduated based on their intended customers. While sandbox projects are ideal for innovators, incubating projects are for early adopters. Graduated projects are best suited for early majority adopters. With over 30 million downloads, as in October 2021, Falco is the de facto threat detection engine for containers and Kubernetes.     

How Falco works 

Falco monitors events from multiple sources and applies a default set of security rules (defined in its configuration file) to them to detect and trigger alerts on any unusual behavior that is in violation of such rules. Such event sources include Linux system calls (syscall) from containers and Kubernetes audit events (k8s_audit). A syscall represents the basic communication between an application and the Linux kernel. This is generally invoked via a wrapper (shell script) in the emulation layer over the kernel. A syscall event happens when an application invokes a syscall. Alternatively, it occurs when the hardware signals the processor to interrupt the code it is executing in order to process another external event on priority.     

The tool draws on the continuous flow of syscalls on a machine and passes these to your user space, i.e., the packaged user application programs inside a container image. Software libraries in the user space sift through syscalls, after which the Falco’s rich set of security rules filter out suspicious events. These rules make use of powerful Sysdig filtering expressions, such as comparison operators, which compare expressions to return “True” or “False” results. Multiple checks are combined using Boolean operators (and, or, not). Falco sends security alerts to one or more output channels like Syslog, standard output, https URL, as well as to client applications via a gRPC API. The output might also take the form of a file or a child process spawned for the purpose.  

Since version v0.13.0, Falco has been supporting Kubernetes Audit Events, which maintains a log of all the requests and responses to the Kubernetes API server (kube-apiserver). The server is responsible for validating and structuring data for various API objects (e.g., pods, services, pod replication controllers) as well as overall cluster management. So, Kubernetes Audit Events present a chronological record of the changes made to the cluster.  

What are Falco rules?

Falco’s rules have been framed keeping in mind the specific requirements of Kubernetes, Linux, and cloud-native ecosystems. Essentially, the rules define the conditions under which Falco will generate an alert in real time such as, say, unauthorized access to the Secure Shell (SSH) network communication protocol used to connect to a cluster node. The powerful monitoring tool is capable of surfacing various anomalous behaviors in your Linux kernel that indicate the operating system is under attack by malicious hackers. Running a shell command inside a container unauthorizedly to perform some action is another suspicious activity. Another creepy scenario is where the child process spawned by a main server process, supposedly to handle client request. But it turns out to be of an unexpected type and starts running codes with root privileges like the main process. The escalation of user privilege or any attempt to do so in order to gain root access to encrypted passwords of system users (stored in confidential files, like “/etc/shadow”) is also an instance of odd systems behavior. Typically, these are some of the most popular methods hackers employ to gain illegal access to container services. The Falco rules engines checks for all of these unauthorized actions and much more. As a matter of fact, the tool checks the kernel for an entire set of unusual behaviors as below. 

Falco’s checklist of unusual behaviors

  • Privilege escalation using privileged containers

  • Namespace changes using tools like setns

  • Read/Writes to well-known directories such as /etc, /usr/bin, /usr/sbin, etc.Creating symlinks

  • Ownership and Mode changes

  • Unexpected network connections or socket mutations

  • Spawned processes using execve

  • Executing shell binaries such as sh, bash, csh, zsh, etc. 

  • Executing SSH binaries such as ssh, scp, sftp, etc

  • Mutating Linux coreutils executables

  • Mutating login binaries

  • A standard system binary, such as ls, is making an outbound network connection.Mutating shadowutil or passwd executables such as shadowconfig, pwck, chpasswd, getpasswd, change, useradd, etc, and others. 

Falco architecture 

Custom rules definition

A salient feature of the Falco security monitoring tool is that it allows you to create new rules and modify existing ones in line with the specifics of your user environment. For instance, you may want to define a custom rule to trigger an alert where the tool comes across a piece of software that is risky or malicious for the container. At the same time, you want to whitelist external libraries and build tools commonly accessed by containers. Falco’s rules engine is flexible enough to accommodate all of these custom scenarios. Its rule set comes with a set of defined rule-condition snippets (“macros”), which you can use inside the rules you define or within other macros. Apart from rules and macros, Falco also features lists, a collection of items that can be appended to rules, macros and other lists. 

Pre-defined macros and lists trigger notifications for attempts by containers to establish outbound connections with suspicious resources in the external world. That’s not all. Together, macros and lists help enhance your user experience around framing custom rules. Lists, by the way, are very granular by nature and cannot be filtered or drilled down any further. On top of these features, the tool lets you override macros if your enterprise’s cybersecurity posture requires that. The team behind Falco is working to create a command line tool (“falcoctl”) in Go open-source programming language for deploying and managing security rules.  

eBPF Probe for improved workload monitoring 

It doesn’t take much to get started with Falco and deploy the tool across all the nodes in your Kubernetes cluster. You can install the tool using the DaemonSet feature and Helm charts, the file collection format used by the Kubernetes’ Helm package manager. The Falco rules engines requires a system driver to collect the stream of system call data and pass on the same to user space. This consumption of data from the kernel has been traditionally performed by the kernel module and default driver called Falco. However, more recently, kernels have started supporting another kind of driver, namely, the Berkeley Packet Filter (eBPF) probe. eBPF has some clear advantages over the basic kernel driver of the past. 

Firstly, eBPF will only inject such code into your Linux kernel that is absolutely safe to run! The code is run just-in-time to ensure a secure execution environment. The eBPF kernel technology is relatively new and to take full advantage of it, you need at least at least Linux 5.13 (June 2021) or above. The eBPF mechanism takes the observability capability to a new level with more accurate readings of the internal kernel environment based on kernel data. More often than not, traditional security scanning approaches tend to fail Kubernetes’ highly distributed framework and automated workflows. By contrast, an eBPF driver, with well-written and maintainable codes, can collect network data on every single connection that one container makes with another or with the outside world. Once this data is coalesced with data captured from applications, services, and hosts, a more complete picture will emerge about how secure your Kubernetes workloads are. Thus, the eBPF approach imparts better understanding of the run-time behavior of your applications and the Kubernetes system at large. 

Sidekick: Expanded alerting features   

This is a simple daemon that remarkably extends Falco’s alerting functions. Falco sidekick exports alerts in parallel (“fan-out way”) to a large number of desired outputs. Such outputs include messaging apps, monitoring and security platforms, incident management tools, storage services, data stream management systems, email protocols, and webhooks. All you need to do is configure Falco sidekick to send those alerts. The sidekick comes with a simple web user interface for displaying outputs. And, with its port-forward settings, your remote users can also access the latest events from Falco. Falco metrics can be exposed to Prometheus open-source data monitoring/alerting tool using “falco-exporter”. So, make the best of it.  

These are some ways you can get the most value from Falco’s security features. The Falco project at CNCF is a continuously evolving one, and you will likely see new features being added to the security tool as we go along. Do write in with your comments and suggestion. Catch you soon. Till then, 

Falco Sidekick: Outputs