Enhancing data security with file integrity monitoring

Leveraging Watchdog for compliance and risk mitigation

Organizations remain compliant so long as they are able to, among other required things, consistently demonstrate proper handling of customers’ personally identifiable information (PII). One facet of this strategy involves how the integrity of files containing PII within a user-facing application is managed and ensured. 

File integrity monitoring (FIM) is a strategy put in place to enable and define alerting mechanisms that notify security teams when the state of users’ data is at risk and a cyber attack may be occurring. 

File integrity on Upsun

Upsun (and Platform.sh, for that matter), is built on a strong separation between code and data. Our platform is built on Git, and that creates a relationship for your organization between what is committed to a repository and how that code results in provisioned infrastructure, active environments, and what is ultimately served to your users. 

We tie individual commits – the state of your code and configuration – to unique build images. While these images can be moved between environments, resulting in our central environment cloning capabilities, they cannot be modified once deployed at runtime in any way. 

This feature is made possible through the short list of rules placed upon your projects, which include:

• Changes to infrastructure must be made via commits
• Access, whether to projects, environments, applications, services or to the outside world must always be explicitly defined
• Data and code are different. Code, infrastructure, and configuration should be committed, while data never should be.
• Read-only access to the filesystem remains at runtime unless explicitly defined otherwise
• Data flows down to child environments

So long as your projects and team members abide by these tenets, things committed to Git (including infrastructure) should be protected. 

The attack surface is not eliminated completely, however, when we stop to consider your application itself and the things you control. It’s necessary, for instance, that you sanitize user-facing fields and validate inputs in your app to prevent SQL injections and other methods that would allow bad actors to execute arbitrary code that would put PII at risk.

Watchdog

Protection in this way is only one half of the puzzle of FIM, but even best practices cannot completely ensure that PII will never be exposed and/or altered. For the things we cannot expect, we need monitoring in place that triggers notifications when changes happen to files kept in mounts. 

Notifications allow us to not only respond to attacks that are underway, but also for us to fulfill certain reporting that may be required in order to remain compliant in the event of a breach. 

While such a notification system does not exist out of the box on Upsun, we can put together something using open-source tooling that monitors changes to our directories to build out such a notification strategy. 

Watchdog is an open-source Python library and shell tool for monitoring file system events. We can use Watchdog on top of our access control best practices to set up a notification system to watch for filesystem changes.

Below is a simple app, which in effect contains no code save what is relevant to configure the infrastructure to setup a Watchdog worker to listen to filesystem changes made to a single mount.

mkdir notify && cd notify && git init
upsun project:create

...

touch .upsun/config.yaml

Update the Upsun configuration for the worker:

applications:
  notify:
    # This example uses the Composable image for Python, thought simply
    # `type:'python:3.12'` would work just fine as well
    stack:
      - python@3.12
      - python312Packages.pip
    hooks:
      build: |
        set -e
        pip install -r requirements.txt
    # Some best practices for reference
    access:
      ssh: admin
    web:
      commands:
        start: sleep infinity
    # Our mount containing sensitive data.
    mounts:
      user_pii:
        source: storage

This configuration sets up a single Python 3.12 container with a single writable directory containing our sensitive user data (user_pii). Next, we install Watchdog through a virtual environment

python3 -m venv env
env/bin/activate
(env) pip install watchdog requests
(env) pip freeze > requirements.txt
(env) deactivate
echo "env" > .gitignore

Then produce the notifier script that will be run from our worker container:

# https://github.com/gorakhargosh/watchdog
# https://pypi.org/project/requests/
# https://philipkiely.com/code/python_watchdog.html

import os
import sys
import time
import datetime
import requests
import logging
from watchdog.observers import Observer
from watchdog.events import FileSystemEventHandler

class Watcher:

    def __init__(self, directory=".", handler=FileSystemEventHandler()):
        self.observer = Observer()
        self.handler = handler
        self.directory = directory

    def run(self):
        self.observer.schedule(
            self.handler, self.directory, recursive=True)
        self.observer.start()
        print("\nWatcher Running in {}/\n".format(self.directory))
        try:
            while True:
                time.sleep(1)
        except:
            self.observer.stop()
        self.observer.join()
        print("\nWatcher Terminated\n")

class MyHandler(FileSystemEventHandler):

    webhook_url = "https://webhook.site/3cbc9b0d-d2c1-4b95-8576-1e3070f704cd"

    if os.environ.get("WATCH_WEBHOOK_URL") is not None:
        webhook_url = os.environ["WATCH_WEBHOOK_URL"]

    logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s - %(message)s',
                    datefmt='%Y-%m-%d %H:%M:%S')

    def on_any_event(self, event):
        if not event.is_directory:
            event_data = {
                "src_path": event.src_path,
                "dest_path": event.dest_path,
                "event_type": event.event_type,
                "is_directory": event.is_directory,
                "is_synthetic": event.is_synthetic,
                "timestamp": datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S%z")
            }
            logging.info(f'{event.event_type.upper()!r}: {event_data!r}')
            resp = requests.post(self.webhook_url, json = event_data)
            alert_data = {
                "status_code": resp.status_code,
                "url": resp.url
            }
            logging.info(f'{"FORWARDED".upper()!r}: {alert_data!r}')

if __name__=="__main__":
    watchpath = sys.argv[1] if len(sys.argv) > 1 else '.'
    w = Watcher(watchpath, MyHandler())
    w.run()

In the above script notice that:

• A FileSystemEventHandler class use used to define MyHandler, piped into the primary Watcher when the script is run
• The Watcher monitors file system changes recursively a target provided that’s passed as a command line argument (sys.argv)
• MyHandler listens for file system changes on_an_event and forwards an event_data JSON object to either a default webhook URL (in this case to a temporary webhook URL at Webhook.site), otherwise to a user-provided URL via the WEBHOOK_URL environment variable.

With the script in place, we can run it in parallel to the man application via a Worker:

    workers:
      watchdog:
        commands:
          start: |
            python watch.py $PLATFORM_APP_DIR/user_pii

Before we push, we can include a few (admin only!) runtime operations to test our changes:

    operations:
      init-secret:
        role: admin
        commands:
          start: |
            echo "Oceanic 815 SYD -> LAX, 13C; Status: DEPARTED" > \
              $PLATFORM_APP_DIR/user_pii/austen_kate.txt
      read-secret:
        role: admin
        commands:
          start: cat $PLATFORM_APP_DIR/user_pii/austen_kate.txt
      rewrite-secret:
        role: admin
        commands:
          start: >
            echo "Oceanic 815 SYD -> LAX, 13C; Status: LOST" > \
              $PLATFORM_APP_DIR/user_pii/austen_kate.txt
      destroy-secret:
        role: admin
        commands:
          start: rm $PLATFORM_APP_DIR/user_pii/austen_kate.txt

Finally, we commit and push all of our changes:

git add . && git commit -m "Create a simple notifier app."
git push upsun main

Once the app has deployed, we can leverage our runtime operations to make sure that file system events that change sensitive data in a mount result in notifications to our desired webhook URL. 

1. Create a file

upsun operation:run init-secret -y

Indeed, the operation (creating the file austen_kate.txt) triggers four notifications (open, created, modified, closed):

2. Update file

upsun operation:run rewrite-secret -y

Once again, the action (updating the file austen_kate.txt) results in three events (open, modify, close) that we can be alerted to.

3. Read a file

upsun operation:run read-secret -y

We can likewise get notified of the two events (opened, close_no_write) that take place when a file containing sensitive data is read (cat austen_kate.txt),

4. Delete a file

upsun operation:run destroy-secret -y

Finally, removing the file completely from the filesystem can trigger its own notification for the single deleted filesystem event.

Above is a very simple example of how open-source tools can be used to construct a custom notification system for FIM on Upsun. You could update the WEBHOOK_URL environment variable to use a Slack webhook URL in your organization’s security notifications channel, but as is that channel would be flooded with false positives. 

Hopefully, however, you can see how these basic tools could be adapted to listen to:

• a subset of filesystem events
• only those filesystem events executed by agents other than known admins or your application itself
• details of what exactly has been changed to include in the forwarded message
• links to the most recent backups in storage to shorten time to recovery

Modify this example to your needs, and never underestimate what can be built on an application platform with so few rules baked into it. Best of luck!