Mystery Writes

We recently observed constant write activity on our development server, while watching dstat. With the help of iotop (iotop), we identified the Apache web server was the culprit.

But why would apache be doing so many writes? That's not normal behaviori (excluding the the logs). We then used auditd to log writes by apache:

sudo auditctl -a exit,always -S write -F uid=33

We also logged opens that were not O_RDONLY:

sudo auditctl -a exit,always -S open -F uid=33 -F a1'!=0'`

This resulted in the following rules:

sudo auditctl -l LIST_RULES: exit,always uid=33 (0x21) syscall=write \
    LIST_RULES: exit,always uid=33 (0x21) a1!=0 syscall=open

This allowed us to isolate a PHP module that was completely broken (using aureport).

Man Pages

• man 8 auditctl - a utility to assist controlling the kernel's audit system
• man 8 aureport - a tool that produces summary reports of audit daemon logs
• man 1 dstat - versatile tool for generating system resource statistics
• man 1 iotop - simple top-like I/O monitor
• man 2 open - open and possibly create a file or device