The case of i-node depletion

UAT server was down with 502. We logged into the server and saw nginx was dead. Upon trying to restart, it would say

nginx: [emerg] open() “log/access.log” failed (28: No space left on device)

Naturally you check if the disk space is full and you run following command

df -lh

shows free diskspace

As you can see /mnt1 has 29% diskspace. Then what is the problem?

Then we check if OS ran out of i-nodes

df -ih

shows inode depletion

There you see the problem. There are many small files on the disk. So the disk still has space but a new file can not be created because we ran out of i-nodes.

Following link should clarify what are i-nodes

https://www.redhat.com/sysadmin/inodes-linux-filesystem