UAT server was down with 502. We logged into the server and saw nginx was dead. Upon trying to restart, it would say
nginx: [emerg] open() “log/access.log” failed (28: No space left on device)
Naturally you check if the disk space is full and you run following command
As you can see /mnt1 has 29% diskspace. Then what is the problem?
Then we check if OS ran out of i-nodes
There you see the problem. There are many small files on the disk. So the disk still has space but a new file can not be created because we ran out of i-nodes.
Following link should clarify what are i-nodes