Cfxtrjtrk

I use an application that consumes inotify watches. I've already set

fs.inotify.max_user_watches=32768

in /etc/sysctl.conf but last night the application stopped indexing unless I ran it manually, which leads me to suspect I am out of watches.

Since I don't know what the trade off is when I increase this number (does it consume more RAM?), I don't know if I should just increase this number, so I'd like to know if there's a way I can tell if it's using all these watches and what the tradeoffs might be for increasing it.

How do you know if you are out of watches? tail will tell!

• Start tail with the -f (follow) option on any old file, e.g. tail -f /var/log/dmesg:
  
  
  
  
  • If all is well, it will show the last 10 lines and pause; abort with Ctrl-C
  
  
  • If you are out of watches, it will fail with this somewhat cryptic error:
    
    

    tail: cannot watch '/var/log/dmsg': No space left on device

    
    
  
  
  
  

For the curious: Why is tail a "telltail"?

• Actually, any well-written app should have the courtesy of telling you, since the inotify API/calls clearly tells them what the deal is.


• Try strace tail -f ... instead, and when it succeeds, it ends with:
  
  

  
  
  inotify_add_watch(4, "/var/log/dmesg", IN_MODIFY...) = 1
  
  

  
  


• but if it fails, i.e. you are out of watches, it'll say:
  
  

  
  
  inotify_add_watch(4, "/var/log/dmesg", IN_MODIFY..)
  
  = -1 ENOSPC (No space left on device)
  
  

  
  

Can you increase the watches? By how much? Any tradeoffs?

Short answer: Sure, no sweat. Go to straight to a half-million (524288) if you want...the additional memory used should be negligible on a modern system with 4GB+ of memory.

• Each used inotify watch takes up 540 bytes (32-bit system), or 1 kB (double - on 64-bit) [sources: 1, 2]


• This comes out of kernel memory, which is unswappable.


• 
  

  So, assuming you set the max at 524288, and all were used (improbable), you'd be using approx. 256MB/512MB of 32-bit/64-bit kernel memory

  
  
  
  
  
  • Note that your application will also use additional memory to keep track of the inotify handles, file/directory paths, etc. -- how much depends on its design.
  
  
  
  



• What's the max value? I guess none, in theory, as long as you have enough RAM. In practice, 524288 has been officially recommended by apps, and people have been setting it to 2 million, with the accompanying memory usage, of course.

I don't know if I should just increase this number

The easy way of checking if you reached your max_user_watches value is, with your user, to use inotifywatch, from the package inotify-tools, and check if you can still collect information from a file.

For example inotifywatch -v /home/bruno/.profile for me returns:

Establishing watches...

Total of 1 watches.

Finished establishing watches, now collecting statistics.

So inotify has no issues creating a new watch, no issues here.

If you have reached your maximum limit in inotify watches it will return something like

Failed to watch /home/bruno/.profile; upper limit on inotify watches reached!

If you see something like this then you have reached the limit and will need to increase the allowed watches limit.

Does it consume more RAM?

Yes, it does. But according to this old article the amount it consumes is minimal compared with other aspects of a running desktop.

--MEMORY USAGE--

The inotify data structures are light weight:

inotify watch is 40 bytes
inotify device is 68 bytes
inotify event is 272 bytes

So assuming a device has 8192 watches, the structures are only going
to consume 320KB of memory. With a maximum number of 8 devices allowed
to exist at a time, this is still only 2.5 MB

Each device can also have 256 events queued at a time, which sums to
68KB per device. And only .5 MB if all devices are opened and have
a full event queue.

So approximately 3 MB of memory are used in the rare case of
everything open and full.

Each inotify watch pins the inode of a directory/file in memory,
the size of an inode is different per file system but lets assume
that it is 512 byes.

So assuming the maximum number of global watches are active, this would
pin down 32 MB of inodes in the inode cache. Again not a problem
on a modern system.

I am of course assuming things did not change a lot since the article was written but looking at the numbers I would not worry and increasing the limit will not increase RAM consumption by much.

Related posts about inotify

• Dropbox error - 'echo 100000 | sudo tee / proc/sys/fs/inotify/max_user_watches'




• kernel-inotify-watch-limit-reached