Server settings for a LXD production setup

To allow your LXD server to run a large number of instances, configure the following settings to avoid hitting server limits.

The Value column contains the suggested value for each parameter.

/etc/security/limits.conf

Note

For users of the snap, those limits are automatically raised.

/etc/sysctl.conf

Note

Reboot the server after changing any of these parameters.

fs.aio-max-nr

Maximum number of concurrent asynchronous I/O operations

Key: fs.aio-max-nr
Type: integer
Default: 65536
Suggested value: 524288

You might need to increase this limit further if you have a lot of workloads that use the AIO subsystem (for example, MySQL).

fs.inotify.max_queued_events

Upper limit on the number of events that can be queued

Key: fs.inotify.max_queued_events
Type: integer
Default: 16384
Suggested value: 1048576

This option specifies the maximum number of events that can be queued to the corresponding inotify instance (see inotify (opens in a new tab) for more information).

fs.inotify.max_user_instances

Upper limit on the number of inotify instances

Key: fs.inotify.max_user_instances
Type: integer
Default: 128
Suggested value: 1048576

This option specifies the maximum number of inotify instances that can be created per real user ID (see inotify (opens in a new tab) for more information).

fs.inotify.max_user_watches

Upper limit on the number of watches

Key: fs.inotify.max_user_watches
Type: integer
Default: 8192
Suggested value: 1048576

This option specifies the maximum number of watches that can be created per real user ID (see inotify (opens in a new tab) for more information).

kernel.dmesg_restrict

Whether to deny access to the messages in the kernel ring buffer

Key: kernel.dmesg_restrict
Type: integer
Default: 0
Suggested value: 1

Set this option to 1 to deny container access to the messages in the kernel ring buffer. Note that setting this value to 1 will also deny access to non-root users on the host system.

kernel.keys.maxbytes

Maximum size of the key ring that non-root users can use

Key: kernel.keys.maxbytes
Type: integer
Default: 20000
Suggested value: 2000000

kernel.keys.maxkeys

Maximum number of keys that a non-root user can use

Key: kernel.keys.maxkeys
Type: integer
Default: 200
Suggested value: 2000

Set this option to a value that is higher than the number of instances.

net.core.bpf_jit_limit

Limit on the size of eBPF JIT allocations

Key: net.core.bpf_jit_limit
Type: integer
Default: varies
Suggested value: 1000000000

On kernels < 5.15 that are compiled with CONFIG_BPF_JIT_ALWAYS_ON=y, this value might limit the amount of instances that can be created.

net.ipv4.neigh.default.gc_thresh3

Maximum number of entries in the IPv4 ARP table

Key: net.ipv4.neigh.default.gc_thresh3
Type: integer
Default: 1024
Suggested value: 8192

Increase this value if you plan to create over 1024 instances. Otherwise, you will get the error neighbour: ndisc_cache: neighbor table overflow! when the ARP table gets full and the instances cannot get a network configuration. See ip-sysctl (opens in a new tab) for more information.

net.ipv6.neigh.default.gc_thresh3

Maximum number of entries in IPv6 ARP table

Key: net.ipv6.neigh.default.gc_thresh3
Type: integer
Default: 1024
Suggested value: 8192

Increase this value if you plan to create over 1024 instances. Otherwise, you will get the error neighbour: ndisc_cache: neighbor table overflow! when the ARP table gets full and the instances cannot get a network configuration. See ip-sysctl (opens in a new tab) for more information.

vm.max_map_count

Maximum number of memory map areas a process may have

Key: vm.max_map_count
Type: integer
Default: 65530
Suggested value: 262144

Memory map areas are used as a side-effect of calling malloc, directly by mmap and mprotect, and also when loading shared libraries.

Related topics

How-to guides:

• How to benchmark performance
• How to increase the network bandwidth
• How to monitor metrics

Explanation:

• Performance tuning