I. Exploring the need to limit the use of system resources
For a script, the most basic restrictions are to limit the number of process instances to a single process instance to ensure that there are no more than one, to check the amount of system resources left before running the main logic of the program to ensure that you are not the last straw that breaks the system's back, and to set a timeout for the program to run to ensure that the process instances don't run endlessly.
Further, in hosts where services with high availability requirements are deployed, we also need to be concerned about the issue of system resources consumed during the running of scripts, and we need a way to limit the amount of system resources such as cpu, memory, and io used by scripts to prevent them from impacting these critical services.
II. Introduction to cgroups
2.1 Introduction to the back
cgroups, short for Control groups, is a feature provided by the Linux kernel that restricts processes from using and seeding system resources.
The earliest cgroups came from Google engineers Paul Menage and Rohit Seth, who developed the "Processing Container" feature, and their work was integrated into Linux version 2.6.24 in 2008, which is also known as cgroups V1.
Since then various features have been added to cgroups, enriching the functionality of cgroups but also complicating and making it inconsistent. For this reason, Tejun Heo has rewritten cgroups as cgroupsV2, which was integrated in Linux 3.10 and released in Linux 4.5.
V2 is intended to be a replacement for V1, but both can exist at the same time for compatibility. Nothing in the latter description specifically refers to cgroupsV2.
2.2 Introduction to use
In presentation, cgroups are a filesystem, usually mounted in the /sys/fs/cgroup directory, and can be viewed directly using df -h
Each subfolder of this directory represents a resource, and the restriction of that resource is configured in the corresponding folder.
For example, limiting cpu is configured in the cpu,cpuacct folder, limiting io is configured in the blkio folder, and limiting memory is configured in the memory folder.
Each file under the folder is an indicator for that resource, configure each indicator, and finally write the pid of the target process you want to restrict in tasks, and the linux kernel will restrict this process according to the configuration.
However, it is a common need that we may want to have different restrictions for different processes. So instead of modifying the files under each resource folder directly, we usually create a separate folder for our target process under the resource folder, but the system will automatically generate the configuration files for each resource metrics under the folder when you create the folder using mkdir. But when you create a folder with mkdir, the system will automatically generate the configuration files for each resource metrics under the folder.
2.3 Configuration Realization Example
Example 1, restricting processes with a pid of 1234 to use at most 90% of the arithmetic power of a single core.
pid_num=1234
cgroup_base_dir="/sys/fs/cgroup"
resource_dir="cpu,cpuacct"
self_define_dir="test_process"
target_path="${cgroup_base_dir}/${resource_dir}/${self_define_dir}"
# Create a separate folder under the cpu configuration folder first
mkdir ${target_path}
# Set the time period length in microseconds. Here it is set to 1000000 microseconds, i.e. 1 second
echo 1000000 > ${target_path}/cpu.cfs_period_us
# Set the maximum amount of time, in microseconds, that can be used during the time period. Here it is set to 900000, which is 0.9 seconds.
# When cpu.cfs_quota_us is larger than cpu.cfs_period_us it means multiple cpu cores can be used
echo 900000 > ${target_path}/cpu.cfs_quota_us
# Limitations on processes
echo ${pid_num} >> ${target_path}/tasksExample 2: Limit processes with a pid of 1234 to read and write to disks with device number 252:0 at a maximum speed of 5M.
pid_num=1234
# Here the device number is given, you can use lsblk to see the device number of your own disk
device_num=252:0
limit_read_rate=5242880
limit_write_rate=5242880
cgroup_base_dir="/sys/fs/cgroup"
resource_dir="blkio"
self_define_dir="test_process"
target_path="${cgroup_base_dir}/${resource_dir}/${self_define_dir}"
# Create a separate folder under the cpu configuration folder first
mkdir ${target_path}
# Limit read speed in B per second.
echo "${device_num} ${limit_read_rate}" > ${target_path}/.read_bps_device
# Limit the write speed in B per second.
echo "${device_num} ${limit_write_rate}" > ${target_path}/.write_bps_device
# Limitations on processes
echo ${pid_num} >> ${target_path}/tasksExample 3, Limit processes with a pid of 1234 to a maximum of 1 Gigabyte of memory
pid_num=1234
limit_memory_byte=1024000000
cgroup_base_dir="/sys/fs/cgroup"
resource_dir="memory"
self_define_dir="test_process"
target_path="${cgroup_base_dir}/${resource_dir}/${self_define_dir}"
# Create a separate folder under the cpu configuration folder first
mkdir ${target_path}
# Limit the size of memory usage in B.
echo ${limit_memory_byte} > ${target_path}/memory.limit_in_bytes
# Limitations on processes
echo ${pid_num} >> ${target_path}/tasks2.4 Some notes on configuration
All configuration entries, if there is more than one, are separated by newlines; the same applies to process pids, including those of tasks.
The configuration of cgroups has some active troubleshooting, if the configuration is wrong it will refuse to write. For example, in the configuration above for limiting disk read/write rates, if you don't have a device with a device number of 252:0 then configuration writes will fail.
The cgroups process limitation has some adaptive features. For example, if you limit a process, then its children will be automatically added to the tasks, and if a process is terminated, then its pid will be automatically moved out of the tasks.
Third, cgroupspy installation and use
If you want to be free of constraints, you can implement your own resource constraints as described in the second section, but if you want simplicity and robustness, you can use libraries written by others; cgroupspy is a python implementation of cgroups configuration.
Project home page:/cloudsigma/cgroupspy
3.1 Installation
Just pip install it directly:
pip install cgroupspy
3.2 Example of use (as an example of limiting memory)
from cgroupspy import trees
pid_num = 1234
resource_item = "memory"
group_name = "test_process"
limit_memory_byte = 1024000000
# Instantiate a resource tree
t = ()
# Get the memory configuration object
memory_limit_obj = t .get_node_by_path("/{0}/".format(resource_item))
# Create a test_process group (in effect, a folder)
test_process_group = memory_limit_obj.create_cgroup(group_name)
# Limit memory usage. Fewer memory dropouts relative to the actual file
test_process_group.controller.limit_in_bytes = limit_memory_byte
# Restricted processes
test_process_group. = pid_numReference:
/wiki/Cgroups
/linux/man-pages/man7/cgroups.
to this article on the use of Python3+cgroupspy installation tutorials are introduced to this article, more related Python3 cgroupspy installation content, please search for my previous articles or continue to browse the following related articles I hope you will support me in the future more!