cgroup version jdk version k8s

bug info:

in centos, linux 3.10, cgroup:v1, service works well.

in ubuntu 22.04 LTS, linux 5, systemd:v2, service is always crash on boot, or running some minutes then killed by OOM.

deploy.yaml文件相关内容:

specify limits:memory 1000M in yaml, killed by OOM.

die into pod, use jmap -heap pid, the max heap size is more bigger than 2000M.

so, the memory limit does not in effect.

what is cgroup

cgroup is: control group(控制组)，专为子系统分配资源(cpu, memory, disk, net...)

cgroups(7) - Linux manual page

cgroup v1/v2 - linux version

cgroups(7) - Linux manual page

cgroup v1 implementation was in linux 2.6.24.

cgroup v2 starting in linux 3.10 and official with linux 4.5

cgroup in docker

Runtime metrics | Docker Docs

cgroup in k8s

About cgroup v2 | Kubernetes

Configuring a cgroup driver | Kubernetes

Container Runtimes | Kubernetes

k8s - ??? - containerd

k8s - docker shim - docker - containerd

k8s - ctr/crictl containerd

【K8S】ctr和crictl的区别_u010157986的博客-CSDN博客

https://www.cnblogs.com/aozhejin/p/16193359.html

k8s都开始抛弃dockerd了，ctr需要搞起了_ctr pause_柳清风09的博客-CSDN博客

UseContainerSupport最小支持版本

当时只有cgroup:v1

最小支持版本, oraclejdk-8u191/openjdk-8u191

Java™ SE Development Kit 8, Update 191 Release Notes

Bug Database

验证方法：

https://medium.com/ci-cd-devops/effect-on-container-mem-usage-with-usecontainersupport-f0071a6a577e

docker run -m 1gb openjdk:8u131 java -XshowSettings:vm -version

docker run -m 1gb openjdk:8u191 java -XshowSettings:vm -version

cgroup:v2/systemd:v2

openjdk-8u372， (未发布),   hub.docker上最新是8u342

openjdk-11.0.16，(已发布),   hub.docker上最新是11.0.16

Bug Database

[JDK-8230305] Cgroups v2: Container awareness - Java Bug System

how max heap size calculate?

jmap -heap pid | grep heap #jdk11相关命令稍有不同

free -h

max heap size = available * 25%

+UseContainerSupport to the Rescue

how container know memory info of pod?

服务启动后，kubectl exec进入pod，下图2个位置文件和 xx.yaml文件里配置的limits:memory应该是一样的才对。

cgroup:v1，对应 /sys/fs/cgroup/memory/memory.limit_in_bytes

cgroup:v2，对应 /sys/fs/cgroup/memory.max

Support cgroups memory limit detection for cgroups V2 · Issue #6894 · dask/distributed · GitHub

other links：

+UseContainerSupport to the Rescue

https://medium.com/ci-cd-devops/effect-on-container-mem-usage-with-usecontainersupport-f0071a6a577e

运行Debian11的Linux服务器中 docker run -m 命令 不生效 不能能限制住内存使用 cgroups_memory cgroup out of memory-CSDN博客

https://hostadvice.com/how-to/how-to-limit-a-docker-containers-resources-on-ubuntu-18-04/

How to Limit Memory and CPU for Docker Containers - Serverlab

修改 docker cgroup 版本的方法 （changing cgroup version）_shida_csdn的博客-CSDN博客

Runtime metrics | Docker Docs

Docker: Placing limits on container memory using cgroups | Fabian Lee : Software Engineer

Support cgroups memory limit detection for cgroups V2 · Issue #6894 · dask/distributed · GitHub

/sys/fs/cgroup/memory/memory.limit_in_bytes is missing in the container in version > 4.2.0 · Issue #6118 · docker/for-mac · GitHub

[JDK-8230305] Cgroups v2: Container awareness - Java Bug System

https://medium.com/ci-cd-devops/effect-on-container-mem-usage-with-usecontainersupport-f0071a6a577e

jvm参数配置看这里（dockerfile方式和k8s的yaml方式）：JVM参数

the command line:

kubectl edit 编辑yaml，设置limits:memory

kubectl exec 进入pod容器

top 查看内存

free -h 查看内存

top -Hp pid 查看cpu

jmap -heap pid 查看jvm 内存

jinfo -flags pid 查看java启动参数

jstat -gc pid 2000 查看gc状态

java -version 查看jdk版本

java -XshowSettings:vm -XX:+PrintFlagsFinal -version | grep -Ei "maxheapsize|maxram|UseContainerSupport" 查看内存和UseContainerSupport是否生效

conclusion:

方案1： 保持linux kernel使用cgroup:v2或systemd:v2：

linux 4.5+(含)

使用opendjk-8u372+(含)， 未发布，当前8u342， 统计时间为2022-12-07。

使用openjdk-11.0.16+(含)，已发布，当前11.0.16，统计时间为2022-12-07。

方案2： 修改linux kernel后，使用cgroup:v1：

linux 2.6.24(含)-4.5(不含)

使用openjdk-8u191(含)+，已发布，统计时间为2022-12-07。

方案3：或直接使用openjdk:11.0.16，兼容cgroup:v1和cgroup:v2