周梦康 发表于 2021-05-06 200 次浏览
$dmesg -T|grep "Out of memory" -5
[Thu Apr 22 11:22:20 2021] [29882]     0 29882    28287       47      11        0             0 egrep
[Thu Apr 22 11:22:20 2021] [29883]     0 29883    28385       33       9        0             0 awk
[Thu Apr 22 11:22:20 2021] [29884]     0 29884    28385       33      12        0             0 awk
[Thu Apr 22 11:22:20 2021] [29885]     0 29885    11312       54      22        0             0 sshd
[Thu Apr 22 11:22:20 2021] [29887]     0 29887     3178       18       8        0             0 su
[Thu Apr 22 11:22:20 2021] Out of memory: Kill process 7312 (php) score 523 or sacrifice child
[Thu Apr 22 11:22:20 2021] Killed process 7312 (php) total-vm:4677456kB, anon-rss:4186860kB, file-rss:4kB
[Fri Apr 23 06:03:41 2021] logagent invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[Fri Apr 23 06:03:41 2021] logagent cpuset=staragent mems_allowed=0
[Fri Apr 23 06:03:41 2021] CPU: 0 PID: 30244 Comm: logagent Tainted: G           OE  ------------   3.10.0-327.ali2000.alios7.x86_64 #1
[Fri Apr 23 06:03:41 2021] Hardware name: Alibaba Cloud Alibaba Cloud ECS, BIOS rel-1.7.5-0-ge51488c-20140602_164612-nilsson.home.kraxel.org 04/01/2014
--
[Fri Apr 23 06:03:41 2021] [ 9769]     0  9769    35278      216      25        0             0 crond
[Fri Apr 23 06:03:41 2021] [ 9770]     0  9770    19315       88      38        0             0 sshd
[Fri Apr 23 06:03:41 2021] [ 9772]     0  9772    42349     9636      80        0             0 AliYunDun
[Fri Apr 23 06:03:41 2021] [ 9773]     0  9773    28287       37      10        0             0 sh
[Fri Apr 23 06:03:41 2021] [ 9776]     0  9776    28287       35      11        0             0 sh
[Fri Apr 23 06:03:42 2021] Out of memory: Kill process 17685 (php) score 528 or sacrifice child
[Fri Apr 23 06:03:42 2021] Killed process 17685 (php) total-vm:4714788kB, anon-rss:4223632kB, file-rss:0kB

虽然记录了 oom 的进程id,想要知道执行了什么脚本,比较兜底的方案是根据 coredump 的文件用gdb调试,也就可以知道具体执行的代码是什么,然后反推执行的脚本是什么。

记得开启 coredump ,gdb 的时候使用 .gdbinit + zbacktrace

另一方面,发现计划任务脚本都会将执行的脚本的完整命令和 pid 记录到/var/log/messages里面去

Apr 23 06:02:33 xxxx CROND[9647]: (admin) CMD (/usr/bin/flock -xn /home/admin/watch.lock -c 'cd /xxx/ && php bin/console xxx>> /home/admin/xxx.log 2>&1')

所以我们的业务代码也可以在执行脚本的入口处记录类似的日志,方便反查

Logger::addInfo("[" . getmypid() . "] CMD (" . implode(" ", $argv) . ")");

评论列表