zabbix-server.service: control process exited, code=exited status=1

今天zabbix服务器mysql突然占用了大量内存,导致zabbix server服务器本身出现了内存占用大于95%的报警。占用大量内存,本着重启解决一切的原则,重启了该服务器。重启后发现zabbix server服务出现了无法启动的问题。

故障现象

一,检查服务是否启动


[root@shengbao.org ~]# ps aux|grep zabbix
zabbix 1063 0.0 0.0 78800 1300 ? S 14:00 0:00 /usr/sbin/zabbix_agentd -c /etc/zabbix/zabbix_agentd.conf
zabbix 1085 0.0 0.0 78800 1412 ? S 14:00 0:00 /usr/sbin/zabbix_agentd: collector [idle 1 sec]
zabbix 1086 0.0 0.0 78800 2136 ? S 14:00 0:00 /usr/sbin/zabbix_agentd: listener #1 [waiting for connection]
zabbix 1087 0.0 0.0 78800 2136 ? S 14:00 0:00 /usr/sbin/zabbix_agentd: listener #2 [waiting for connection]
zabbix 1088 0.0 0.0 78800 2404 ? S 14:00 0:00 /usr/sbin/zabbix_agentd: listener #3 [waiting for connection]
zabbix 1089 0.0 0.0 78800 2208 ? S 14:00 0:00 /usr/sbin/zabbix_agentd: active checks #1 [idle 1 sec]

二,重启zabbix_server服务


[root@shengbao.org ~]# systemctl restart zabbix_server

三,重启后检查该服务状态。


[root@shengbao.org ~]# systemctl status zabbix-server
● zabbix-server.service - Zabbix Server
Loaded: loaded (/usr/lib/systemd/system/zabbix-server.service; enabled; vendor preset: disabled)
Active: activating (auto-restart) (Result: exit-code) since Mon 2020-11-30 14:06:18 CST; 12s ago
Main PID: 11458 (code=exited, status=0/SUCCESS)
CGroup: /system.slice/zabbix-server.service
Nov 30 14:06:18 shengbao.org systemd[1]: zabbix-server.service: control process exited, code=exited status=1
Nov 30 14:06:18 shengbao.org systemd[1]: Unit zabbix-server.service entered failed state.
Nov 30 14:06:18 shengbao.org systemd[1]: zabbix-server.service failed.

四,出现失败后,检查/var/log/message检查系统日志


[root@shengbao.org ~]# cat /var/log/messages
Nov 30 14:01:47 shengbao.org systemd: zabbix-server.service holdoff time over, scheduling restart.
Nov 30 14:01:47 shengbao.org systemd: Starting Zabbix Server...
Nov 30 14:01:47 shengbao.org systemd: PID file /run/zabbix/zabbix_server.pid not readable (yet?) after start.
Nov 30 14:01:47 shengbao.org systemd: Started Zabbix Server.
Nov 30 14:01:50 shengbao.org kill: Usage:
Nov 30 14:01:50 shengbao.org kill: kill [options] <pid|name> [...]
Nov 30 14:01:50 shengbao.org kill: Options:
Nov 30 14:01:50 shengbao.org kill: -a, --all do not restrict the name-to-pid conversion to processes
Nov 30 14:01:50 shengbao.org kill: with the same uid as the present process
Nov 30 14:01:50 shengbao.org kill: -s, --signal <sig> send specified signal
Nov 30 14:01:50 shengbao.org kill: -q, --queue <sig> use sigqueue(2) rather than kill(2)
Nov 30 14:01:50 shengbao.org kill: -p, --pid print pids without signaling them
Nov 30 14:01:50 shengbao.org kill: -l, --list [=<signal>] list signal names, or convert one to a name
Nov 30 14:01:50 shengbao.org kill: -L, --table list signal names and numbers
Nov 30 14:01:50 shengbao.org kill: -h, --help display this help and exit
Nov 30 14:01:50 shengbao.org kill: -V, --version output version information and exit
Nov 30 14:01:50 shengbao.org kill: For more details see kill(1).
Nov 30 14:01:50 shengbao.org systemd: zabbix-server.service: control process exited, code=exited status=1
Nov 30 14:01:50 shengbao.org systemd: Unit zabbix-server.service entered failed state.
Nov 30 14:01:50 shengbao.org systemd: zabbix-server.service failed.

五,通过系统日志仅仅能查看到服务启动失败,造成失败的愿意并没有直观的显示出来。查看zabbix应用日志。


[root@shengbao.org ~]# cat /var/log/zabbix/zabbix-server.log
3520:20201130:144346.442 [Z3001] connection to database 'zabbix' failed: [1040] Too many connections
3520:20201130:144346.442 Cannot connect to the database. Exiting...
3521:20201130:144346.444 [Z3001] connection to database 'zabbix' failed: [1040] Too many connections
3521:20201130:144346.444 Cannot connect to the database. Exiting...

六,通过观察zabbix应用日志,发现报错是数据库Too many connections,修改数据库连接数解决该问题


[root@shengbao.org ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 1247
Server version: 5.5.65-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#查看当前最大连接数
MariaDB [(none)]> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 151 |
+-----------------+-------+
1 row in set (0.00 sec)
#设置全局最大连接数为1000.
MariaDB [(none)]> set global max_connections=1000;
Query OK, 0 rows affected (0.00 sec)
#检查是否设置成功。
MariaDB [(none)]> show variables like 'max_connections';
+-----------------+-------+
| Variable_name | Value |
+-----------------+-------+
| max_connections | 1000 |
+-----------------+-------+
1 row in set (0.00 sec)
MariaDB [(none)]> exit

七,重启服务故障解决


#systemctl restart zabbix_server
#systemctl status zabbix_server

为TA充电
共{{data.count}}人
人已赞赏
笔记

如何将操作系统从 CentOS 或 Oracle Linux 转换为 RHEL

2021-3-18 15:06:36

笔记

Device /dev/sdb excluded by a filter

2021-3-18 15:25:47

此心远送浑河岸,斟别酒,唱阳关,临别无语空长叹,酒已阑,曲未残,人初散,心长怀去后,杳鱼雁,对遥山,当时无计锁雕鞍,去后思量悔应晚,别时容易见时难!
0 条回复 A文章作者 M管理员
    暂无讨论,说说你的看法吧
个人中心
购物车
优惠劵
今日签到
有新私信 私信列表
搜索