请选择 进入手机版 | 继续访问电脑版

设为首页 收藏本站
思科服务支持社区 点击关注
思科服务支持社区

  
 找回密码
 立即注册

扫一扫,访问微社区

搜索
热搜: 邮件服务器
查看: 998|回复: 3

[原创分享] EEM 对付HIGH cpu

[复制链接]
发表于 2018-3-27 11:40:12 | 显示全部楼层 |阅读模式
本帖最后由 ywanxin 于 2018-3-27 11:42 编辑

在做CPU突高的时候经常会使用EEM,一般咱们是使用SNMP OID去get cpu的值做EEM的触发条件。但是由于OID在不同的IOS下可能会不一样,所以很肯能取不到CPU的值,进而EEM无法触发。

我在设备上发现了另一种EEM的配置方法。原理是通过配置让CPU在高过threshold的时候报一条log,EEM的触发条件是这个log的pattern。


下面贴出配置




process cpu threshold type total rising 85 interval 5  全局模式下配置,命令解释:
process cpu threshold type {total | process | interrupt} rising percentage interval seconds [falling percentage interval seconds]

!
event manager applet high-cpu
event syslog pattern "CPURISINGTHRESHOLD"      报出的log有这个pattern。
  action 0.1 syslog msg "EEM: HIGH CPU detected. Writing info to flash:eem-log.txt"
  action 0.2 cli command "enable"
  action 0.3 cli command "term exec prompt timestamp"
  action 0.4 cli command "term len 0"
  action 1.1 cli command "show process cpu sorted | append flash:eem-log.txt"
  action 1.2 cli command "show proc mem sorted | append flash:eem-log.txt"
  action 1.3 cli command "show mem alloc total | append flash:eem-log.txt"
  action 1.4 cli command "show buffers | append flash:eem-log.txt"
  action 1.5 cli command "show interfaces | append flash:eem-log.txt"
  action 1.6 cli command "show interfaces stat | append flash:eem-log.txt"
  action 1.7 cli command "show ip traffic | append flash:eem-log.txt"
  action 2.2 syslog msg "EEM: Self-removing applet from configuration..."
  action 2.3 cli command "configure terminal"
  action 2.4 cli command "no event manager applet high-cpu"
  action 2.5 cli command "end"
!
End

实验:
我是用rack上的RSRP2 (3945)测试的,使用长ping loopback接口IP把CPU冲高,配置如下:
process cpu threshold type total rising 25 interval 5  每5秒检测一次CPU, total cpu utilization 高于25%则报log,log如下:
*Mar  1 05:21:56.354: %SYS-1-CPURISINGTHRESHOLD: Threshold: Total CPU Utilization(Total/Intr): 30%/0%, Top 3 processes(Pid/Util):  132/14%, 5/10%, 185/0%
Log 中会看到占用最多CPU的3个进程ID和占用CPU多少

然后EEM触发
*Mar  1 05:21:56.363: %HA_EM-6-LOG: High_CPU: High CPU DETECTED. Please wait - logging Information to nvram:high_cpu.txt
看到文件已经抓下来了
RSRP2#dir nvram:
Directory of nvram:/

  505  -rw-        3976                    <no date>  startup-config
  506  ----        1912                    <no date>  private-config
    1  ----          35                    <no date>  persistent-data
    2  -rw-           0                    <no date>  ifIndex-table
    3  -rw-       32048                    <no date>  high_cpu.txt
   36  -rw-         557                    <no date>  IOS-Self-Sig#1.cer
   37  -rw-      336174                    <no date>  cpu_stats
**************************************************************************************
process cpu threshold type total rising 85 interval 5           <<<<<<< 5秒钟检测大于85
event manager applet high-cpu
event syslog pattern "CPURISINGTHRESHOLD"  
  action 0.1 syslog msg "EEM: HIGH CPU detected. Writing info to flash:eem-log.txt"
  action 0.2 cli command "enable"
  action 0.3 cli command "term exec prompt timestamp"
  action 0.4 cli command "term len 0"
  action 1.1 cli command "show process cpu sorted | append flash:eem-log.txt"
  action 1.2 cli command "show proc mem sorted | append flash:eem-log.txt"
  action 1.3 cli command "show mem alloc total | append flash:eem-log.txt"
  action 1.4 cli command "show buffers | append flash:eem-log.txt"
  action 1.5 cli command "show interfaces | append flash:eem-log.txt"
  action 1.6 cli command "show interfaces stat | append flash:eem-log.txt"
  action 1.7 cli command "show ip traffic | append flash:eem-log.txt"
  action 2.2 syslog msg "EEM: Self-removing applet from configuration..."
  action 2.3 cli command "configure terminal"
  action 2.4 cli command "no event manager applet high-cpu"
  action 2.5 cli command "end"
!
End
****************************************************************************************





如果出现一下提示 可以用""""event manager session cli username <user>""""
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$
more flash:eem-log.txt
Command authorization failed.
Command authorization failed.
Command authorization failed.
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$





并且这个可以跟EEM 连用 直接抓包下来

*******************************************************************************************
配置:


monitor capture buffer cc
monitor capture point ip cef cc gX/X both  对应接口
monitor capture point associate cc cc

或者

monitor capture buffer cisco-buf
monitor capture buffer cisco-buf size 10240
monitor capture buffer cisco-buf max-size 1500
monitor capture point ip process-switched cisco-point both
monitor capture point associate cisco-point cisco-buf


process cpu threshold type total rising 85 interval 5       每5秒检测cpu超过85
event manager applet high-cpu
event syslog pattern "CPURISINGTHRESHOLD"  
  action 0.1 syslog msg "EEM: HIGH CPU detected. Writing info to flash:eem-log.txt"
  action 0.2 cli command "enable"
  action 0.3 cli command "term exec prompt timestamp"
  action 0.4 cli command "term len 0"
  action 0.8 cli command "monitor capture point start cc"
  action 0.9 cli command "show process cpu sorted | append flash:eem-log.txt"
  action 1.0 cli command "show proc mem sorted | append flash:eem-log.txt"
  action 1.1 cli command "show mem alloc total | append flash:eem-log.txt"
  action 1.2 cli command "show buffers | append flash:eem-log.txt"
  action 1.3 cli command "show interfaces | append flash:eem-log.txt"
  action 1.4 cli command "show interfaces stat | append flash:eem-log.txt"
  action 1.5 cli command "show ip traffic | append flash:eem-log.txt"
  action 1.6 cli command "show process cpu history | append flash:eem-log.txt"
  action 1.7  wait 15
  action 1.8 cli command "monitor capture point stop all"
  action 1.9 syslog msg "EEM: Self-removing applet from configuration..."
  action 2.0 cli command "configure terminal"
  action 2.1 cli command "no event manager applet high_cpu"
  action 2.2 cli command "end"
!


  • 1
  • 2
  • 3
  • 4
  • 5
  • 1
  • 2
  • 3
  • 4
  • 5
平均得分5 (4 评价)
发表于 2018-3-27 11:44:39 | 显示全部楼层
厉害了。。不过监控我一般都是用的外部的服务器获取snmp,还没在设备上做过
  • 1
  • 2
  • 3
  • 4
  • 5
  • 1
  • 2
  • 3
  • 4
  • 5
平均得分0 (0 评价)
发表于 2018-3-27 14:44:56 | 显示全部楼层
好好学习一下此类型故障的解决方法
  • 1
  • 2
  • 3
  • 4
  • 5
  • 1
  • 2
  • 3
  • 4
  • 5
平均得分0 (0 评价)
发表于 2018-5-5 17:10:41 | 显示全部楼层
我学会了新方法。
  • 1
  • 2
  • 3
  • 4
  • 5
  • 1
  • 2
  • 3
  • 4
  • 5
平均得分0 (0 评价)
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

Archiver | 思科服务支持社区  

GMT+8, 2018-6-21 12:25 , Processed in 0.084287 second(s), 37 queries .

京ICP备09041801号-187

版权所有 :copyright:1992-2019 思科系统  重要声明 | 保密声明 | 隐私权政策 | 商标 |

快速回复 返回顶部 返回列表