Showing posts with label monitoring. Show all posts
Showing posts with label monitoring. Show all posts

29.11.10

Lm-sensors in net-snmp

Readout of integrated motherboard sensors was already a topic here, for VIA Epia boards and for WD Mybook World Edition as well. Such data is good to have visualized, so plotting it using MRTG was discussed too. This tool and its fundamental, net-snmp was designed to visualize network traffic data, so for sensor readings, an external script had to be constructed, which was quite inconvenient.
There is however a better way, briefly mentioned by lm-sensors.org, for which net-snmp has to be installed from source (download here, install instructions here) and has to be configured with the following options:
~$: cd net-snmp
~$: ./configure --with-mib-modules="ucd-snmp/lmsensorsMib" --with-ldflags="-lsensors" 
~$: make
~$: sudo make install

If installation was successful, lm-sensor readouts can be accessed via snmp, the same way as if they were polled by the command-line sensors tool:

~#: sensors

coretemp-isa-0000
Adapter: ISA adapter
Core 0:      +33.0 °C  (high = +78.0 °C, crit = +100.0 °C) 

coretemp-isa-0001
Adapter: ISA adapter
Core 1:      +32.0 °C  (high = +78.0 °C, crit = +100.0 °C) 

~#: snmpwalk -v 2c -c public localhost Sensors

LM-SENSORS-MIB::lmTempSensorsIndex.1 = INTEGER: 1
LM-SENSORS-MIB::lmTempSensorsIndex.2 = INTEGER: 2
LM-SENSORS-MIB::lmTempSensorsDevice.1 = STRING: Core 0
LM-SENSORS-MIB::lmTempSensorsDevice.2 = STRING: Core 1
LM-SENSORS-MIB::lmTempSensorsValue.1 = Gauge32: 33000
LM-SENSORS-MIB::lmTempSensorsValue.2 = Gauge32: 32000

Sadly, only CPU core temperatures can be read out for ICH8/ICH9 chipsets at the moment, still it is now way easier to integrate lm-sensor readings to net-snmp based logging and plotting tools such as MRTG.

12.6.10

VIA Epia Homeserver Project - On-board Sensor Readings

The scope of this post is to make use of the onboard voltage and temperature sensors that can be useful to avoid dangerous operating conditions such as overheating.
In order to make them work, compiling a recent kernel was necessary so I first obtained the latest version from kernel.org (2.6.34 as of today).
On EN12000EG, the following hardware supports built-in sensors:
  • Temperature sensor in VIA Eden CPU;
  • Winbond W83697 SuperIO chip featuring power supply voltage and temperature readings. Link to datasheet is here.
The following kernel options have to be set (followed by the actual name of the option in the .config file):
  • ISA support: CONFIG_ISA=y
  • hardware monitoring support: CONFIG_HWMON=y
  • VID support: CONFIG_HWMON_VID=y
  • VIA CPU temp sensor support: CONFIG_SENSORS_VIA_CPUTEMP=y
  • Winbond driver: CONFIG_SENSORS_W83627HF=y
  • the ACPI drivers: CONFIG_THERMAL=y and CONFIG_THERMAL_HWMON=y
Note: these can be compiled as modules as well, in which case OPTION=m will be in the at the corresponding line.
Next, obtain latest lm-sensors (as of today, it is 3.1.2). Manual for compiling and installing is together with the source, it worked without any problem. Then the sensors can be detected with the sensors-detect tool.
If the options above were compiled as modules, then the following two lines should be add to /etc/modules.conf:
w83627hf
via_cputemp
which will cause the kernel to load these modules on startup.
We are almost ready, however because of recent changes in the kernel architecture (specific version is 2.6.31), some sensors cannot be read because the ACPI code claims the corresponding resources. More information and discussion on the topic is here. The point is, that if we try to load the module for the Winbond chip:
# modprobe w83627hf
we get a "FATAL: [...] Device or resource busy" error message.
The solution is considered to be dangerous, however, I did not notice any problem (yet :)): The option "acpi_enforce_resources=lax" has to be add to the kernel command line upon boot. For grub, this means that the default item in /boot/grub/menu.lst will look something like:
title        Debian GNU/Linux, kernel 2.6.34 for VIA C7 and lm-sensors root        (hd0,0) kernel       /boot/vmlinuz-2.6.34 root=/dev/sdb1  ro acpi_enforce_resources=lax
Now the modules should be able to load, and raw sensor values should read using
$ sensors
Yet, the actual voltage readings need to be calculated using the scheme provided in /etc/sensors3.conf and running sensors -s (as root!) to get the proper readings afterwards.
It is interesting, that for older versions of lm-sensors, the Winbond chip was included in the config file, but it is removed from the latest one. No problem, one can just copy it back:

  • chip "w83697hf-*"

  • # no in1 on this chip.
  •     label in0 "VCore"    
  •     label in2 "+3.3V"
  •     label in3 "+5V"
  •     label in4 "+12V"
  •     label in5 "-12V"
  •     label in6 "-5V"
  •     label in7 "V5SB"
  •     label in8 "VBat"


  •     compute in3 ((6.8/10)+1)*@ ,  @/((6.8/10)+1)
  •     compute in4 ((28/10)+1)*@  ,  @/((28/10)+1)
  •     compute in5 (5.14 * @) - 14.91  ,  (@ + 14.91) / 5.14
  •     compute in6 (3.14 * @) -  7.71  ,  (@ +  7.71) / 3.14
  •     compute in7 ((6.8/10)+1)*@ ,  @/((6.8/10)+1) 


  • # 697HF does not have VID inputs so you MUST set your core
  • # voltage limits below. Currently set for 1.8V core.
  • #               vvv
  •    set in0_min 0.6
  •    set in0_max 1.2
  •    set in2_min 3.3 * 0.95
  •    set in2_max 3.3 * 1.05
  •    set in3_min 5.0 * 0.95
  •    set in3_max 5.0 * 1.05
  •    set in4_min 12 * 0.90
  •    set in4_max 12 * 1.10
  •    set in5_max -12 * 0.90
  •    set in5_min -12 * 1.10
  •    set in6_max -5 * 0.95
  •    set in6_min -5 * 1.05
  •    set in7_min 5 * 0.90
  •    set in7_max 5 * 1.10
  •    set in8_min 3.0 * 0.80
  •    set in8_max 3.0 * 1.20

  • #   ignore temp2


  • The rest of the sample config file can even be removed, it makes no use. I also modified the limits of the line in7 as the standby 5V (5VSB) reading is known to scatter more than the others, so +/- 10% is more appropriate.
    Note: I am aware that now the lm-sensor policy is that platform-specific configuration is supposed to be in /etc/sensors.d, yet I believe that this way is easier. Moreover, I am not aware of any database that could be used for this purpose, and I have no idea what triggered this change of functionality.
    Another issue is that I cannot set the limits nor the type of temperature sensors, so lines in the config file like:
    set temp1 1 
    set set temp1_over 80
    set temp1_hyst 75
    will result in something like:
    # sensors -s
    Error: File /etc/sensors3.conf, line 95: Unknown feature name w83697hf-isa-0290: No such subfeature known
    Not only the limits but the sensor type cannot be set as well, therefore the second temperature sensor (temp2) always reads false values below even room temperature. I am not aware of the resolution of this issue, but I will update the post as soon as I find the proper solution.
    Omitting the invalid lines, now we get the actual readings:
    # sensors -s
    $ sensors
    acpitz-virtual-0
    Adapter: Virtual device
    temp1:       +15.5°C  (crit = +120.0°C)                 
  • w83697hf-isa-0290
    Adapter: ISA adapter
    VCore:       +0.83 V  (min =  +0.59 V, max =  +1.20 V)  
    +3.3V:       +3.31 V  (min =  +3.14 V, max =  +3.47 V)  
    +5V:         +5.11 V  (min =  +4.76 V, max =  +5.24 V)  
    +12V:       +12.34 V  (min = +10.82 V, max = +13.19 V)  
    -12V:       -14.83 V  (min = -13.18 V, max = -10.80 V)   ALARM
    -5V:         -7.66 V  (min =  -5.25 V, max =  -4.75 V)   ALARM
    V5SB:        +5.46 V  (min =  +4.49 V, max =  +5.51 V)
    VBat:        +3.22 V  (min =  +2.40 V, max =  +3.60 V)  
    fan1:          0 RPM  (min =   -1 RPM, div = 4)  ALARM
    fan2:          0 RPM  (min = 2109 RPM, div = 4)  ALARM
    temp1:       +55.0°C  (high = +12.0°C, hyst =  +2.0°C)  ALARM  sensor = transistor
    temp2:       +15.5°C  (high = +120.0°C, hyst = +115.0°C)  sensor = diode
    beep_enable:enabled
    via_cputemp-isa-0000
    Adapter: ISA adapter
    Core 0:      +50.0°C                                   
  • Note that temp2 and the ACPI virtual device has the same reading, both wrong. For temp2 the quick and dirty solution is to simply ignore it by uncommenting the following line in sensors3.conf:
    ignore temp2
    The rest however works now, so the only thing to do is to make sure that sensors -s is run on every boot of the system. A simple shell script shall do the trick:
    #! /bin/sh
    SENSORS=/usr/local/bin/sensors #just initializing lm-sensors $SENSORS --version $SENSORS -s $SENSORS
    Save it as Sensors.sh in /etc/init.d then chmod it to be executable and link it /etc/rc2.d/S40sensors:
    # chmod a+x Sensors.sh
    # ln -s /etc/init.d/Sensors.sh /etc/rc2.d/S40sensors
    Still, there are remaining issues with the lm-sensors that need to be resolved. For instance, the temp2 and acpitz-virtual-0 readings are both wrong, and you cannot even get rid of the second one! At least there is not option to disable it through /etc/sensors3.conf most likely, the corresponding kernel module needs to be unloaded. Also, the spurious readings of the -12V, -5V lines might not be the real things as most likely they not even used by the motherboard. It also worth mentioning that the new kernel policy of claiming all ACPI related resources resulting in the device drivers not to work anymore might need to be reconsidered, as this is a definite loss of functionality as well, resulting in additional hacking being necessary.

    19.2.10

    WD MyBook Hacks - MRTG

    This post is about using the versatile MRTG tool visualize usage data and health status of the storage device.

    Prerequisites
    • Web server to see the plots over http. For this the built-in mini_httpd will do, but lighthttpd is good as well if installed already. I however advice to use the first if there is no other reason for using the second one so that the configuration of the server does not have to be altered.
    • Ipkg packages: cron, net-snmp and mrtg (also installs libgd and perl). The first is for periodic execution of the data acquisition, while net-snmp will collect the data which will be visualized by mrtg. Luckily again, these packages work out of the box, no compilation is required. For the usage of ipkg see my post on the basic setup.
    • For cron and net-snmp to start on device startup, copy the contents of /opt/etc/init.d to /etc/init.d.
    Setup MRTG

    This can be done by using the built-in tool cfgmaker, for which the usage reference can be found here. The basic setup includes measuring the throughput of the ethernet interface:

    #~: cfgmaker --global 'WorkDir: /proto/SxM_webui/mrtg' --global 'Options[_]: bits,growright' --output /opt/etc/mrtg.cfg public@localhost 

    which sets up WorkDir where the graphic output and html files will be stored to be in the working directory of the html server. The config file will be stored at /opt/etc/mrtg.cfg.
    I realized however that usage of the cfgmaker tool is not displayed in case of an input error or issuing --help. It seems that the /opt/lib/mrtg2/Pod/Usage.pm file shipped with the package is broken and has to be replaced with the one found here.
    This basic config can be tested by issuing:
    #~: mrtg /opt/etc/mrtg.cfg
    at least twice and checking the result in the browser at http://IP_of_storage/mrtg/localhost_2.html (localhost_1 would be the loopback interface which is not plotted). The data can be collected automatically by adding the following line to /opt/etc/crontab:
    */5 * * * * root /opt/bin/mrtg /opt/etc/mrtg.cfg
    and restarting the cron daemon:
    #~: /etc/init.d/S10cron
    This tells cron to run mrtg every 5 minutes. According to the mrtg manual it is not recommended to make it more frequent, and this value is usually good choice anyway.
    Eventually, graphs will be filled with data like this:

    This is good, but we can go even further: CPU load, memory and hard disk usage and temperature data can be plotted as well. The first three data values can be collected by net-snmp, while for the last one we need to modify the script that I have posted already:

    #!/bin/sh
    Y1=`cat /sys/module/thermAndFan/parameters/current_temp `
    C1='display(0);config("tilde","off");3865/(ln('
    C2=')+9.1411)-273.16'
    /usr/sbin/smartctl -a -d ata /dev/sda | awk '/Temperature/ {print $10}'
    /opt/bin/calc $C1$Y1$C2 | awk 'NR==3 {print $1}'
    echo 0
    echo Temperature Data on Storage
    exit 0

    The output of this script will consist of four lines that are needed by mrtg:
    1.  "Input bytes" data: Here it will be the temperature of the hard disk;
    2. "Output bytes" data: Here it will be the temperature of the system thermometer;
    3. Uptime: we don't need this so set it to zero.
    4. Text output: the name of the plots.
     I saved this script to /opt/etc/mrtg/tempscript1.sh which we will need in the revised mrtg.cfg:

    # Created by
    # /opt/bin/cfgmaker --global "WorkDir: /proto/SxM_webui/mrtg" --global "Options[_]: bits,growright" --output /opt/etc/mrtg.cfg public@localhost

    EnableIPv6: no
    WorkDir: /proto/SxM_webui/mrtg
    Options[_]: bits,growright

    LoadMIBs: /opt/share/snmp/mibs/HOST-RESOURCES-MIB.txt

    kilo[_]: 1024
    XSize[_]: 600
    RouterUpTime[_]: hrSystemUptime.0:public@localhost

    ### Interface 2 >> Descr: 'eth0' | Name: 'eth0' | Ip: '192.168.1.1' | Eth: '' ###

    Target[localhost_2]: 2:public@localhost:
    SetEnv[localhost_2]: MRTG_INT_IP="
    192.168.1.1" MRTG_INT_DESCR="eth0"
    MaxBytes[localhost_2]: 12500000
    Options[localhost_2]: nobanner,noborder,growright,transparent
    Title[localhost_2]: Traffic Analysis for eth0

     
    ### CPU

    Target[localhost.cpu]: hrProcessorLoad.768&hrProcessorLoad.768:public@localhost
    Title[localhost.cpu]: CPU Load
    PageTop[localhost.cpu]:Active CPU load
     
    MaxBytes[localhost.cpu]: 100
    Unscaled[localhost.cpu]: ymwd
    Options[localhost.cpu]: growright,gauge,nopercent,noo,transparent,nolegend,nobanner,noborder
    YLegend[localhost.cpu]: %
    ShortLegend[localhost.cpu]: %
    LegendI[localhost.cpu]: Processor Load

    ### Temperature

    Target[localhost.temp]: `/opt/etc/mrtg/tempscript1.sh`
    Title[localhost.temp]: Temperature Reading
    PageTop[localhost.temp]: Temperature reading
     
    MaxBytes[localhost.temp]: 75
    Unscaled[localhost.temp]: ymwd
    Options[localhost.temp]: growright,gauge,nopercent,transparent,nolegend,nobanner,noborder
    YLegend[localhost.temp]: Celsius
    ShortLegend[localhost.temp]: C
    LegendI[localhost.temp]: HDD Temperature
    LegendO[localhost.temp]: Sys Temperature

    ### Memory

    Target[localhost.mem]: hrStorageUsed.1&hrStorageUsed.1:public@localhost * 1024
    Title[localhost.mem]: Memory Consumption
    PageTop[localhost.mem]:Memory Consumption

    Options[localhost.mem]: growright,gauge,nopercent,noo,transparent,nolegend,nobanner,noborder
    MaxBytes[localhost.mem]: 129871873
    LegendI[localhost.mem]: Used Memory
    ShortLegend[localhost.mem]: Bytes
    Unscaled[localhost.mem]: ymwd
    YLegend[localhost.mem]: Bytes
    Kilo[localhost.mem]: 1024

    ### Hard drive: /
    Target[localhost.hd1]: hrStorageUsed.31&hrStorageUsed.31:public@localhost * 4096
    Title[localhost.hd1]: Disk Usage: / 

    PageTop[localhost.hd1]:Disk Usage: /
    Options[localhost.hd1]: growright,gauge,nopercent,noo,transparent,nolegend,nobanner,noborder
    MaxBytes[localhost.hd1]: 100000000000000000
    LegendI[localhost.hd1]: Storage Used
    ### LegendO[localhost.hd1]: Total
    Kilo[localhost.hd1]: 1024
    YLegend[localhost.hd1]: Used Bytes
    ShortLegend[localhost.hd1]: B

           
    There are some important notes however:
    • Hard disk usage is normalized against the 1TB version of the MBWE. The corresponding MaxBytes property  has to be modified if this is to be used with a hard disk of an other size.
    • The global RouterUpTime property is modified so that it indeed reflects the uptime and not just the uptime of the net-snmp daemon.
    • Memory usage is set to plot against the maximum of 128 MB for the white light edition. For the blue ring edition 64MB is to be used.
    • If the script for temperature reading is put somewhere else, the corresponding Target line has to be altered as well.
    In case everything is OK, which can be tested by manually issuing ´mrtg /opt/etc/mrtg.cfg´ in the shell, the other graphs are filled with data as well:

    14.2.10

    WD MyBook Hacks - Temperature Readings

    This is a topic that puzzled me for some time. Built-in temperature sensors are useful for monitoring the health status of the device and making proper action if for instance overheating occurs. For personal computing, lm-sensors is a frequently-used tool for this purpose which is able to read the sensors via SMBus on most of the motherboards. However, the MyBook has a different facility for this, so we will see how to take advantage of it. In fact, not only the on-board temperature sensor can be read, but the one on the hard drive can be used as well.

    Reading sensor on the hard drive

    This sensor can simply be reached through reading the S.M.A.R.T. data of the drive as I posted earlier. What we need to do is to grep the useful data and get rid of the garbage by selecting the line and the field we are interested in:

    #~: smartctl -a -d ata /dev/sda | awk '/Temperature/ {print $10}'
    44

    will do the job, and we get the temperature of the hard drive in Celsius.

    Reading the on-board temperature sensor

    This one can be interrogated through:

    #~: cat /sys/module/thermAndFan/parameters/current_temp
    27

    which is a reasonable value. If however we look further into the details:

    #~: cat /sys/module/thermAndFan/parameters/hot_limit
    16
    #~: cat /sys/module/thermAndFan/parameters/cold_limit
    104

    this should give the clue that this is not a direct temperature reading. Indeed this sensor does not read in Celsius but rather in "Counts" which is proportional to the sensor resistance value. Luckily enough google finds the lookup table for this platform here (see TvsCnt[] array). As the sensor is a semiconductor thermistor, there is no simple linear equation to determine the temperature, rather the following exponential dependence should be used: Cnt=a×exp(b/T), with Cnt being the reading above and T being the absolute temperature in Kelvin. For this specific sensor the parameters are a=1/9331 and b=3865 as a result by a simple fitting procedure. So after some algebra we find that the temperature will be: t=3865/(ln(Cnt)+9.1411)-273.16 in Celsius (feel free to do it in Fahrenheit :) ).

    In order to implement this equation, we will use the command-line calc tool:
    #~: ipkg install calc

    Finally using this tool a simple script can be used to determine the temperature:

    #!/bin/sh
    Y1=`cat /sys/module/thermAndFan/parameters/current_temp `
    C1='display(0);config("tilde","off");3865/(ln('
    C2=')+9.1411)-273.16'
    /opt/bin/calc $C1$Y1$C2 | awk 'NR==3 {print $1}'  

    which results in a single return value, e. g. 37.

    Finally the question remains: Why would we do this? The reason is that these readings can be plotted using for instance MRTG, so that the health status of the storage can be checked easily.

    30.12.09

    WD MyBook Hacks - S.M.A.R.T.

    There was something that bothered me quite a lot: while the MyBook seems to have smartmontools preinstalled, I was not able to read SMART data from the hard disk:

    ~#:smartctl -a /dev/sda
    smartctl version 5.38 [arm-unknown-linux-gnu] Copyright (C) 2002-8 Bruce Allen
    Home page is http://smartmontools.sourceforge.net/

    Device: WDC      WDC WD10EADS-00L Version: 01.0
    Serial number:      WD-WCAU4C518954
    Device type: disk
    Local Time is: Wed Dec 30 16:23:35 2009 CET
    Device does not support SMART

    Error Counter logging not supported

    [GLTSD (Global Logging Target Save Disable) set. Enable Save with '-S on']
    Device does not support Self Test logging

    This is strange because 3.5" SATA drives supposed to possess SMART capability. After reading the smartmontools wiki through I came up with the solution:

     ~#:smartctl -d sat -a /dev/sda

    an additional "-d sat" parameter is necessary for proper operation. I note however that "-d ata" works as well. In the output we have the table that should look like something like this:

    SMART Attributes Data Structure revision number: 16
    Vendor Specific SMART Attributes with Thresholds:
    ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
      1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
      3 Spin_Up_Time            0x0027   157   156   021    Pre-fail  Always       -       7116
      4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       53
      5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
      7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
      9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       171
     10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
     11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
     12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       11
    192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       4
    193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       53
    194 Temperature_Celsius     0x0022   111   100   000    Old_age   Always       -       39
    196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
    197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
    198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
    199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
    200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

    SMART Error Log Version: 1
    No Errors Logged


    For detailed explanation of the parameters above I suggest reading the corresponding Wikipedia article.