Icon

Comments have been closed on this page. Please use AppMon & UEM Plugins forum for questions about this plugin.

Varnish is a web application accelerator.  You install it in front of your web application and it will speed it up significantly.

 

The varnishStat Plugin enables monitoring key stats from the varnishstas utility.  The plugin uses ssh to connect to a Linux server and then executes
varnishstats –j to pull back the varnish stats in Json format.  The plugin then creates the varnish stat measures in dynaTrace by parsing the Json inputStream.

Name and Version

VarnishStats Monitoring Plugin

Compatible with

dynaTrace 5.x, 6.x

Author

Jeff Yarbrough - Dynatrace

Bill Gillette - Dynatrace (1.0.9.1)

License

dynaTrace BSD

 

Support Level

Not Supported 
If you have any questions or suggestions for these plugins, please add a comment to this page, use our forum, or drop us an email at community@dynatrace.com!

 

Download

varnishstat.1.0.7 (Varnish 3.x)

varnishstat.1.0.8 (Varnish 3.x)

VarnishStats.1.0.8.1.jar (Varnish 3.x) - with RSA keyFile option

varnishstat.1.0.9.1 (Varnish 4.x)

VarnishStat.1.1.0 dtp file -(Varnish 4.x) - with RSA keyFile option

varnish_1.1.0.jar (jar file version)

Installation

Import the plugin into dynaTrace server

Release History

2013-07-25 Initial Release

2016-02-19 version 1.0.9 for use with Varnish 4.x

2016-08-03 added RSA keylogin


User Verification:

You must verify the user has the permissions on the target machine to pull the stats with the varnishstat -j command.

Steps:

Import Plugin:

Import the Plugin into the dynaTrace Server. For details how to do this please refer to the dynaTrace documentation.

After you import the plugin you will see the plugin as VarnishStats, type of plugin "Monitor Metric Group, Monitor".

Setup Monitor:

Under your system profile, you will need to create a monitor for the varnish plugin.  Select Monitors under the profile and select "Create".

Select the VarnishStats monitor and click "ok".

Configure Monitor:

You will need to provide the login credentials (username/password) for the Linux host and give the monitor a name.

Make sure to add a host or hosts to the host configuration.

Set the schedule,

(Optionally) You can select which measures you would like to subscribe.

 

Click "OK", then you should see the Monitor has been created.

Validation:

You will need select the monitor and click "Run Now", the monitor should display a green indicator to let you know everything worked.

Measures:

You can now see the measure for VarnishStats under "System Monitoring".

version 1.0.7 and 1.0.8 (Varnish 3.x)

Version 1.0.9 (Varnish 4.x)

Configuration

Name

Value

Username

The username to use for authentication to the restful interface

Password

The password for the username used to authenticate to the restful interface.

commanddefault is "varnish stat -j" Do Not Change

Troubleshooting

1) User executing command needs proper permissions.

/var/lib/varnish/<host>/_.vsm and make it read/write (chmod +rw)

2) If you see this error in the log, <DTHOME>/log/collector/<collector target>/com.dynatrace.diagnostics.plugin.Varnish.Monitor.0.0.log

2016-02-24 14:46:12 INFO [VarnishMonitor@7b956074-a4c4-4347-a93e-665dc1312a20] com.jcraft.jsch.JSchException: Algorithm negotiation fail

Fix:

In my case - OpenSSH_6.7p1 on server - I had to modify KexAlgorithms and MACs (additional hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96 values):

KexAlgorithms curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,hmac-ripemd160,umac-128@openssh.com,hmac-md5,hmac-sha1,hmac-sha1-96,hmac-md5-96

Above should be placed:

/etc/ssh/sshd_config

And then restart the ssh:

sudo /etc/init.d/ssh restart

Contribution

Feel free to contribute any changes on Github

 

  1. Anonymous (login to see details)

    Nice work Jeff.

    1. Anonymous (login to see details)

      Thanks, please let me know if there is anything else we need for this plugin.

  2. Anonymous (login to see details)

    Hello Jeff,

    I have been trying to use your plugin. However, it looks like retrieved values are cumulated values since Varnish instance got started.
    Would you have an option to get the instaneaous values (ie:  Client requests received, Cache hits, Cache misses...) we have in the second column when running varnishstat in live mode?

    If you do not have time to update your plugin, would you hou mind sharing the source so that I can see how we could update it?

    Thanks,

  3. Anonymous (login to see details)

    What command are you running to retrieve the Varnish Stats you want?  Can you give me a list of the values you want added to the plugin?

    I can look at updating the plugin once I get this information.

    Thanks,

    1. Anonymous (login to see details)

      Jeff,

      Command I run is : /appl/varnish/bin/varnishstat -f client_conn,client_req,cache_hit,cache_miss,backend_fail,n_wrk,n_wrk_failed,n_wrk_max,n_wrk_drop,n_wrk_queued,n_lru_nuked

      Values I would like to get are values from the second column which correspond to instantaneaous ones (xxx/sec)

       

      Thanks!

  4. Anonymous (login to see details)

    Hey Jeff,

    Have you add time to look at my suggestion?

    Thanks!

    1. Anonymous (login to see details)

      I added the additional fields from your list above.

       com.dynatrace.diagnostics.plugin.Varnish_1.0.8.jar

      About the second column which corresponds to instantaneous ones.  When you run the command as you do, the output is to the screen.
      Which gives you updated stats on the screen.

      Such as this,

      However, if you run the command and have it output to a JSON file,

      varnishstat -j -f client_conn,client_req,cache_hit,cache_miss,backend_fail,n_wrk,n_wrk_failed,n_wrk_max,n_wrk_drop,n_wrk_queued,n_lru_nuked

      Note the -j flag.

      You will see the JSON output.

      {
      "timestamp": "2014-02-10T16:10:16",
      "client_conn": {"value": 1893, "flag": "a", "description": "Client connections accepted"},
      "client_req": {"value": 3746, "flag": "a", "description": "Client requests received"},
      "cache_hit": {"value": 66, "flag": "a", "description": "Cache hits"},
      "cache_miss": {"value": 1023, "flag": "a", "description": "Cache misses"},
      "backend_fail": {"value": 0, "flag": "a", "description": "Backend conn. failures"},
      "n_wrk": {"value": 10, "flag": "i", "description": "N worker threads"},
      "n_wrk_failed": {"value": 0, "flag": "a", "description": "N worker threads not created"},
      "n_wrk_max": {"value": 0, "flag": "a", "description": "N worker threads limited"},
      "n_wrk_queued": {"value": 0, "flag": "a", "description": "N queued work requests"},
      "n_wrk_drop": {"value": 0, "flag": "a", "description": "N dropped work requests"},
      "n_lru_nuked": {"value": 0, "flag": "i", "description": "N LRU nuked objects"}
      }

      Since this is a monitor plugin and is intended to pull the stats at a certain interval, then process the JSON output, it is not going to pull instantaneous results in the same manner.

      I have this monitor plugin running every minute, which I can then use to graph the output over time.  You could create a new schedule and run it every 30 sec, not sure if I would go much below that.

      Does this help, or do you need it to do some other function?

      Thanks,

      Jeff

      1. Anonymous (login to see details)

        Thanks Jeff for your feedback.

        I have tried to install this 1.0.8 but I am then not able to create any Varnish monitor. It does not appear in the "Monitor Type" list.
        When I switch back to 1.0.7 i can find the Varnish monitor type. I am runing Dynatrace 5.5.

        Regarding values, the problem with the JSON output, is that values fetched are cumulative values since the server startup. Graphs will just show increasing values but not instantaneous ones. ie : for client_requests, I am interested in knowing how many requests there have been between current and last checks.

        1. Anonymous (login to see details)

          odd, I have it running on my system, I will check the export and try to update it once I have a chance.

          The issue is how the varnishstats utility works.  It does not really provide options to reset the data at any given point.

          The data needs to be exported and processed to input into dynaTrace.   The JSON format provides the best option for this import of the data.  It is not as easy to try and take a screen scrap or something of that nature.

          I really didn't see a flag that would give what you are referring to in a JSON format.

          Thanks,

          1. Anonymous (login to see details)

            I understand that JSON is much easier on Java side to process.

            But by getting cumulative values, I am wondering how they can easily be interpreted on dynaTrace side since we just get lines that keep increasing and we cannot really compare trends among days.

            Would you be able on plugin side to export in dynaTrace the difference between what you get and the last polled value? Or maybe this can be achieved afterwards with dynatrace values?

            1. Anonymous (login to see details)

              What would be great, if we could get the varnish stat to produce the instantaneous with the Json format.

              Maybe passing it a flag or something.  I looked, but could not find an easy way to request this from the Varnish team.

  5. Anonymous (login to see details)

    Would be nice to have option for SSH using key based auth, as hardcoding passwords is potentially problematic.

    The Dynatrace Unix System Monitor Plugin provides this capability, for reference,

    1. Anonymous (login to see details)

      I will look at adding key based authentication in the next version.  I will try to get this added sometime in the next week.

      Thank you for your feedback.

  6. Anonymous (login to see details)

    Hi Jeff,

     

    Is this plugin also compatible with dynaTrace 6.0?

     

    Kind regards,

     

    Chris

    1. Anonymous (login to see details)

      Yes - it is. I just tested it out

      1. Anonymous (login to see details)

        Thank you Andreas, yes, I tested it also with 6.0.

         

  7. Anonymous (login to see details)

    Hello,

    I am using Varnish 4.02

    The stats labels seems change:

    {

            "timestamp": "2015-02-06T11:44:14",

            "MAIN.uptime": {"type": "MAIN", "value": 6862, "flag": "a", "description": "Child process uptime"},

            "MAIN.sess_conn": {"type": "MAIN", "value": 144596, "flag": "c", "description": "Sessions accepted"},

            "MAIN.sess_drop": {"type": "MAIN", "value": 0, "flag": "c", "description": "Sessions dropped"},

            "MAIN.sess_fail": {"type": "MAIN", "value": 0, "flag": "c", "description": "Session accept failures"},

            "MAIN.sess_pipe_overflow": {"type": "MAIN", "value": 0, "flag": "c", "description": "Session pipe overflow"},

            "MAIN.client_req_400": {"type": "MAIN", "value": 0, "flag": "a", "description": "Client requests received, subject to 400 errors"},

            "MAIN.client_req_411": {"type": "MAIN", "value": 0, "flag": "a", "description": "Client requests received, subject to 411 errors"},

            "MAIN.client_req_413": {"type": "MAIN", "value": 0, "flag": "a", "description": "Client requests received, subject to 413 errors"},

            "MAIN.client_req_417": {"type": "MAIN", "value": 0, "flag": "a", "description": "Client requests received, subject to 417 errors"},

            "MAIN.client_req": {"type": "MAIN", "value": 144645, "flag": "a", "description": "Good Client requests received"},

            "MAIN.cache_hit": {"type": "MAIN", "value": 349, "flag": "a", "description": "Cache hits"},

            "MAIN.cache_hitpass": {"type": "MAIN", "value": 0, "flag": "a", "description": "Cache hits for pass"},

            "MAIN.cache_miss": {"type": "MAIN", "value": 144245, "flag": "a", "description": "Cache misses"},

            "MAIN.backend_conn": {"type": "MAIN", "value": 1101, "flag": "a", "description": "Backend conn. success"},

            "MAIN.backend_unhealthy": {"type": "MAIN", "value": 0, "flag": "a", "description": "Backend conn. not attempted"},

     

     

     

    1. Anonymous (login to see details)

      Damien,

      I will look at creating a new version of the plugin to handle the name changes above.  Once I get it into a testing phase, will it be possible to have you test it on your system as I do not have that version of varnish.

      Thanks,

  8. Anonymous (login to see details)

    I would like to test it as well. I am running varnish 4.x

  9. Anonymous (login to see details)

    Version 1.0.9 is now available for your use.

    I did some initial testing and all looks good.  

    Change List:

    --Change made for measure names, match JSon format
    --Loginfo to show target machine
    --Loginfo to show plugin steps
    --Added feature to build xml from JSon Map (for development)
    --Added Additional measures for entire JSon Map

    If you are running Varnish 4.x, please give this plugin a try and provide any feedback.

    RFE: Key Based Authentication
    RFE: Difference between Values (this may be better done in the Dynatrace Client) 

    1. Anonymous (login to see details)

      I am getting this:

       

      Schedule Details
      Name: localhost.localdomain
      Status: scheduled
      Description:
      Type: VarnishStats
      Last Run: 19:05:18 (CET)
      Last Run Result: ok
      Next Run: 19:06:18 (CET)
      Current Run: -
      Duration (last run): 2s
      Schedule: Every Minute
      Execution Target: dynaTrace Collector@localhost.localdomain
      Plugin Active: active
      Result Status: success
      Result: Last polling execution terminated successfully
      Message: Success
      Detailed message: Plugin execution successful
      Retrieved measurements: LCK.backend.creat: null
      LCK.backend.destroy: null
      LCK.backend.locks: null
      LCK.backend_tcp.creat: null
      LCK.backend_tcp.destroy: null
      LCK.backend_tcp.locks: null
      LCK.ban.creat: null

      1. Anonymous (login to see details)

        Can you verify the same user has permissions to retrieve the stats?  I had to change the permissions on this file,

        /var/lib/varnish/<host>/_.vsm and make it read/write (chmod +rw), then the user I was using to login into the linux machine was able to run the command.

         

  10. Anonymous (login to see details)

    And I am not able to create graphs

  11. Anonymous (login to see details)

    This is the output from the command line:

     

    {
    "timestamp": "2016-02-19T19:04:16",
    "MAIN.uptime": {"type": "MAIN", "value": 282959, "flag": "a", "description": "Child process uptime"},
    "MAIN.sess_conn": {"type": "MAIN", "value": 143278, "flag": "c", "description": "Sessions accepted"},
    "MAIN.sess_drop": {"type": "MAIN", "value": 0, "flag": "c", "description": "Sessions dropped"},
    "MAIN.sess_fail": {"type": "MAIN", "value": 0, "flag": "c", "description": "Session accept failures"},
    "MAIN.sess_pipe_overflow": {"type": "MAIN", "value": 0, "flag": "c", "description": "Session pipe overflow"},
    "MAIN.client_req_400": {"type": "MAIN", "value": 13, "flag": "a", "description": "Client requests received, subject to 400 errors"},
    "MAIN.client_req_411": {"type": "MAIN", "value": 2, "flag": "a", "description": "Client requests received, subject to 411 errors"},
    "MAIN.client_req_413": {"type": "MAIN", "value": 0, "flag": "a", "description": "Client requests received, subject to 413 errors"},
    "MAIN.client_req_417": {"type": "MAIN", "value": 7, "flag": "a", "description": "Client requests received, subject to 417 errors"},
    "MAIN.client_req": {"type": "MAIN", "value": 692030, "flag": "a", "description": "Good client requests received"},
    "MAIN.cache_hit": {"type": "MAIN", "value": 53357, "flag": "a", "description": "Cache hits"},
    "MAIN.cache_hitpass": {"type": "MAIN", "value": 21748, "flag": "a", "description": "Cache hits for pass"},
    "MAIN.cache_miss": {"type": "MAIN", "value": 35858, "flag": "a", "description": "Cache misses"},
    "MAIN.backend_conn": {"type": "MAIN", "value": 34053, "flag": "a", "description": "Backend conn. success"},
    "MAIN.backend_unhealthy": {"type": "MAIN", "value": 0, "flag": "a", "description": "Backend conn. not attempted"},
    "MAIN.backend_busy": {"type": "MAIN", "value": 0, "flag": "a", "description": "Backend conn. too many"},
    "MAIN.backend_fail": {"type": "MAIN", "value": 0, "flag": "a", "description": "Backend conn. failures"},
    "MAIN.backend_reuse": {"type": "MAIN", "value": 604628, "flag": "a", "description": "Backend conn. reuses"},
    "MAIN.backend_toolate": {"type": "MAIN", "value": 29210, "flag": "a", "description": "Backend conn. was closed"},
    "MAIN.backend_recycle": {"type": "MAIN", "value": 633867, "flag": "a", "description": "Backend conn. recycles"},
    "MAIN.backend_retry": {"type": "MAIN", "value": 3, "flag": "a", "description": "Backend conn. retry"},

  12. Anonymous (login to see details)

    Version 1.1.0 is now available

    The plugin now allows Key Based Authentication. 
    Setup the target machine with the RSA keys from the machine hosting the Dynatrace Collector to the target machine. 

    I also added a small change to allow the port to be changed.

    Note, Added version 1.0.8.1 for Varnish version 3.x.

    Please select appropriate plugin version for your version of varnish.