Testing and benchmarking log collection pipeline

,
#1

I’m looking for tools, or a set of tools, to test and benchmark logstash+kafka+ES setups :

  • define test cases (log types, throughputs)
  • maintain a base of log examples
  • generate inputs for logstash at a given rate from the example files (some like https://www.snmpsoft.com/cmd-tools/syslog-gen/)
  • reproduce results with minimal humain vérification
Does something like a standard toolset exist to do thing sort of thing ?
#2

I think Elastic have a bunch of performance test suites somewhere, but other than that I haven’t heard of the kind of end-to-end testing tools that you describe. However, for testing Logstash filters specifically I think my own tool Logstash Filter Verifier is useful. Verifying that filters are working as expected is probably easier done on the unit level than on the system level.

#3

Thanks Magnus,
I agree with you on unittesting your filters and patterns. I actually build

a tool similar to yours (a lot simpler, using rspec) that allows you to

define test cases for patterns, as well as filters: Logstash Tester

https://github.com/gaspaio/logstash-tester.
I think an end to end testing suite would be really useful, for those cases

when you have a defined set of logs that gets parsed, indexed and triggers

some some of app behavior, and you want to test the whole chain and

reproduce behaviros. Oh well, i’ll probably end up hacking something myself

for this.