How to use logstash filters

I am new to the whole BELK stack (or The Elastic Stack from 5.0.0) so sorry if the question is not well described.
I have logs that looks like the example below that I ship with filebeat to logstash
Log example:

2/24/2016 6:18:20 AM +00:00|FdsResponder|Information|18437|{“ActionName”:“ResumeUpload”,“Exception”:null,“Metadata”: … well defined JASON 2/24/2016 6:18:21 AM +00:00|FdsResponder|Information|18436|{“ActionName”:“ResumeDownload”,“Exception”:null,“Metadata”: … well defined JASON 2/24/2016 6:20:21 AM +00:00|SDS Responder|Information|0|Loading manifest Guid: … clear text 2/24/2016 6:20:22 AM +00:00|SDS Responder|Warning|4107|No load priority found for plug-in … clear text 2/24/2016 6:20:23 AM +00:00|FdsResponder|Information|18437|{“ActionName”:“ResumeUpload”,“Exception”:null,“Metadata”: … well defined JASON
Logstash conf:

input { beats { port => 5044 } } output { stdout { codec => rubydebug } }
Filebeat.yml:

`filebeat:

prospectors:

paths:

  • D:\filebeat\input*.log

    input_type: log
    registry_file: “C:/ProgramData/filebeat/registry”
    output:

    logstash:

    hosts: [“10.114.21.91:5044”]
    shipper:
    logging:

    files:

    path: D:\filebeat\logs

    rotateeverybytes: 10485760 # = 10MB

    level: debug`
    What I would love to accomplish is to first of all only ship the lines in the log file where the second column = FdsResponder to Logstash. Then have Logstash only output the JSON part of the logfile but using the time/date from the logfile (column 1) as the @Danvi. Is this possible?
    Thank you all for your help an input.

It’s possible!

You will need to build a grok pattern to create fields for each of the values, then use a conditional to drop anything you don’t want. Once you are there you can use the date filter to do the date matching.
Check out the docs around grok, and use http://grokdebug.herokuapp.com/ to build your pattern.
if you run into problems we’re here to help!

I appreciate your answer Mark but could you or others possible show me an little example of how I do this just to get started? Like the Date Filter, I am not sure how to use it correctly.

The first thing you want to do is build the grok pattern, without that you cannot use the date filter.

So how does that look?