How to get kv filter to ignore value_split in data

My conf file:

kv {
  source => "cp_keyValueData"
  field_split => ";"
  value_split => ":"
  trim_key => " "

data example:

key1: val1; key2: val2; key3:  https://site/?g={......"...;%20%20CLR%20;%20rv:11.0)"..}; key4: val4;

it split to:

key1:             val1
key2:             val2;
key3:            https://site/?g={......"...;%20%20CLR%20;
%20rv:         11.0)"..};
key4:            val4;

How can i prevent this mistake?
There is some way to ignore value_split that found in string?
thank u !

I’ve have had a similar problem.

My solution was to implement my own splitter in Ruby.
Here’s my topic:

How to handle ‘=’ in values, splitting on | but KV takes over all ‘=’ not only the first
But as I see, you have a problem with pair split parameter’;’, mine was with key/value split parameter ‘=’.

Hope you’re can use mine as a waypoint.

Your issue is not with the value_split, it is with the field_split of ;.
You have two options:

  1. If your values are always percent encoded when it contains a ; e.g. https://site/?g={......"...;%20%20CLR%20;%20rv:11.0)"..}, then inside the value you will never see ; i.e. "semi-colon space" so you can make that your field_split value.
  2. If your values are not percent encoded and contains a ; then your only option is to use mutate gsub but not simply to replace the ; because that will replace the semi-colon in the values too. You will need know all the possible keys and gsub for them but also to use a named pattern to capture the found key and substitute it back.
input {
  generator {
    message => 'key1: val1; key2: val2; key3:  https://site/?g={......"...;  CLR  rv:11.0)"..}; key4: val4;'
    count => 1

filter {
mutate {
gsub => ["[message]", “;\s*(?<key>key1|key2|key3|key4)”, ‘|^|\k<key>’]
kv {
field_split => “|^|”
value_split => “:”
source => “message”

output {
stdout {
codec => rubydebug {metadata => true}

          "key1" => "val1",
      "sequence" => 0,
          "key2" => "val2",
    "@timestamp" => 2018-01-17T11:57:10.574Z,
          "key3" => "https://site/?g={......\"...;  CLR  rv:11.0)\"..}",
          "key4" => "val4;",
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
       "message" => "key1: val1|^|key2: val2|^|key3:  https://site/?g={......\"...;  CLR  rv:11.0)\"..}|^|key4: val4;"

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.