Filter Nested Data based on pattern

I have incoming JSON data that looks like:

{ “DataType1”: { “system:subsystem” : [{…some nested data fields}]

“DataType2”: { “system:subsystem” : [{…some nested data fields}]

}
I am trying to separate the events, so they are based on system:subsystem

I want each of my event to be as of such:

{system:subsystem

[{…some nested data fields}]

}

system will always be the same, subsystem will differ each time.
whats the best way to extract this from a nested structure?

Any ideas? I need to somehow save that “system:subsystem” is a pattern and that based on that,for all of these I will have separate events.

In my case, subsystem will be always a different value.
Whats the best way to split nested fields into separate events?

I tried the split filter and no luck. Trying something like:

split { field => “[DataType2][subsystem]” }

}

it doesn’t work as my subsystem is unique each time. I also tried grok with match but not succeeding with the match.

Tried something like the below, just to see if I can match the system:subsystem:

grok {

match => { “message” => “%{WORD:DataType}: {%{GREEDYDATA:reeest}”} }

You’ll need to use a ruby filter to iterate over the fields and transform them into a single array of objects containing the fields you want to have after the split, followed by a split filter to split the array.

Can you provide an example of such a configuration for ruby?

Also, would it be possible to save the “system:subsystem” as key values with kv, instead of using ruby?

Actually, my data has the following format, bit more complicated. How can I use ruby for this?

“DataType1” => {

“system:subsystem123” => [

[0] {

data to be parsed, some in further arrays

}

],

“system:subsystem234” => [

[0] {

data to be parsed some in further arrays

}

],

“system:subsystem837” => [

[0] {

data to be parsed some in further arrays

}

],

etc.
Maybe I could use grok/regex to parse field which looks like system:subsystem?
NOTE; I found ruby code that almost does the job, but it doesn’t take account the changeable system:subsystem, the field is static.

filter {

ruby {

code => "

event.get(‘traits’).each_index {|i|

event.set([traits][i][0], [traits][i][2])

}

}

}

Any idea how to get the ‘traits’ in above example to work with a field that is not a static name? For example: “system:subsystem123”, “system:subsystem235”, “system:subsystem203”?

Hi Magnus,
I am trying to attempt with ruby, but its not working- I am getting no output. Can you give advice:

ruby {

code =>

“event.get[DataType1].each { |e|

h = {}

h[SYSTEM:SUBSYSTEM] = e.first

h.merge! e[1][0]

event.set[DataType1] << h

}”

}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.