Traefik module for filbeat don't parse traefik logs

Log message example:

85.115.248.206 - - [19/Jan/2018:06:08:56 +0000] “GET /assets/513050f1e4b0a08cc2435786/valuation?oauth_token=ya29.Gl0bBckbBUoFLYZtYEHu4_7LWKAKfNkQpaM9P0FerOmiQXyjth9qOWw0NJjp9_nocSPs2XA_icHafiqbaT7oac-MvSt2zvUkUOko2Q5T9Pe0ORw9rX2bvJFE5bcCJRY HTTP/1.1” 200 320 - “Android” 435365 “Host-api-wearerealitygames-com-2” “http://172.25.0.9:4140” 28ms
Grok pattern:

%{IPORHOST:traefik.access.remote_ip} - %{DATA:traefik.access.user_name} [%{HTTPDATE:traefik.access.time}] “%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}” %{NUMBER:traefik.access.response_code} (?:%{NUMBER:traefik.access.body_sent.bytes}|-)( “%{DATA:traefik.access.referrer}”)?( “%{DATA:traefik.access.agent}”)?(?:%{NUMBER:traefik.access.request_count}|-)?( “%{DATA:traefik.access.frontend_name}”)?( “%{DATA:traefik.access.backend_url}”)?
error message:

field [agent] not present as part of path [traefik.access.agent]
how can I fix that ?

Thanks in advance

Hi @Advik,
Perhaps you can open a bug report on github? Please specify these details there, also the log output from Filebeat & Elasticsearch would help.
If you want to work on the fix, It looks to me there is some error during ingestion: https://github.com/elastic/beats/blob/master/filebeat/module/traefik/access/ingest/pipeline.json#L37, I think it’s failing there, adding “ignore_failure” : true there would probably help.
Best regards

thanks, I’ll do that asap, also I see that user agent “android” is not parsed. No fields after traefik.access.body_sent.bytes are parsed.
input:

85.115.248.206 - - [19/Jan/2018:06:08:56 +0000] “GET /assets/513050f1e4b0a08cc2435786/valuation?oauth_token=ya29.Gl0bBckbBUoFLYZtYEHu4_7LWKAKfNkQpaM9P0FerOmiQXyjth9qOWw0NJjp9_nocSPs2XA_icHafiqbaT7oac-MvSt2zvUkUOko2Q5T9Pe0ORw9rX2bvJFE5bcCJRY HTTP/1.1” 200 320 - “Android” 435365 “Host-api-wearerealitygames-com-2” “http://172.25.0.9:4140” 28ms
pattern:

%{IPORHOST:traefik.access.remote_ip} - %{DATA:traefik.access.user_name} [%{HTTPDATE:traefik.access.time}] “%{WORD:traefik.access.method} %{DATA:traefik.access.url} HTTP/%{NUMBER:traefik.access.http_version}” %{NUMBER:traefik.access.response_code} (?:%{NUMBER:traefik.access.body_sent.bytes}|-)( “%{DATA:traefik.access.referrer}”)?( “%{DATA:traefik.access.agent}”)?(?:%{NUMBER:traefik.access.request_count}|-)?( “%{DATA:traefik.access.frontend_name}”)?( “%{DATA:traefik.access.backend_url}”)?
output:

{

“traefik”: {

“access”: {

“response_code”: “200”,

“remote_ip”: “85.115.248.206”,

“method”: “GET”,

“user_name”: “-”,

“http_version”: “1.1”,

“body_sent”: {

“bytes”: “320”

},

“time”: “19/Jan/2018:06:08:56 +0000”,

“url”: “/assets/513050f1e4b0a08cc2435786/valuation?oauth_token=ya29.Gl0bBckbBUoFLYZtYEHu4_7LWKAKfNkQpaM9P0FerOmiQXyjth9qOWw0NJjp9_nocSPs2XA_icHafiqbaT7oac-MvSt2zvUkUOko2Q5T9Pe0ORw9rX2bvJFE5bcCJRY”

}

}

}
related GitHub issue: https://github.com/elastic/beats/issues/6111

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.