This page (revision-13) was last changed on 23-Apr-2022 17:05 by HarryMetske

This page was created on 23-Apr-2022 17:05 by unknown

Only authorized users are allowed to rename pages.

Only authorized users are allowed to delete pages.

Page revision history

Version Date Modified Size Author Changes ... Change note
13 23-Apr-2022 17:05 10 KB HarryMetske to previous
12 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
11 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
10 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
9 23-Apr-2022 17:05 10 KB Harry Metske to previous | to last
8 23-Apr-2022 17:05 8 KB Harry Metske to previous | to last
7 23-Apr-2022 17:05 8 KB Harry Metske to previous | to last
6 23-Apr-2022 17:05 7 KB Harry Metske to previous | to last
5 23-Apr-2022 17:05 7 KB Harry Metske to previous | to last
4 23-Apr-2022 17:05 5 KB HarryMetske to previous | to last
3 23-Apr-2022 17:05 5 KB Harry Metske to previous | to last
2 23-Apr-2022 17:05 5 KB Harry Metske to previous | to last
1 23-Apr-2022 17:05 4 KB unknown to last

Page References

Incoming links Outgoing links
ELK...nobody
ELK

Version management

Difference between version and

At line 4 removed one line
[{TableOfContents }]
At line 127 removed 2 lines
! logstash.conf examples
At line 139 removed one line
At line 148 removed 7 lines
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
At line 158 changed one line
elasticsearch { hosts => ["athena:9200","10.0.0.162:9200"] }
elasticsearch { hosts => ["athena:9200"] }
stdout { codec => rubydebug }
At line 161 removed one line
At line 166 changed 106 lines
%%collapsebox
__logstash.conf__
%%prettify
{{{
input { stdin { } }
filter {
grok {
match => { "message" => "\"(?:%{IPORHOST:clientip}|-)\" %{IPORHOST:vhost} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken} %{QS:sessionid} %{QS:remove1} %{QS:referrer} %{QS:useragent}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "source", "websphere" ]
remove_field => [ "%{remove1}" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
convert => { "bytes" => "integer" }
convert => { "timetaken" => "integer" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
useragent {
source => "useragent"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
# stdout { codec => rubydebug }
}
}}}
%%
%%
%%collapsebox
__logstash.conf__
%%prettify
{{{
input { stdin { } }
filter {
grok {
match => { "message" => "%{IPORHOST:clientip} %{IPORHOST:vhost} %{NOTSPACE:remove1} %{NOTSPACE:remove2} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken} %{NOTSPACE:remove3} %{NUMBER:keepalivenr} %{QS:referrer} %{QS:useragent} (?:%{PATH:filename}|-)" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "source", "statics" ]
remove_field => [ "%{remove1}","%{remove2}","%{remove3}" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
convert => { "bytes" => "integer" }
convert => { "timetaken" => "integer" }
convert => { "keepalivenr" => "integer" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
useragent {
source => "useragent"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
# stdout { codec => rubydebug }
}
}}}
%%
%%
Also first download a ["GeoLiteCity DB"|http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz] and unzip it to /etc/logstash.
Make the user logstash part of the {{adm}} group (so it can read logfiles) and restart: ''/etc/init.d/logstash restart'' and there we have an logstash-* index in elasticsearch with all requested fields, hurray !
!! Elasticsearch URLs
! Delete an index
{{{
curl -XDELETE 'http://localhost:9200/logstash-*/'
}}}
! Index a document
{{{
curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
"user" : "kimchy",
"post_date" : "2015-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}'
}}}
Add and index a document to the twitter index, with documentid 1
Restart: ''/etc/init.d/logstash restart'' and there we have an logstash-* index in elasticsearch with all requested fields, hurray !
This page (revision-13) was last changed on 23-Apr-2022 17:05 by HarryMetskeTop