This page (revision-13) was last changed on 23-Apr-2022 17:05 by HarryMetske

This page was created on 23-Apr-2022 17:05 by unknown

Only authorized users are allowed to rename pages.

Only authorized users are allowed to delete pages.

Page revision history

Version Date Modified Size Author Changes ... Change note
13 23-Apr-2022 17:05 10 KB HarryMetske to previous
12 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
11 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
10 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
9 23-Apr-2022 17:05 10 KB Harry Metske to previous | to last
8 23-Apr-2022 17:05 8 KB Harry Metske to previous | to last
7 23-Apr-2022 17:05 8 KB Harry Metske to previous | to last
6 23-Apr-2022 17:05 7 KB Harry Metske to previous | to last
5 23-Apr-2022 17:05 7 KB Harry Metske to previous | to last
4 23-Apr-2022 17:05 5 KB HarryMetske to previous | to last
3 23-Apr-2022 17:05 5 KB Harry Metske to previous | to last
2 23-Apr-2022 17:05 5 KB Harry Metske to previous | to last
1 23-Apr-2022 17:05 4 KB unknown to last

Page References

Incoming links Outgoing links
ELK...nobody
ELK

Version management

Difference between version and

At line 4 added one line
[{TableOfContents }]
At line 127 added 2 lines
! logstash.conf examples
At line 139 added one line
At line 148 added 7 lines
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
At line 147 changed 2 lines
elasticsearch { hosts => ["athena:9200"] }
stdout { codec => rubydebug }
elasticsearch { hosts => ["athena:9200","10.0.0.162:9200"] }
At line 161 added one line
At line 155 changed one line
Restart: ''/etc/init.d/logstash restart'' and there we have an logstash-* index in elasticsearch with all requested fields, hurray !
%%collapsebox
__logstash.conf__
%%prettify
{{{
input { stdin { } }
filter {
grok {
match => { "message" => "\"(?:%{IPORHOST:clientip}|-)\" %{IPORHOST:vhost} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken} %{QS:sessionid} %{QS:remove1} %{QS:referrer} %{QS:useragent}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "source", "websphere" ]
remove_field => [ "%{remove1}" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
convert => { "bytes" => "integer" }
convert => { "timetaken" => "integer" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
useragent {
source => "useragent"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
# stdout { codec => rubydebug }
}
}}}
%%
%%
%%collapsebox
__logstash.conf__
%%prettify
{{{
input { stdin { } }
filter {
grok {
match => { "message" => "%{IPORHOST:clientip} %{IPORHOST:vhost} %{NOTSPACE:remove1} %{NOTSPACE:remove2} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken} %{NOTSPACE:remove3} %{NUMBER:keepalivenr} %{QS:referrer} %{QS:useragent} (?:%{PATH:filename}|-)" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "source", "statics" ]
remove_field => [ "%{remove1}","%{remove2}","%{remove3}" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
convert => { "bytes" => "integer" }
convert => { "timetaken" => "integer" }
convert => { "keepalivenr" => "integer" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
useragent {
source => "useragent"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
# stdout { codec => rubydebug }
}
}}}
%%
%%
Also first download a ["GeoLiteCity DB"|http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz] and unzip it to /etc/logstash.
Make the user logstash part of the {{adm}} group (so it can read logfiles) and restart: ''/etc/init.d/logstash restart'' and there we have an logstash-* index in elasticsearch with all requested fields, hurray !
!! Elasticsearch URLs
! Delete an index
{{{
curl -XDELETE 'http://localhost:9200/logstash-*/'
}}}
! Index a document
{{{
curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
"user" : "kimchy",
"post_date" : "2015-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}'
}}}
Add and index a document to the twitter index, with documentid 1
This page (revision-13) was last changed on 23-Apr-2022 17:05 by HarryMetskeTop