This page (revision-13) was last changed on 23-Apr-2022 17:05 by HarryMetske

This page was created on 23-Apr-2022 17:05 by unknown

Only authorized users are allowed to rename pages.

Only authorized users are allowed to delete pages.

Page revision history

Version Date Modified Size Author Changes ... Change note
13 23-Apr-2022 17:05 10 KB HarryMetske to previous
12 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
11 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
10 23-Apr-2022 17:05 10 KB HarryMetske to previous | to last
9 23-Apr-2022 17:05 10 KB Harry Metske to previous | to last
8 23-Apr-2022 17:05 8 KB Harry Metske to previous | to last
7 23-Apr-2022 17:05 8 KB Harry Metske to previous | to last
6 23-Apr-2022 17:05 7 KB Harry Metske to previous | to last
5 23-Apr-2022 17:05 7 KB Harry Metske to previous | to last
4 23-Apr-2022 17:05 5 KB HarryMetske to previous | to last
3 23-Apr-2022 17:05 5 KB Harry Metske to previous | to last
2 23-Apr-2022 17:05 5 KB Harry Metske to previous | to last
1 23-Apr-2022 17:05 4 KB unknown to last

Page References

Incoming links Outgoing links
ELK...nobody
ELK

Version management

Difference between version and

At line 4 removed one line
[{TableOfContents }]
At line 10 removed one line
* [logstash grok patterns|https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns]
At line 13 changed one line
!! Installing Elasticsearch
!! Installing
At line 56 removed 2 lines
The docker images also runs logstash, which we don't need now (see further), we will send the processed logs directly to elasticsearch.
At line 116 removed 156 lines
The net result of the above actions is that we do get data in elasticsearch, but all loglines are stored as one field called ''message''. \\
What we want is that the apache logfile is parsed and we store alle fields (clientip, request, response code and so on) be stored in elasticsearch.\\
I spent several hours to find out how this should be done with filebeat, but could not find it, I guess it must be something with the filebeat.template.json.\\
Anyways, I continued with the classic logstash, see next chapter.
!! Installing (classic) logstash
I first installed the [logstash deb|https://download.elastic.co/logstash/logstash/packages/debian/logstash_2.0.0-1_all.deb], and next created the following logstash config file :
! logstash.conf examples
%%collapsebox
__logstash.conf__
%%prettify
{{{
input {
file {
path => "/var/log/apache2/access.log"
type => "apache2"
}
}
filter {
grok {
match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} \[%{HTTPDATE:timestamp}\] %{NUMBER:timetaken} \"%{IPORHOST:vhost}\" \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{QS:referrer} %{QS:agent}" }
add_field => [ "received_at", "%{@timestamp}" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
}
output {
elasticsearch { hosts => ["athena:9200","10.0.0.162:9200"] }
}
}}}
%%
%%
%%collapsebox
__logstash.conf__
%%prettify
{{{
input { stdin { } }
filter {
grok {
match => { "message" => "\"(?:%{IPORHOST:clientip}|-)\" %{IPORHOST:vhost} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken} %{QS:sessionid} %{QS:remove1} %{QS:referrer} %{QS:useragent}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "source", "websphere" ]
remove_field => [ "%{remove1}" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
convert => { "bytes" => "integer" }
convert => { "timetaken" => "integer" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
useragent {
source => "useragent"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
# stdout { codec => rubydebug }
}
}}}
%%
%%
%%collapsebox
__logstash.conf__
%%prettify
{{{
input { stdin { } }
filter {
grok {
match => { "message" => "%{IPORHOST:clientip} %{IPORHOST:vhost} %{NOTSPACE:remove1} %{NOTSPACE:remove2} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NUMBER:timetaken} %{NOTSPACE:remove3} %{NUMBER:keepalivenr} %{QS:referrer} %{QS:useragent} (?:%{PATH:filename}|-)" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "source", "statics" ]
remove_field => [ "%{remove1}","%{remove2}","%{remove3}" ]
}
date {
match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
mutate {
convert => { "bytes" => "integer" }
convert => { "timetaken" => "integer" }
convert => { "keepalivenr" => "integer" }
}
geoip {
source => "clientip"
target => "geoip"
database => "/etc/logstash/GeoLiteCity.dat"
add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
useragent {
source => "useragent"
}
}
output {
elasticsearch { hosts => ["localhost:9200"] }
# stdout { codec => rubydebug }
}
}}}
%%
%%
Also first download a ["GeoLiteCity DB"|http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz] and unzip it to /etc/logstash.
Make the user logstash part of the {{adm}} group (so it can read logfiles) and restart: ''/etc/init.d/logstash restart'' and there we have an logstash-* index in elasticsearch with all requested fields, hurray !
!! Elasticsearch URLs
! Delete an index
{{{
curl -XDELETE 'http://localhost:9200/logstash-*/'
}}}
! Index a document
{{{
curl -XPUT 'http://localhost:9200/twitter/tweet/1' -d '{
"user" : "kimchy",
"post_date" : "2015-11-15T14:12:12",
"message" : "trying out Elasticsearch"
}'
}}}
Add and index a document to the twitter index, with documentid 1
This page (revision-13) was last changed on 23-Apr-2022 17:05 by HarryMetskeTop