ELK
This page explains how to set up a central syslog server on OS X using ELK.
There is a lot of documentation out there on the ELK stack, Elasticsearch, Logstash, and Kibana. The ELK stack is very powerful and can do many different things. However, I am only interested in a central syslog server on OS X.
For this setup I used OS X 10.9, Elasticsearch 1.3.4, Logstash 1.4.2 and Kibana 3.1.1. I also run it all on one computer, the server. The only other computers are the clients, and there is only one thing you need to do on them and it's easy so I'll actually explain that first.
Then I describe how to download and get everything running but not as daemons. After you have it all working then I explain how to daemonize everything.
Configure syslog on the Clients
OS X (syslog)
To have OS X computers send log information to your server, edit /etc/syslog.conf and add this line to the end. Replace "10.0.0.1" with the IP of your server.
*.* @10.0.0.1:5000
Then tell syslogd to reread the config files.
sudo killall -HUP syslogd
That's it.
Note, I'm using port 5000 instead of the default 514. The reason I'm using port 5000 is so that I can daemonize Logstash and have it run as an unprivileged user, which means that the port has to be greater than 1023.
Windows and Linux (rsyslog, syslog-ng)
You can also have Windows, Linux, and even network hardware send their logs to your server.
Linux might have syslog, rsyslog or syslog-ng, look in /etc to see what config files you've got. You configure rsyslog exactly how you configure syslog except edit /etc/rsyslog.conf. To configure syslog-ng, edit /etc/syslog-ng.conf and find a line that starts with "source", something like "source s_sys {...}". Then add these lines to the end of that file, and replace "s_sys" with the name above, usually something like "s_sys", "src", "s_all", or "s_local".
destination d_elk {
udp("10.0.0.1" port(5000));
};
# Replace "s_sys"
log {
source(s_sys);
destination(d_elk);
};
All of the rest of the instructions are for the server.
Download and install everything
Go to www.elasticsearch.org and download Elasticsearch, Logstash, and Kibana, decompress. I saved them to /usr/local/elk/ and so all of the paths on this page refer to that path, but you can put them anywhere really, just make sure you've got the correct paths listed in all of your files.
Install Java. You can use Oracle's installer, or you can run Elasticsearch (Open Terminal and cd to the elasticsearch dir and type ./bin/elasticsearch) and the OS will tell you what to do.
Elasticsearch
Once you've got it downloaded, open Terminal and type these commands.
Run it
/usr/local/elk/elasticsearch/bin/elasticsearch
Test it
curl 'http://localhost:9200/_search?pretty'
You should see something like this.
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 0,
"successful" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : 0.0,
"hits" : [ ]
}
}
Wow that was easy.
Unless you get a crash like the text below. If that happens, make sure you installed a version of Java that includes "JDK" in the name. There are actually several versions of Java.
Exception in thread "main" java.lang.UnsupportedClassVersionError: org/elasticsearch/bootstrap/Elasticsearch : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:637)
at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLClassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Logstash
Configure it
The following code is from Getting started with Logstash (I changed it a little). Save the text in a file, it doesn't matter where you put it. I saved it to /usr/local/elk/elk.conf. Note, again, I'm using port 5000. You can choose whatever port you like, just configure your clients to use what you specify here.
input {
udp {
port => 5000
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch { host => localhost }
}
Run it
/usr/local/elk/logstash/bin/logstash -f /usr/local/elk/elk.conf
Test it
Note, I use the word "Supertastic" in my telnet test because if you've setup your clients already, then they are probably already saving messages to your server and I wanted to use a word that probably wont be in any log message.
telnet localhost 5000
Supertastic
To exit telnet, type cntl-] then quit. Search Elasticsearch to see if it worked. Note, it might take a second or two for the data to show up in Elasticsearch, so if it isn't there, run the command a few times and see if it eventually shows up.
curl 'http://localhost:9200/_search?q=Supertastic&pretty'
You should see this.
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 3.5194955,
"hits" : [ {
"_index" : "logstash-2014.11.05",
"_type" : "syslog",
"_id" : "jSuTg_-_T3ON3R19QQurQw",
"_score" : 3.5194955,
"_source":{"message":"Supertastic\r","@version":"1","@timestamp":"2014-11-05T00:30:35.230Z","host":"10.0.0.1:54888","type":"syslog"}
} ]
}
}
If it worked you now have Elasticsearch and Logstash running. If your setup isn't working you've probably got bigger problems because everything up to now is suppose to be super easy. If you need to debug more, try following the instructions on Getting started with Logstash because it shows how to run Logstash without Elasticsearch, which might help you figure out where the problem is.
Test clients
If you haven't setup your clients, now might be a good time to do so (instructions are at the top). You can see if your clients are working by running the same command as above but without the q=Supertastic part.
curl 'http://localhost:9200/_search?pretty'
If you get a lot of hits, well, your clients are saving log messages. A syslog hit looks like this (yes, there is irony in this real message):
{
"_index" : "logstash-2014.11.04",
"_type" : "syslog",
"_id" : "eRjVdiNpQq6aXN5XDTUZ1g",
"_score" : 1.0,
"_source":{"message":"<45>Nov 4 16:26:57 example syslogd[27]: Configuration Notice:\n\tASL Module \"com.apple.securityd\" claims selected messages.\n\tThose messages may not appear in standard system log files or in the ASL database.\n","@version":"1","@timestamp":"2014-11-04T23:27:05.688Z","type":"syslog","host":"10.0.0.2"}
},
If the telnet test from above worked but you don't get anything from your clients, try going to a client and run logger Dangit and search for that using this.
curl 'http://localhost:9200/_search?q=Dangit&pretty'
If it still doesn't work, you can rule out network or firewall problems by configuring the server's /etc/syslog.conf to send messages to Logstash.
Now let's add Kibana to the mix.
Start a webserver
Kibana requires a web server. The webserver doesn't have to be the server running Elasticsearch or Logstash, but later on when I secure the system I only discuss securing Kibana when it is running on the same server as Elasticsearch and Logstash.
Use OS X Server, the Web Sharing System Preference Pane, run sudo apachectl start, or install and run your favorite webserver. Describing more about this is beyond the scope of these instructions.
You can also use Elasticsearch as a webserver if you'd like. Just put the Kibana folder at /usr/local/elk/elasticsearch/plugins/kibana/_site (so all of the files including index.html are inside of "_site").
Kibana
Move the Kibana folder into your web directory. On OS X Server that is at /Library/Server/Web/Data/Sites/Default/. If you use the Web Sharing Preference Pane or start apache manually it's /Library/WebServer/Documents/. I'm assuming the folder name is "kibana", not "kibana-3.1.1" or something.
Open http://localhost/kibana/index.html and you should see an intro page. Click the link to Logstash and you should see a graph with bars.
If you are using Elasticsearch as your webserver, the URL is: http://localhost:9200/_plugin/kibana/
If you put in a lot of test data into Elasticsearch while setting it up, you can clean out the Elasticsearch database by running this.
curl -XDELETE 'http://localhost:9200/_all/_query?q=*'
Note, Kibana doesn't have any daemons or CGI's running on your webserver. Kibana is JavaScript that runs in the web browser and so the web browser is connecting to the server over port 9200. To be specific, the web browser is running javascript that is using this url to access your data.
'http://localhost:9200/logstash/_search?pretty'
To verify this, in the top right of every Kibana viewport there is a round icon with an "i" in it. Click that and you will see the exact URL that Kibana is using to show the current viewport.
Next I will talk about how to secure this so that only you can view your data.
Limit access
I want to block the ports that we've opened so far, 9200 and 5000.
I'm using ipfw here because it's easiest. However it's deprecated and you should use pf. When I figure out how to use pf I'll update this page. For now (10.9, I haven't tested 10.10), this works.
To block port 9200 (Elasticsearch).
sysctl net.inet.ip.fw.enable=1
ipfw add deny ip from any to any 9200 in
Note, the services running on the server can still connect to the port. This means that when Logstash runs on the same computer as Elasticsearch they can still connect.
It also means that if you load the Kibana website on the same machine using http://localhost/kibana/ you can connect. But you can't open Kibana from any other computer. That is, it will read the Kibana files, but it won't show any data.
If you want to read Kibana from other computers you'll have to poke holes for those IP's or setup a reverse proxy. The following commands will poke a hole. Replace 10.0.0.1/24 with the CIDR that meets your needs.
ipfw -f flush
ipfw add allow ip from 10.0.0.1/24 to any 9200 in
ipfw add deny ip from any to any 9200 in
Now I want to limit who can save log messages to the syslog server. First, I want to poke a hole for the allowed clients, then I want to block everything else.
ipfw add allow ip from 10.0.0.1/24 to any 5000 in
ipfw add deny ip from any to any 5000 in
Daemonize everything
Create elk user
To make this as secure as possible, I want to create an unprivileged user named Elk, and all it does is run Elasticsearch and Logstash, it can't login. This is so that if there are security vulnerabilities in any of the ELK daemons, then the damage possible is limited to what the Elk user can do (at least it's not root). I basically duplicated and modified the www user plist file. I used the uid/gid of 400 because it's greater than the system users (none of the system users/groups were above 300), less than the users created by System Preferences (those start at 501), and less than the users created by OS X Server (those start at the 1000).
Just putting these 2 files in place does not give life to the user. I probably could've restarted DirectoryServices to get the system to recognize the new users. But I wanted to restart after I had all of the following files created to see if it was working at startup. So at the end of this section just reboot.
/var/db/dslocal/nodes/Default/users/_elk.plist is:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>generateduid</key>
<array>
<string>FFFFEEEE-DDDD-CCCC-BBBB-AAAA00000190</string>
</array>
<key>gid</key>
<array>
<string>400</string>
</array>
<key>home</key>
<array>
<string>/usr/local/elk</string>
</array>
<key>name</key>
<array>
<string>_elk</string>
<string>elk</string>
</array>
<key>passwd</key>
<array>
<string>*</string>
</array>
<key>realname</key>
<array>
<string>ELK</string>
</array>
<key>shell</key>
<array>
<string>/usr/bin/false</string>
</array>
<key>uid</key>
<array>
<string>400</string>
</array>
</dict>
</plist>
/var/db/dslocal/nodes/Default/groups/_elk.plist is:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>generateduid</key>
<array>
<string>ABCDEFAB-CDEF-ABCD-EFAB-CDEF00000190</string>
</array>
<key>gid</key>
<array>
<string>400</string>
</array>
<key>groupmembers</key>
<array/>
<key>name</key>
<array>
<string>_elk</string>
<string>elk</string>
</array>
<key>passwd</key>
<array>
<string>*</string>
</array>
<key>realname</key>
<array>
<string>ELK</string>
</array>
<key>users</key>
<array/>
</dict>
</plist>
Be sure to run chown -R 400:400 /usr/local/elk.
These two LaunchDaemons will start everything up at startup.
/Library/LaunchDaemons/org.elasticsearch.elasticsearch.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>org.elasticsearch.elasticsearch</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/elk/elasticsearch/bin/elasticsearch</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<key>UserName</key>
<string>_elk</string>
</dict>
</plist>
/Library/LaunchDaemons/net.logstash.logstash.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>net.logstash.logstash</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/elk/logstash/bin/logstash</string>
<string>-f</string>
<string>/usr/local/elk/elk.conf</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<true/>
<key>UserName</key>
<string>_elk</string>
</dict>
</plist>
Also make sure your Logstash config file (/usr/local/elk/elk.conf) does not have stdin or stdout as an input or ouptut.
The very last piece is to start the firewall at startup. This is the absolutely simplest firewall startup script ever. There are better scripts out there. Modify this to meet your needs.
/usr/local/bin/firewall.sh
#!/bin/sh
sysctl net.inet.ip.fw.enable=1
ipfw -f flush
ipfw add allow ip from 10.0.0.1/24 to any 9200 in
ipfw add deny ip from any to any 9200 in
ipfw add allow ip from 10.0.0.1/24 to any 5000 in
ipfw add deny ip from any to any 5000 in
Be sure the script is executable (chmod 755 /usr/local/bin/firewall.sh).
/Library/LaunchDaemons/firewall.plist
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>firewall</string>
<key>ProgramArguments</key>
<array>
<string>/usr/local/bin/firewall.sh</string>
</array>
<key>RunAtLoad</key>
<true/>
<key>KeepAlive</key>
<false/>
</dict>
</plist>
Restart your server then run this command to make sure Elasticsearch and Logstash are running as daemons.
ps axj | grep elk
You should see something like this (note, the user running the processes is _elk).
_elk 11454 1 11454 0 0 Ss ?? 2:51.24 /usr/bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Delasticsearch -Des.foreground=yes -Des.path.home=/usr/local/elk/elasticsearch -cp :/usr/local/elk/elasticsearch/lib/elasticsearch-1.3.4.jar:/usr/local/elk/elasticsearch/lib/*:/usr/local/elk/elasticsearch/lib/sigar/* org.elasticsearch.bootstrap.Elasticsearch
_elk 11456 1 11456 0 0 Ss ?? 1:12.41 /usr/bin/java -Xmx500m -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -jar /usr/local/elk/logstash/vendor/jar/jruby-complete-1.7.11.jar -I/usr/local/elk/logstash/lib /usr/local/elk/logstash/lib/logstash/runner.rb agent -f /usr/local/elk/elk.conf
root 15698 9396 15697 0 2 S+ s000 0:00.00 grep elk
Check your Kibana interface to make sure that is still getting data.
Now all that is left is figuring out how to use Kibana.
Read more
- http://logstash.net/docs/1.4.2/tutorials/getting-started-with-logstash
- http://krypted.com/windows-server/stashbox-turning-a-mac-mini-into-a-logstash-server/
- http://krypted.com/mac-security/redirect-logs-to-a-syslog-server-in-os-x/
- http://krypted.com/windows-server/use-syslog-on-windows
Published: 2014-11-05, last edited: 2020-05-11
Copyright © 2024 James Reynolds