Graylog Installation Record

[aws-pls-mongo1b] Shell record

$mkdir mongo

$sudo vi /etc/yum.repos.d/mongodb-org-3.6.repo file

[mongodb-org-3.6]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/amazon/2013.03/mongodb-org/3.6/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.6.asc

$sudo yum install -y mongodb-org

<ulimit settings for mongo>
$ sudo su
ulimit -f unlimited
ulimit -t unlimited
ulimit -v unlimited
ulimit -l unlimited
ulimit -n 64000
ulimit -m unlimited
<Verify>
ulimit -u 64000

reboot now

ulimit -a

$sudo service mongod status
$sudo service mongod start
$sudo service mongod status

$sudo chkconfig mongod on

$sudo vi  /etc/mongod.conf
# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0  # Listen to local interface only, comment to listen on all interfaces.

$sudo service mongod stop
$sudo service mongod start
$sudo service mongod status

# Verification and Passed
$ mongo --host 10.104.187.162:27017

mongo --host LB-PLS-BETA-MONGO-INT-77e51851111d7088.elb.us-west-2.amazonaws.com:27017

================================================

[aws-pls-elastic1b] Shell record


$ sudo yum install -y java
$ sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
$ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.5.4.rpm
$ sudo rpm --install elasticsearch-6.5.4.rpm
$ sudo systemctl start elasticsearch.service
$ sudo systemctl status elasticsearch.service

$ sudo vi /etc/elasticsearch/elasticsearch.yml
network.host: 0.0.0.0
http.port: 9200

$ sudo vi /etc/elasticsearch/jvm.options
-Xms4g
-Xmx4g

$ sudo systemctl restart elasticsearch.service
$ sudo systemctl status elasticsearch.service

# Verification and Passed
$ sudo curl "http://127.0.0.1:9200/_cat/nodes"
$ sudo curl "http://localhost:9200/_cat/nodes"
$ sudo curl "http://10.104.187.148:9200/_cat/nodes"
$ sudo curl -XGET 'http://localhost:9200/_cluster/health?pretty=true’
$ sudo curl -XGET 'http://127.0.0.1:9200/_cluster/health?pretty=true’
$ sudo curl -XGET "http://10.104.187.148:9200/_cluster/health?pretty=true"

[ec2-user@ip-10-104-187-162 ~]$ sudo curl "http://10.104.187.148:9200/_cat/nodes"
10.104.187.148 4 66 0 0.10 0.12 0.05 mdi * 4wlbKCJ

[ec2-user@ip-10-104-187-145 ~]$ sudo curl "http://10.104.187.148:9200/_cat/nodes"
10.104.187.148 3 67 1 0.02 0.05 0.01 mdi * 4wlbKCJ

[ec2-user@ip-10-104-187-145 ~]$ sudo curl "http://LB-PLS-BETA-ES-INT-78477d32e333724d.elb.us-west-2.amazonaws.com:9200/_cat/nodes"
10.104.187.148 3 67 1 0.02 0.05 0.01 mdi * 4wlbKCJ


[ec2-user@ip-10-104-187-145 ~]$ sudo curl "http://LB-PLS-BETA-ES-INT-78477d32e333724d.elb.us-west-2.amazonaws.com:9200/_cat/nodes"
10.104.187.148 5 67 0 0.00 0.00 0.00 mdi * 4wlbKCJ

sudo curl -XGET "http://LB-PLS-BETA-ES-INT-78477d32e333724d.elb.us-west-2.amazonaws.com:9200/_cluster/health?pretty=true"

================================================
<Passed>
telnet 10.104.187.148 9200
telnet 10.104.187.162 27017

<Passed>
LB-PLS-BETA-ES-INT-78477d32e333724d.elb.us-west-2.amazonaws.com
LB-PLS-BETA-MONGO-INT-77e51851111d7088.elb.us-west-2.amazonaws.com

telnet LB-PLS-BETA-ES-INT-78477d32e333724d.elb.us-west-2.amazonaws.com 9200
telnet LB-PLS-BETA-MONGO-INT-77e51851111d7088.elb.us-west-2.amazonaws.com 27017
sudo curl -XGET "http://LB-PLS-BETA-ES-INT-78477d32e333724d.elb.us-west-2.amazonaws.com:9200/_cluster/health?pretty=true"

[aws-pls-graylog1b] Shell record

$ sudo yum update
$ sudo yum install -y java
$ wget https://packages.graylog2.org/releases/graylog/graylog-2.5.1.tgz
             https://packages.graylog2.org/releases/graylog/graylog-2.5.1.tgz
             https://packages.graylog2.org/repo/packages/graylog-2.5-repository_latest.rpm
            https://packages.graylog2.org/repo/packages/graylog-2.5-repository_latest.rpm

$ tar xvfz graylog-2.5.1.tgz

$ sudo mkdir /etc/graylog
$ sudo mkdir /etc/graylog/server/

$ sudo cp ~/graylog-2.5.1/graylog.conf.example /etc/graylog/server/server.conf
$ sudo vi /etc/graylog/server/server.conf

<1>*
password_secret = sKzW2vDDkqOQTrKC
root_password_sha2 = 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
elasticsearch_shards = 1
elasticsearch_hosts = http://LB-PLS-BETA-ES-INT-78477d32e333724d.elb.us-west-2.amazonaws.com:9200
mongodb_uri = mongodb://LB-PLS-BETA-MONGO-INT-77e51851111d7088.elb.us-west-2.amazonaws.com/graylog
rest_listen_uri = http://10.104.187.145:9000/api/
web_listen_uri = http://10.104.187.145:9000/
=>
rest_listen_uri = http://0.0.0.0:9000/api/
web_listen_uri = http://0.0.0.0:9000/

telnet 10.104.187.148 9200
telnet 10.104.187.162 27017
<2>
password_secret = sKzW2vDDkqOQTrKC
root_password_sha2 = 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
elasticsearch_shards = 4
elasticsearch_hosts = http://10.104.187.148:9200
mongodb_uri = mongodb://10.104.187.162/graylog
rest_listen_uri = http://10.104.187.145:9000/api/

===================================================================
Configure at least the following variables in /etc/graylog/server/server.conf:

***
# List of Elasticsearch hosts Graylog should connect to.
# Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
# If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that
# requires authentication.
#
# Default: http://127.0.0.1:9200
#elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200


# MongoDB connection string
# See https://docs.mongodb.com/manual/reference/connection-string/ for details
<add>
mongodb_uri = mongodb://LB-PLS-BETA-MONGO-INT-77e51851111d7088.elb.us-west-2.amazonaws.com/graylog

# Authenticate against the MongoDB server
#mongodb_uri = mongodb://grayloguser:secret@localhost:27017/graylog

# Use a replica set instead of a single host
#mongodb_uri = mongodb://grayloguser:secret@localhost:27017,localhost:27018,localhost:27019/graylog
***

root_password_sha2 = 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
elasticsearch_shards = 1
elasticsearch_hosts = http://LB-PLS-BETA-ES-INT-78477d32e333724d.elb.us-west-2.amazonaws.com:9200
mongodb_uri = mongodb://LB-PLS-BETA-MONGO-INT-77e51851111d7088.elb.us-west-2.amazonaws.com/graylog

#New ES
elasticsearch_hosts = https://vpc-pls-log-nonprod-jql2okbojnqn5npwbkbo5qjsgy.us-west-2.es.amazonaws.com


[ec2-user@ip-10-104-187-145 ~]$ echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1
Enter Password: admin
8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
===================================================================

Starting the server

$ sudo ./bin/graylogctl start
Starting graylog-server ...

$ sudo ./bin/graylogctl status
graylog-server running with PID 4175

$ sudo ./bin/graylogctl stop
Stopping graylog-server (4175) ...
Waiting for graylog-server to halt.
graylog-server stopped



$ tail ./log/graylog-server.log

Verification
http://10.104.187.145:9000
http://lb-pls-beta-graylog-int-51246890d683260e.elb.us-west-2.amazonaws.com:9000/gettingstarted

echo `date` | nc 10.104.187.145 12201
echo `date` | nc graylog-test.bapls.net 12201

http://graylog-test.bapls.net:9000

graylog-test.bapls.net


===========================================================
Sending in log data
http://docs.graylog.org/en/2.5/pages/sending_data.html

**Sending GELF messages via HTTP using curl
curl -XPOST http://10.104.187.145:12202/gelf -p0 -d '{"short_message":"Hello there IP0124", "host":"tw-pls-ta-dock", "facility":"test", "_foo":"bar"}'

curl -XPOST http://graylog-test.bapls.net:12202/gelf -p0 -d '{"short_message":"Hello there FQDN0124", "host":"tw-pls-ta-dock", "facility":"test", "_foo":"bar"}'

curl -XPOST http://LB-PLS-BETA-GRAYLOG-INT-51246890d683260e.elb.us-west-2.amazonaws.com:12202/gelf -p0 -d '{"short_message":"Hello there FQDN0124", "host":"tw-pls-ta-dock", "facility":"test", "_foo":"bar"}'

**Sending GELF messages via TCP using netcat
echo -n -e '{ "version": "1.1", "host": "tw-pls-ta-dock2", "short_message": "A short message by IP0124", "level": 5, "_some_info": "foo" }'"\0" | nc -w0 10.104.187.145 12201

echo -n -e '{ "version": "1.1", "host": "tw-pls-ta-dock2", "short_message": "A short message by FQDN0124", "level": 5, "_some_info": "foo" }'"\0" | nc -w0 graylog-test.bapls.net 12201

echo -n -e '{ "version": "1.1", "host": "tw-pls-ta-dock2", "short_message": "A short message by FQDN0124", "level": 5, "_some_info": "foo" }'"\0" | nc -w0 LB-PLS-BETA-GRAYLOG-INT-51246890d683260e.elb.us-west-2.amazonaws.com 12201




FQDN seems not works stable??

Testing Telnet 12201 port

max@tw-pls-ta-dock2:~$ telnet 10.104.187.145 12201
Trying 10.104.187.145...
Connected to 10.104.187.145.
Escape character is '^]'.

max@tw-pls-ta-dock2:~$ telnet graylog-test.bapls.net 12201
Trying 10.104.187.171...
Connected to lb-pls-beta-graylog-int-51246890d683260e.elb.us-west-2.amazonaws.com.
Escape character is '^]'.


Test curl http post
max@tw-pls-ta-dock2:~$ curl -XPOST http://10.104.187.145:12202/gelf -p0 -d '{"short_message":"Hello there IP1", "host":"example.org", "facility":"test", "_foo":"bar"}'
max@tw-pls-ta-dock2:~$ curl -XPOST http://graylog-test.bapls.net:12202/gelf -p0 -d '{"short_message":"Hello there FQDN1", "host":"example.org", "facility":"test", "_foo":"bar"}'


curl -XPOST http://graylog-test.bapls.net:12202/gelf -p0 -d '{"short_message":"Hello there FQDN1", "host":"example.org", "facility":"test", "_foo":"bar"}'

curl -X POST -u "ext_maxm:03e1ef098a609c3718cd3e10322a3acf" -H "Jenkins-Crumb:a32e3694b05057ca7942ae3d8d692693" http://tw-pls-jenk1.client.tw.trendnet.org:8080/view/Operation/view/2_Stage/job/DRI_PVT_STG_aws-pls-dris1s/buildWithParameters?token=thisismysecret&cause=trigger+by+shavlik+POC


Invoke-RestMethod -Uri http://graylog-test.bapls.net:12202/gelf -Method POST '{"short_message":"Hello there FQDN1", "host":"example.org", "facility":"test", "_foo":"bar"}'

Invoke-RestMethod -Uri http://10.104.187.145:12202/gelf -Method GET

$Url = "http://graylog-test.bapls.net:9000"
Invoke-RestMethod -Uri $url -Method Get


Test curl http
curl http://10.104.187.145:12202
<No Message Retured>
curl http://graylog-test.bapls.net:9000
<Some Retured>

==========================================

curl -o certificaterequest.csv --user certreport:Cert#Report9  'http://siteaddress/Reports/CertificateReport?starttime=2014-02-01T00:00:00&endtime=2014-05-01T00:00:00'


$cred = Get-Credential #will prompt you to enter userame and password
$csv="d:\output.csv"
$url="url"
Invoke-RestMethod -Uri $url -OutFile $csv -Credential $cred

curl -XPOST http://10.104.187.145:12202/gelf -p0 -d '{"short_message":"Hello there IP1", "host":"example.org", "facility":"test", "_foo":"bar"}'
echo -n -e '{ "version": "1.1", "host": "example.orgIP", "short_message": "A short message", "level": 5, "_some_info": "foo" }'"\0" | nc -w0 10.104.187.145 12201

$Url = "http://10.104.187.145:9000"
Invoke-RestMethod -Uri $url -Method Get

$Url = "http://10.104.187.145:12202"
$body= "Test"
Invoke-RestMethod -Uri $url -Method Post -Body $body -ContentType 'application/json' -Headers $hdrs

NXLog Installation

nxlog-ce-2.10.2150

• NXLog Manager
• NXLog Community Edition: Send data to most popular solutions. Need to ship data to ELK, Graylog, Loggly or some other SIEM? We got you covered!
• NXLog Enterprise Edition

安裝的是 NXLog Community Edition，安裝好之後我們要去修改設定檔，
設定檔是 C:\Program Files (x86)\nxlog\conf\nxlog.conf，內容如下。

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input iislog> 我們先給 Input 標籤一個自定義的名字 iislog，我們要在標籤裡面去設定一組輸入的來源。
    Module      im_file Module：設定 為im_file
    File      "C:\\inetpub\\logs\\LogFiles\\W3SVC1\\u_ex*" File：設定 IIS Log 檔案的儲存位置
    SavePos      TRUE SavePos：設定為 TRUE，用來記住上次的讀檔位置。
</Input>

<Output logstash> 給 Output 檔籤一個自定義的名字 logstash，我們在標籤裡面去設定一組輸出的目的。
    Module      om_tcp Module：設定 為om_tcp
    Host        IP or hostname  Host：設定為 Logstash or GrayLog 伺服器的名稱或位址
    Port        12201 Port：設定為 Logstash 伺服器開啟監聽的埠號
</Output>

<Route 1>
    Path        iislog => logstash
</Route>

<Route>
Route 是告訴 NXLog 哪一個 Input 要對應到哪一個 Output，Input 及 Output 可以用逗號隔開設定多組，例如這樣：

<Route 1>多對多
    Path        iislog1,iislog2,… => logstash1,logstash2,…
</Route>

但是如果我們想指定 iislog1 只輸出給 logstash1、iislog2 則輸出給 logstash1 及 logstash2，這個時候就要設定第二組 Route 去另外指定，像這樣：

<Route 1>一對一
    Path        iislog1 => logstash1
</Route>

<Route 2>一對多
    Path        iislog2 => logstash1,logstash2
</Route>

Configuration file:
C:\Program Files (x86)\nxlog\conf\nxlog.conf

Log file
C:\Program Files (x86)\nxlog\data\nxlog.log

Refer [料理佳餚] ELK 搭檔 NXLog 收集 IIS Log