Using Elasticsearch Snapshot with Minio

Data migration from elasticsearch to elasticsearch

and creating snapshot using MinIO

(MinIO-The secure alternative to Amazon S3 object storage)

?What is Minio

Minio is an open source distributed object storage server written in Go, designed for Private Cloud infrastructure providing S3 storage functionality. Minio is the best server which is suited for storing unstructured data such as photos, videos, log files, backups, and container. Size of an object can be range from a KBs to a maximum of 5TB.

Benefits of Minio

S3 API compatibility
Data Redundancy
High Availability
Horizontal and vertical scaling
Supports multiple Pluggable storage backend
Data security using encryption on both server and client side

Install MinIO

Add a minio user

useradd -s /sbin/nologin -d /opt/minio minio

Create directories

mkdir -p /opt/minio/bin
mkdir /opt/minio/data # this will be your data partition

Download minio server binary

yum install wget
wget https://dl.minio.io/server/minio/release/linux-amd64/minio -O /opt/minio/bin/minio
chmod +x /opt/minio/bin/minio

Configuration file

vi /opt/minio/minio.conf
MINIO_VOLUMES=/opt/minio/data

Ensure all files are owned by minio in /opt/minio

chown -R minio:minio /opt/minio

Add minio systemd service files

 :vi /etc/systemd/system/minio.service
[Unit]
Description=Minio
Documentation=https://docs.minio.io
Wants=network-online.target
After=network-online.target
AssertFileIsExecutable=/opt/minio/bin/minio
[Service]
WorkingDirectory=/opt/minio
User=minio
Group=minio
PermissionsStartOnly=true
EnvironmentFile=-/opt/minio/minio.conf
ExecStartPre=/bin/bash -c "[ -n \"${MINIO_VOLUMES}\" ] || echo \"Variable MINIO_VOLUMES not set in /opt/minio/minio.conf\""
ExecStart=/opt/minio/bin/minio server $MINIO_OPTS $MINIO_VOLUMES
StandardOutput=journal
StandardError=inherit
# Specifies the maximum file descriptor number that can be opened by this process
LimitNOFILE=65536
# Disable timeout logic and wait until process is stopped
TimeoutStopSec=0
# SIGTERM signal is used to stop Minio
KillSignal=SIGTERM
SendSIGKILL=no
SuccessExitStatus=0
[Install]
WantedBy=multi-user.target

Enable and start the minio service

systemctl enable minio
systemctl start minio

Check that minio has started

systemctl status minio

Default Access key & Secret Key

Access Key: minioadmin
Secret Key: minioadmin

Change Access Key & Secret Key

vi /opt/minio/data/.minio.sys/config/config.json
 Search for "access_key" and "secret_key"
"key":"access_key","value""
"key":"secret_key","value""
systemctl restart minio.service

You can access the Minio Web interface by going to http://server_ip:9000. Login to Web interface using the access & secret

S3 Repository Plugin

Installation (In both Source and Destination)

mkdir  /usr/share/elasticsearch/plugins/repository-s3
cd /usr/share/elasticsearch/plugins/repository-s3
wget https://artifacts.elastic.co/downloads/elasticsearch-plugins/repository-s3/repository-s3-7.14.0.zip
unzip repository-s3-7.4.2.zip
systemctl restart elasticsearch

Add Access Key and Secret Key (In both Source and Destination)

Access Key: minioadmin

     echo minioadmin | /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.access_key -f

Secret Key: minioadmin

echo 'minioadmin' | /usr/share/elasticsearch/bin/elasticsearch-keystore add s3.client.default.secret_key -f

curl -s -f -H "Content-Type: application/json"  -X POST http://localhost:9200/_nodes/reload_secure_settings -d '{"secure_settings_password": ""}'

Create bucket in minio

For example : elkbucket

Add Repository (In both Source and Destination)

"bucket" : BucketName

"endpoint" : Minio Server

Miniorepository : RepositoryName

      curl -s  -H "Content-Type: application/json"  \
           -X POST http://localhost:9200/_snapshot/miniorepository \
             -d '{ "type": "s3", "settings": {  "bucket": "elkbucket",  "client":"default",  "endpoint":"http://192.168.71.162:9000", "path_style_access":"true" }  }'

Check Repository in Kibana (In both Source and Destination)