Configuring and Managing Elastic Filebeat on CentOS for Monitoring, Collecting, and Forwarding Logs to a Centralized Repository like Elasticsearch

Elastic Filebeat is a lightweight log shipper that enables monitoring, collecting, and forwarding log files to a centralized repository such as Elasticsearch. This article focuses on configuring and managing Elastic Filebeat on the CentOS operating system to efficiently handle log data. We'll delve into the installation process, configuration, basic management of Filebeat, as well as integration with Elasticsearch for centralized logging.

Installing Elastic Filebeat on CentOS

1. Adding the Elastic repository Firstly, add the Elastic repository to your system. Create a new repository file for YUM:

   sudo tee /etc/yum.repos.d/elastic.repo<<EOF
   [elastic-7.x]
   name=Elastic repository for 7.x packages
   baseurl=https://artifacts.elastic.co/packages/7.x/yum
   gpgcheck=1
   gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
   enabled=1
   autorefresh=1
   type=rpm-md
   EOF
   

2. Installing Filebeat After adding the repository, you can install Filebeat using YUM:

   sudo yum install filebeat
   

   Once the installation is complete, you can start and enable Filebeat to automatically start on system boot:

   sudo systemctl start filebeat
   sudo systemctl enable filebeat
   

Configuring Filebeat for Sending Logs to Elasticsearch

1. Editing the Filebeat configuration file The Filebeat configuration file is located at /etc/filebeat/filebeat.yml. Open this file in a text editor and make the necessary changes:

   sudo nano /etc/filebeat/filebeat.yml
   

2. Setting the paths to logs In the configuration file, specify the paths to the log files you want to monitor. For example:

   filebeat.inputs:
   - type: log
     enabled: true
     paths:
       - /var/log/*.log
   ​

3. Configuring output to Elasticsearch Next, set up Filebeat output to send logs to Elasticsearch. Enter the address of your Elasticsearch instance:

   output.elasticsearch:
     hosts: ["localhost:9200"]
   

 

Running and Verifying Filebeat Functionality

1. Applying configuration changes After making changes in the configuration file, restart the Filebeat service to apply the new configuration:

   sudo systemctl restart filebeat
   

2. Verifying log transmission You can verify that Filebeat successfully communicates with Elasticsearch and sends logs using the filebeat test output command:

   filebeat test output
   ​

 

Management and Maintenance

To ensure smooth operation of Filebeat and effective monitoring of logs, it's important to regularly check the service status, perform updates, and monitor performance. Monitoring Filebeat and Elasticsearch logs will help identify potential issues and ensure optimal performance of your logging system.

Updating and Maintenance

• Updating Filebeat and Elasticsearch: Keep your Filebeat and Elasticsearch installations up to date to ensure they are secure and benefit from the latest features. Use the YUM package manager to update the software.
• Backing up configuration: Regularly back up configuration files and other important data. In case of any issues, you'll be able to quickly restore your configuration or settings.

Integration with Kibana for Log Visualization

Elasticsearch, along with Kibana, provides a powerful tool for log visualization and analysis. After successfully configuring Filebeat and sending logs to Elasticsearch:

1. Setting up Kibana: Ensure you have Kibana properly configured to connect to your Elasticsearch instance.
2. Creating an index pattern: In Kibana, create an index pattern that matches the indices created by Filebeat (typically filebeat-*).
3. Browsing and analyzing logs: Using Kibana, you can browse logs, create visualizations, and dashboards to monitor key metrics and events from your logging data.

Securing Log Transmission

In environments where security is crucial, consider encrypting communication between Filebeat and Elasticsearch using SSL/TLS. This involves generating and configuring SSL certificates for Elasticsearch and configuring Filebeat to use these certificates for secure connections.

• SSL Configuration on Elasticsearch side: Modify Elasticsearch configuration to enable SSL communication and upload the appropriate certificates.
• Configuring Filebeat to use SSL: Modify the filebeat.yml configuration file to reference your SSL certificates and keys for secure connection to Elasticsearch.

 

Effective configuration and management of Elastic Filebeat on CentOS for monitoring, collecting, and forwarding logs to Elasticsearch is crucial for securing, performance, and availability of your systems. Regular maintenance, updates, and monitoring of your logging infrastructure will ensure you have access to important information for analysis and troubleshooting in your IT infrastructure.