Install Splunk on Linux – Complete Setup Guide

We have many guides on securing common web hosting solutions: cPanel servers, bare cloud servers, and even the popular WordPress content management system (CMS). Each includes great technical controls for a proactive approach to defense in depth. However, nothing can detect and prevent everything. The only way to address this is to apply administrative controls, primarily log auditing.

Audit log management, also known as security information management (SIM), is more than website analytics such as page visits, bounce rate, and referral URLs. It includes:

• Uptime
• Account-specific actions
• File change management
• Login attempts and failures
• Port and other reconnaissance scans
• Incoming and outgoing network traffic

The processes for auditing system-wide information and events are known respectively as security information management (SIM) and security event management (SEM). Combining the two types of information is known as security information and event management (SIEM).

Server logs are oftentimes neglected if not natively accessible from a graphical user interface (GUI). Proactively scanning these logs can help you understand your residual risk to cyber attacks not prevented by current technical controls and how to strengthen your security stance.

Below we’ll cover:

• What is Splunk?
• Installing Splunk
  • CentOS
  • Debian/Ubuntu
    • Change Default Debian Shell
• Setting up Splunk
• Logging into Splunk
• Resetting Splunk admin credentials
  • Splunk username
• Monitoring data in Splunk
• Splunk apps and installation
  • Within dashboard
  • Manually

What is Splunk?

Splunk Enterprise is a SIEM application that gathers, organizes, and visualizes machine-generated log data from local and remote machines, websites, and cloud services. Having Splunk setup with your physical and cloud systems can be valuable for staying ahead of cybersecurity, connectivity, and other big data initiatives.

How to Install Splunk Enterprise

Install Splunk Enterprise on CentOS

1. Create an account on on Splunk.com.
2. Select Free Splunk in the upper-right corner.
3. Select Free Splunk.
4. Select Linux, then Download Now beside .rpm.
5. Upload the file to your server.
6. SSH into your server as root.
7. Install the Splunk Enterprise RPM file:

   rpm -i path-to-file/splunk-versionnumber.rpm

8. Continue to Splunk setup.

Install Splunk Enterprise on Debian/Ubuntu

1. Create an account on on Splunk.com.
2. Select Free Splunk in the upper-right corner.
3. Select Free Splunk.
4. Select Linux, then Download Now beside .deb.
5. Upload the file to your server with SCP, replacing the filename, username, and server hostname as needed:

   scp splunk-versionnumber.deb [email protected]:/root

6. SSH into your server as root.
7. Install the Splunk Enterprise DEB file:

   dpkg -i splunk-file.deb

8. Verify Splunk installation status:

   dpkg --status splunk

9. Continue to Splunk setup.

Change Your Default Shell

Splunk recommends using bash as your default shell as Debian’s default shell, dash, may cause zombie processes which cannot be killed. Below we’ll cover how to change your default Debian shell.

1. Find your default shell:

   which sh

2. You should see /bin/sh or another symbolic link. Use ls to find the actual shell:

   ls -l /bin/sh

3. If it doesn’t show bash at the end, view installed shells to ensure it is installed:

   cat /etc/shells

4. Delete the symbolic link:

   rm /bin/sh

5. Create a new symbolic link pointing /bin/sh to bash:

   ln -s /bin/bash /bin/sh

Complete Splunk Setup

After you install Splunk, follow the steps below to complete your Splunk setup.

1. Use Splunk to start the Splunk service:

   /opt/splunk/bin/splunk start

2. Read the license agreement. At the end, select y and Enter.
3. Create an username.
4. Create a password with at least eight characters.
5. Once Splunk installation is complete, the last line will provide the URL to access the web interface: http://serverhostname:8000.
6. Open port 8000 in your firewall: Firewalld, UFW, CSF, etc.

   Keep in mind that if you get locked out of your server and restart it, you’ll need to start the Splunk service again before you can access the Splunk dashboard.

7. Open your Splunk web interface in your browser.

How to Log into the Splunk Dashboard

There are multiple options for logging into your Splunk dashboard depending on your server configuration. What matters most is that you use :8000 at the end.

How to Reset Your Splunk Administrator Password

There’s no “Forgot password” link on the native Splunk login page. You’ll need to edit the Splunk passwd file.

1. SSH into your server.
2. Navigate to your Splunk /etc directory (e.g. cd /opt/splunk/etc). You can use the find command if needed.

   find / -iname splunk | grep etc

   Don’t edit a file in the /virtfs directory.

3. Rename the file passwd to something else:

   mv passwd passwd.backup20201030

4. Navigate to the Splunk system/local directory:

   cd ../system/local

5. Create and edit a new file:

   nano user-seed.conf

6. Inside the new file, add the following with a new username and password:
   [user_info]
USERNAME = admin
PASSWORD = C0mp!c@T3Dp@s$w0RD
7. Save your file.
8. Restart Splunk to create a new passwd file:

   /opt/splunk/bin/splunk restart

9. Log into your Splunk web interface with the new username and password.

Change Splunk Admin Username

If you want to change your admin username, possibly because you’ve noticed brute force login attacks for predictable usernames, follow the steps below to better secure your Splunk setup.

1. Edit your Splunk passwd file:

   nano /opt/splunk/etc/passwd

2. Edit the username at the beginning of the file.
3. Save your changes.
4. Restart Splunk:

   /opt/splunk/bin/splunk restart 

5. Log into Splunk.

Monitor Data in Splunk

Below we’ll cover how to add your first log source into your Splunk setup.

1. Log into your Splunk web interface.
2. Select Add Data.
3. At the bottom, select Monitor.
4. On the left of the Select Source page, select Files & Directories.
5. Select Browse.
6. Specify a file or directory to monitor and click Select. For example, you can monitor a cPanel log, Apache access log (similar to GoAccess Analytics), or a cPanel user directory. We’ll use the /var/log/secure log file which tracks SSH logins, and authentication failures on CentOS. Debian/Ubuntu users will instead use /var/log/auth.log.
7. Select Continuously Monitor to show updates to the log file in real-time.
8. At the top, select Next to access the Set Source Type page.
9. Source type on the left should state “linux_secure” so Splunk knows it is Linux security information. Otherwise, select the button and select linux_secure from the drop-down menu.
10. Select Next at the top.
11. (Optional) On the Input Settings page, you can change the App context, machine hostname, and Index.
12. Select Review.
13. Ensure everything is correct and select Submit.
14. You’ll see “File input has been created successfully.” Start Searching.
15. To modify the event results, select Settings and Data Inputs from the upper-right corner.
16. Select the data input type on the left. For this example, we’ll select Files & Directories.
17. Select Enable or Disable for the data path in the Status column.

Splunk Apps and Add-ons

You can customize your Splunk setup with a massive database of apps and add-ons for better data analysis for your specific server environment. You can install Splunk apps from Splunkbase.Splunk.com/apps or directly within your Splunk dashboard.

From your Splunk dashboard, select the gear icon on the left beside Apps. On other pages, select Apps and Manage Apps from the top-left of the page. From here you can:

• Disable and enable installed apps
• Modify permissions
• Update settings
• Launch apps

At the top are links to browse installable Splunk apps, install apps manually, and create an app.

Install Apps in Splunk Dashboard

1. From the Splunk homepage, select + Find More Apps on the left.
2. On the left, search for an app.
3. Select the Install button for the app you wish to install. We’ll use the Website Monitoring Splunk app for our example.
4. Provide your Splunk.com user credentials, then accept the terms and conditions.
5. Select Login and Install.
6. If notified “Restart Required,” select Restart Now. Click OK once the restart is successful.
7. Log back into your Splunk dashboard.
8. Select Apps at the top or return to the homepage to see the new app available on the left.

Manually Install Splunk Apps

1. Visit https://Splunkbase.Splunk.com.
2. At the top, search for an app.
3. Select an app. We’ll use the Splunk Add-on for Cisco ASA app as an example.

   The Cisco ASA hardware firewall is available with our Dedicated Server Hosting plans.

4. On the right, click Download, or Login to Download if applicable.
5. Accept the license agreements and click Agree to Download.
6. Save the file to your computer.
7. Verify the checksum of your downloaded file against the provided message digest. For example:
   Windows:

   certutil -hashfile SplunkFile.tgz sha256; echo ProvidedChecksum

   
   Mac:

   shasum -a 256 SplunkFile.tgz && echo ProvidedChecksum

   
   *nix:

   sha256sum SplunkFile.tgz && echo ProvidedChecksum 

8. Once you’ve verified the checksums match, press OK. If not, troubleshoot why the checksums differ before continuing to ensure you don’t upload a corrupted or malicious file.
9. On the Splunkbase site, select the Details tab for additional installation information.
10. In your Splunk web interface, go to your Apps page.
11. Select Install app from file.
12. Browse your local machine and select the compressed Splunk app file.
13. Click Upload. If successful, you’ll see “[App] was installed successfully” and it’s already enabled.

Learn more about how to get the most out of your SIEM software with official Splunk documentation. Or learn more about free cybersecurity applications and how to stay updated on industry news.

Learn more from our Cloud Server Hosting Product Guide.