The best cloud log management services allows you to monitor, process, analyze, and visualize logs via the cloud in a simple and manageable way.
This is important, as managing log events can be a challenge with cloud computing services for a number of reasons. Firstly, there is the sheer volume of data points but also complicating matters are the number of applications involved, plus reverse engineering identified errors to their source for correction.
Cloud logging services aim to simplify managing everything by providing an interface through which logs are routed, so the data they contain can be more easily sorted and subjected to reporting and analytics for information.
By using the cloud it means that monitoring remains independent of your network, so it should not be slowed by any errors or failures. Additionally, cloud storage makes it easier to save any historical data points you may need to reference later.
By themselves these should be enough reasons to make log management essential to work with, but additional ones include the ability to integrate it not just with other cloud services, but also useful business intelligence and data visualization tools for additional insights and communicating them to stakeholders.
Here then are the best cloud logging and management services currently available.
- We've also highlighted the best cloud brokers.
Azure Monitor is Microsoft's cloud-based tool for providing a comprehensive way to monitor applications, network, and infrastructure. As you'd expect, it works with other Azure services to provide a focused overview from data and logs collected from a range of services, whether based in Azure or on-premises.
It's built to work especially well with Azure Security Center and Azure Automation, to push logs into Azure Monitor for analysis and visualization. You can set up alerts for specific events, as well as use extensive querying to work through data quickly and efficiently, working through multiple sources from tables, datasets, as well as logs.
Smart analytics and machine learning algorithms mean you can detect, respond to, and correct problems quickly and easily, as well as use them to mine for actionable insights.
As you'd expect with a platform as powerful as Azure, there's a lot of potential under the hood, but it will take experience and some time to work most effectively with it.
AWS Centralized Logging uses AWS Elasticsearch and Kibana to collect data and provide analytics and visualizations accordingly. Working in conjunction with other AWS tools it offers the ability to customize not just for single users but for multiple accounts for analyzing logs from their AWS applications and services.
AWS Centralized Logging can also work outside of AWS service logs and can be used for any host-level log files and VPC flow logs. Designed to scale with your business, pre-configured dashboards can also be used to simplify data visualization for stakeholders.
As above, this is a service for experience users, and those new to AWs should expect something of a learning curve to get the most out of the service.
Google Cloud Logging is the log management solution provided with Google Cloud that allows for the storing, searching, analysis and monitoring of log data not just on the Google Cloud platform. The API can also be used from other external sources, such as on-premises servers as well as other cloud services such as AWS.
As with other cloud services, Google Cloud Logging is a scales according to need, and can be used to ingest log data from thousands of virtual machines in real time. This means you can identify and resolve issues on the go using other Google Cloud services such as Monitoring, Trace, Error Reporting, and Debugger.
On top of working through logs, integration with these services mean you can also navigate through incidents, charts, and traces, to identity and correct issues in your infrastructure and applications.
A fully-managed solution, Google Cloud Logging allows you to run applications in multi-cloud environments from a single platform and dashboard, simplifying the process of log management.
As ever with cloud services, it may take a little time for even experienced cloud administrators to get the most of out the Google Cloud Logging platform's potential.
Rapid7 InsightOps (formerly Logentries) aims to make cloud logging management easy by centralizing data, allowing easy visualization of metrics for monitoring, and using natural-language questions to identify problems for troubleshooting.
InsightOps collects logs across all cloud services, as well as directly from servers, such as for applications, DNS servers, VPNs, databases, and firewalls. It then normalizes even unstructured data into a consistent format. You can then monitor how much CPU and memory each asset is using to ensure smooth performance. Auditing logs makes it easier to ensure compliance, and InsightOps can easily integrate into your DevOps stack for automation.
There are visualization options for drilling down through data, and the Endpoint Interrogator allows the user to ask specific questions about what resources each asset is using. There are also alerts available, and the dashboard refreshes automatically in real-time.
Overall, InsightOps is easy to deploy across any network, and because it monitors the entire system, it's easy to spot malicious activity on any part of the network, even at file level. InsightOps is available with a free trial.
Solarwinds is a company that develops software for businesses to help manage their networks, systems and information technology infrastructure. They acquired the log management service ‘Papertrail’ in 2014.
Papertrail allows users to log all apps, text log files and Syslog from one location. Searches and reports are done in real-time. Users can use a command line, browser or API. Alerts are sent instantly to users which can then be used to detect trends.
Papertrail is easy to setup and implement. Users can gain visibility across all their systems quickly. Users can aggregate MySQL, cloud hosting, Apache, Tomcat, Heroku apps, routers and firewalls.
The lowest pricing tier includes 1-week search and 1-year archive. The mid-range tier starts and includes an 8 weeks search and 1-year archive, as well as larger log options. The top tier increases the available limits further.
Also consider these log management services
Log analysis has become an increasing major part of IT operations, especially with regards to cybersecurity concerns on top of existing IT maintenance and management ones. Although we've featured the ones we think are the best above, it's a crowded market with a number of quality vendors, and there are others worth considering such as:
Splunk offers a wide range of log management tools such as log consolidation, retention, IT operations troubleshooting, application troubleshooting and compliance reporting. Users can collect, store, index, search, correlate, visualize and report on any machine-generated data. Subscribers can then resolve operational and security issues.
Sumologic provides real time dashboards to manage and monitor all data logs, metrics, assess graphs and generate reports from here. Dashboards can be customized with pie, bar, map and line charts. Sumologic’s machine learning uses algorithmic support to arrange the patterns in data and establish any inconsistencies, and can work with AWS, Google Cloud or any hybrid applications.
Sematext Logsense uses Elasticsearch API and integrated Kibana for log analysis and visualization. Alerts can be sent via email, ChatOps or WebHooks. Logsense has built-in integrations for PagerDuty, Slack and HipChat amongst others, and queries can be saved with custom tags, labels, or colors.
ManageEngine EventLog Analyzer aims to bring log management, auditing, and compliance together as part of a Security Information and Event Management (SIEM) solution. With the focus on security, this makes it easier to gain insights on potential attack routes and secure them before they become a problem. This means as well as log management it can be used to augment threat intelligence.
- We've also featured the best cloud orchestration services.