Overview
Diagnosing and troubleshooting issues in cloud storage applications can be complex especially if they are not considered upfront. When an issue does occur, it can involve parsing and analyzing multiple log files (Azure Storage Analytics, client-side logs from Azure Storage Client libraries, and network traces) to fully understand and mitigate the issue. To assist with this, we have released a set of Azure Storage specific assets in Microsoft Message Analyzer, including parsers, color rules, charts, filters, and view layouts. This blog provides a brief overview, but for a complete hands-on tutorial see End-to-End Troubleshooting using Azure Storage Metrics and Logging, AzCopy, and Message Analyzer.
Sample Scenario – Searching Storage Logs for Storage Service Versions
For this blog post, we’ll examine a scenario where the customer wants to determine what REST versions their client applications are currently using, so they can prepare (if needed) for the planned Azure Storage Service Removal of several of the earlier service versions.
Assumption is you have Message Analyzer with the Azure Storage Assets installed and Azure storage logging and metrics enabled – for info on that see the complete hands-on tutorial.
Step 1: Download the Azure Storage server-side logs using AzCopy
AzCopy is available for download on the Azure Downloads page. For details about using AzCopy, see How to use AzCopy with Microsoft Azure Storage. As an example, the following command will download the log files for blob operations that took place on January 2, 2015 to the folder C:\Temp\Logs\Server; replace <storageaccountname> with the name of your storage account, and <storageaccountkey> with your account access key:
AzCopy.exe /Source:http://<storageaccountname>.blob.core.windows.net/$logs /Dest:C:\Temp\Logs\Server /Pattern:"blob/2015/01/02" /SourceKey:<storageaccountkey> /S /V
NOTE: It can take up to an hour for log data to become available because of the frequency at which the storage service flushes the log writers.
Step 2: Import your server-side logs into Message Analyzer
- On the File menu in Microsoft Message Analyzer, click New Session > Files> Add Files to browse to the location where you downloaded your server-side logs. Select your server-side logs and click on the Open button.
- In the Session Details panel, set the Text Log Configuration drop-down for each server-side log file to AzureStorageLog (if not already set) to ensure that Microsoft Message Analyzer can parse the log file correctly, and click on the Start button.
Step 3: Add the RequestVersionHeader column to the Analysis Grid
In Microsoft Message Analyzer, under Column Chooser > Azure Storage Log > Azure Storage Log Entry, right click on RequestVersionHeader and select “Add as Column” to make it visible in the Analysis Grid.
Step 4: Search for earlier REST versions
Add the following session filter to determine if there are any requests using an Azure Storage service version that is scheduled to be removed:
AzureStorageLog.RequestVersionHeader < "2012-02-12"
You can double click on each row (if any) in the Analysis Grid to get more information on the individual request. You can also read the “What Should I do?” section on our blog.
Performance Tip: Note that Message Analyzer loads log files into memory. If you have a large set of log data, you will want to apply a session filter before you load the data, in order to get the best performance from Message Analyzer.
Summary
In the above scenario we demonstrated how Message Analyzer along with AzCopy can be used to identify clients that are still using an old REST version. You can use the same combination of tools for your own debugging and analysis when working with Azure Storage.
Next Steps
Follow the complete hands-on tutorial here for more advance scenarios including correlating storage, network, and client side logs to troubleshoot performance issues.
For more information visit the following resources:
• E2E Troubleshooting using Azure Storage Metrics and Logging, AzCopy, and Message Analyzer
• Monitor, diagnose, and troubleshoot Microsoft Azure Storage
• Microsoft Azure Storage Service Version Removal
Perry Skountrianos
Program Manager, Azure Storage