One of the frequent requests we receive is for a simple way to upload or download files between Windows Azure Blob Storage and their local file system. We’re pleased to release AzCopy (Beta Version), which is a command line utility which allows Windows Azure Storage customers to do just that. The utility is designed to simplify the task of transferring data in to and out of a Windows Azure Storage account. Customers can use this as a standalone tool or incorporate this utility in an existing application. The utility can be downloaded from Github.
The command is analogous to other Microsoft file copy utilities like robocopy that you may already be familiar with. Below is the syntax:
AzCopy <Source> <Destination> [filepattern [filepattern…]] [Options]
In this post we highlight AzCopy’s features without going into implementation details. The help command (AzCopy /?) lists and briefly describes all available system commands and parameters.
Key Features:
- Efficient and Flexible: AzCopy allows users to selectively copy the data. By using recursive mode, customers can copy nested directories of files. Users can specify file pattern – wild cards in Windows file system or prefix in the blob storage to identify source files that are candidates for copying. In addition, users can set an option to copy files which have “Archive” attribute set. When copying a large number of files, and if there is a copy failure due to network or other issues, the re-startable option can resume the copy process from where it left off (the files which have not been copied). Using re-startable mode, you will not need to re-copy files which were successful in the previous attempt.
- Support for Windows Azure Storage Data Types: AzCopy provides options for customers to specify the destination data in a storage account as a block blob or a page blob. Default is set to block blobs, as this is best suited for nearly all files. When using the page blob option, the blob will be zero-padded to a 512-byte boundary.
- Naming Rules: URI format (http or https) is used to specify the blob storage path and NTFS Windows file folder path is used for Windows file system. Since some of the blob names cannot be directly translated into Windows file system, AzCopy translates into Windows file system names using its own rules. Here are some rules that we follow in naming translations:
- Blobs that conflict with Windows special file names will be renamed using the following rules: “.” => “dot”; “..” => “dotdot”; “/” => “slash”; etc. As with other conflict resolution rules, if a conflict occurs on any of these names the string “(n)” will be added to the conflicting file or blob name such that the conflict is resolved.
- The Windows file system namespace is case insensitive (but case retentive), and the Windows Azure blob namespace case sensitive, the following rules apply:
- Blobs in the blob namespace are created with the default case for the file name.
- Files are created with the default case of the blob namespace.
- If a case-conflict occurs while copying from the blob-namespace to the file-namespace, we will append the string “(n)” to the conflicting file or blob names.
- Logging: Users can run AzCopy in a verbose mode, which displays lists of files and directories processed, and render the list of files that the utility failed to copy. AzCopy also displays the progress indication for each file while running in verbose mode.
Examples:
Example 1: Copy a directory of locally accessible files to blob storage container in a recursive mode.
AzCopy C:\blob-data https://myaccount.blob.core.windows.net/mycontainer/ /destkey:key /S
The above command will copy all files from the “c:\blob-data” directory and all subdirectories as block blobs to the container named “mycontainer” in storage account “myaccount”. “Blob-data” folder contains the following files and one subdirectory named “subfolder1”;
C:\blob-data\car1.docx
C:\blob-data\car2.docx
C:\blob-data\car3.docx
C:\blob-data\train1.docx
C:\blob-data\subfolder1\car_sub1.docx
C:\blob-data\subfolder1\car_sub2.docx
After the copy operation, “mycontainer” blob container will contain the following blobs:
car1.docx
car2.docx
car3.docx
train1.docx
subfolder1/car_sub1.docx
subfolder1/car_sub2.docx
If we do not use recursive mode (Copying without the “/S” option), the “mycontainer” blob container would only contain the following files under “blob-data” folder and would ignore files under the “subfolder1” folder.
car1.docx
car2.docx
car3.docx
train1.docx
Example 2: Recursively copy a set of blobs from a blob storage to a locally accessible directory in both verbose and recursive modes.
AzCopy https://myaccount.blob.core.windows.net/mycontainer c:\blob-data /sourceKey:key /S /V
The command will copy all blobs under the “mycontainer” blob container in account “myaccount” to the “c:\blob-data” directory in both verbose and recursive modes.
“mycontainer” blob container contains the following files:
car1.docx
car2.docx
car3.docx
train1.docx
subfolder1/car_sub1.docx
subfolder1/car_sub2.docx
Since we are using the verbose mode, the tool will display the following output which contains the file transfer status of each of the file in addition to the transfer summary. By default, the tool will only display the transfer summary:
Finished Transfer: car1.docx
Finished Transfer: car2.docx
Finished Transfer: car3.docx
Finished Transfer: train1.docx
Finished Transfer: subfolder1/car_sub1.docx
Finished Transfer: subfolder1/car_sub2.docx
Transfer summary:
-----------------
Total files transferred: 6
Transfer successfully: 6
Transfer failed: 0
After the copy operation, c:\blob-data folder will contain the files listed below:
C:\blob-data\car1.docx
C:\blob-data\car2.docx
C:\blob-data\car3.docx
C:\blob-data\train1.docx
C:\blob-data\subfolder1\car_sub1.docx
C:\blob-data\subfolder1\car_sub2.docx
Let’s try a slightly different scenario by copying the blobs which start with “subfolder1\” by using the following command:
AzCopy https://myaccount.blob.core.windows.net/mycontainer/subfolder1 c:\blob-data /sourceKey:key /S /V
The above command will only copy blobs which begin with “subfolder1/”, and thus the tool will only copy “subfolder1/car_sub1.docx” and “subfolder1/car_sub2.docx” blobs to “c:\blob-data\” folder. After the copy operation, “C:\blob-data” will contain the following files:
C:\blob-data\car_sub1.docx
C:\blob-data\car_sub2.docx
Example 3: Copy a directory of locally accessible files to a blob account in re-startable mode
AzCopy c:\blob-data https://myaccount.blob.core.windows.net/mycontainer /destkey:key /Z:restart.log /S
Restart.log, a journal file, will be used to maintain a record of the status of the copy operation to allow the operation to restart if interrupted. If there is no text file specifies along with the re-startable mode parameter, the journal file will default to “azcopy.log” in the current working directory.
For instance, “C:\blob-data” folder contains the five large files with each of the file size greater than 100 MB.
C:\blob-data\car.docx
C:\blob-data\car1.docx
C:\blob-data\car2.docx
C:\blob-data\car3.docx
C:\blob-data\car4.docx
When running with restart option, AzCopy allows you to restart the process in the case of failure. If the failure occurred while copying “car.docx”, AzCopy will resume the copy from the part of “car.docx” which has not been copied. If the copy occurred after “car.docx” was successfully copied, AzCopy will resume the copy operation from one of the remaining four files which have yet to be copied.
Example 4: Select number of files in a storage blob container using a file pattern and copy them to a locally accessible directory.
AzCopy https://myaccount.blob.core.windows.net/mycontainer c:\blob-data car /sourceKey:key /Z /S
“mycontainer” contains the following files:
car1.docx
car2.docx
car3.docx
train.docx
carfolder/car_sub1.docx
carfolder/train_sub2.docx
subfolder1/car_sub1.docx
subfolder1/car_sub2.docx
After copy operation, “c:\blob-data” will contain the files listed below. Since the file pattern with the prefix of “car” was specified, the copy operation copies only the file with the prefix of “car”. Note that this prefix is applied to the blob, if it’s directly in the “mycontainer” container, or to the subdirectory name.
C:\blob-data\car1.docx
C:\blob-data\car2.docx
C:\blob-data\car3.docx
C:\blob-data\carfolder\car_sub1.docx
C:\blob-data\carfolder\train_sub2.docx
Performance
Within a Windows Azure datacenter (i.e., between a compute instance and a storage account within the same DC), users should be able to achieve 50MB/s upload and download speed for uploading and downloading large amounts of data using an extra-large compute instance. Transfers to and from a Windows Azure datacenter will be constrained by the bandwidth available to AzCopy.
Known Issues
- When copying without /XN (Exclude Newer) and /XO (Exclude Older), the tool only compares the names of source and target files before copying. Therefore, users will be prompted whether to overwrite the target files although source and target files are identical.
- When using /XN and /XO, note that your local system time and the time stored in the storage service will vary slightly. So if the blob and local file were modified at nearly the same time, this comparison may not filter correctly.
- When copying a file to a page blob with the re-startable option and if there is a copy operation failure in the middle of copying, the tool will restart the copy process from the beginning of the file. This issue does not apply to copying a file to a block blob.
- When copying blobs to local %systemdrive%, the tool will not prompt the confirm message to overwrite or not to overwrite the files if there are existing files with the same name.
- If there are two blobs named “a” and “a/b” under a storage container, copying the blobs under that container with /S will fail. Windows will not allow the creation of folder name “a” and file name “a” under the same folder.
Aung Oo
Matthew Hendel
Windows Azure Storage Team