Introducing Microsoft Azure File Service

We are excited to announce the preview of the Microsoft Azure File service. The Azure File service exposes file shares using the standard SMB 2.1 protocol. Applications running in Azure can now easily share files between VMs using standard and familiar file system APIs like ReadFile and WriteFile. In addition, the files can also be accessed at the same time via a REST interface, which opens a variety of hybrid scenarios. Finally, Azure Files is built on the same technology as the Blob, Table, and Queue Services, which means Azure Files is able to leverage the existing availability, durability, scalability, and geo redundancy that is built into our platform.

Scenarios

• “Lift and Shift” applications

Azure Files makes it easier to “lift and shift” applications to the cloud that use on-premise file shares to share data between parts of the application. To make this happen, each VM connects to the file share (see “Getting Started” below) and then it can read and write files just like it would against an on-premise file share.

• Shared Application Settings

A common pattern for distributed applications is to have configuration files in a centralized location where they can be accessed from many different virtual machines. Such configuration files can now be stored in an Azure File share, and read by all application instances. These settings can also be managed via the REST interface, which allows worldwide access to the configuration files.

• Diagnostic Share

An Azure File share can also be used to save diagnostic files like logs, metrics, and crash dumps. Having these available through both the SMB and REST interface allows applications to build or leverage a variety of analysis tools for processing and analyzing the diagnostic data.

• Dev/Test/Debug

When developers or administrators are working on virtual machines in the cloud, they often need a set of tools or utilities. Installing and distributing these utilities on each virtual machine where they are needed can be a time consuming exercise. With Azure Files, a developer or administrator can store their favorite tools on a file share, which can be easily connected to from any virtual machine.

Getting started

• Step 1: Sign up for service

To sign up, go to the Microsoft Azure Preview Portal, and sign up for the Microsoft Azure Files service using one or more of your subscriptions. As subscriptions are approved for the Azure File preview, you will get an email notifying you of the approval. We will be slowly opening up the service to users in batches, so please be patient after signing up.

• Step 2: Create a new storage account

Once you get the approval notification, you can then go to the Microsoft Azure Management Portal and create a new storage account using any of the approved subscriptions. Whenever a new storage account is created for an approved subscription, it will automatically get a file endpoint provisioned. The file endpoint for the account will be: <account name>.file.core.windows.net.

Note, During the Azure File preview, existing storage accounts do not have access to Azure Files. You have to create a new storage account to access Azure Files.

• Step 3: Create the File Share

You can now create a file share in either of the following two ways:

       ▪ PowerShell CmdLets

Once you download the PowerShell Cmdlets, you can start PowerShell prompt and execute the following:

# go to the directory and you may need to unblock the cmdlet using the following command:

unblock-file .\Microsoft.WindowsAzure.Commands.Storage.File.dll     

# import module and create a context for account and key

import-module .\AzureStorageFile.psd1

$ctx=New-AzureStorageContext <account name><account key>

# create a new share

$s = New-AzureStorageShare <share name> -Context $ctx

# create a directory in the test share just created

New-AzureStorageDirectory -Share $s -Path testdir

# upload a local file to the testdir directory just created

Set-AzureStorageFileContent -Share $s -Source D:\upload\testfile.txt -Path testdir

# list out the files and subdirectories in a directory

Get-AzureStorageFile -Share $s -Path testdir

# download files from azure storage file service

Get-AzureStorageFileContent -Share $s -Path testdir/testfile.txt -Destination D:\download

# remove files from azure storage file service

Remove-AzureStorageFile -Share $s -Path testdir/testfile.txt 

       ▪ REST APIs

You can also create a file share programmatically using the ‘Create Share’ REST API with version 2014-02-14 REST APIs are available via http(s)://<account name>.file.core.windows.net Uri. In addition, the .NET Storage Client Library starting with 4.0 version uses the REST version 2014-02-14 and supports Azure Files. We recommend adding a NuGet reference to your project. Here is an abstract of code that creates a share:

staticvoid Main(string[] args)

{

     CloudStorageAccount account = CloudStorageAccount.Parse(cxnString);

     CloudFileClient client = account.CreateCloudFileClient();

     CloudFileShare share = client.GetShareReference("bar");

     share.CreateIfNotExistsAsync().Wait();

}

• Step 4: Use File Share

Once your share is created, it can be accessed via the SMB or REST protocol from any Azure node (VM/Worker/Web role) hosted in the same region as the storage account hosting the share.

To use the share via SMB 2.1 protocol, log into the VM that requires access to the share and execute “net use” as shown below:

    net use z: \\<account name>.file.core.windows.net\<share name> /u:<account name> <account key>

C:\>net use z: \\myaccount.file.core.windows.net\myshare /u:myaccount StgAccKey==
The command completed successfully.

C:\>dir z:\
  Volume in drive Z has no label.
  Volume Serial Number is 4038-F841

  Directory of z:\

  08/06/2013 10:51 AM 15 HelloFromWindosAzureFileService.txt
  1 File(s) 15 bytes
  0 Dir(s) 109,951,162,777,600 bytes free

C:\>type z:\HelloFromWindowsAzureFileService.txt
Hello World

To remove the mapping of the share from our VM, use: net use z: /delete.

How can I move my data to Azure Files

Azure Files is a separate service from Azure Blobs. In preview, we do not support copying Azure Blobs to Azure Files. In addition, the Microsoft Azure Import/Export Service does not support Azure Files for preview. Both these are in the roadmap and we will share more details in the future.

For preview, the below are the list of options available to transfer files from on premise to Azure File service.

net use z: \\<account name>.file.core.windows.net\<share name> /u:<account name> <account key>

AzCopy

AzCopy version 2.4 now supports moving your local files to Azure files and back. Once your data is in Azure Files, you can easily access this file share from any cloud VM within the same region via SMB protocol.

Assuming you have created a share called “myfileshare” in your file service, here are some examples:

• Upload files from your local disk recursively to Azure Files, with all folder structures copied as well.

             AzCopy d:\test\ https://myaccount.file.core.windows.net/myfileshare/ /DestKey:key /s

             Note: empty folders will not be uploaded.

• Upload files with certain file pattern from your local disk to Azure Files.

             AzCopy d:\test\ https://myaccount.file.core.windows.net/myfileshare/ /DestKey:key ab* /s

• Download all files in the file share recursively to local disk, with all folder structures copied as well.

              AzCopy https://myaccount.file.core.windows.net/myfileshare/ d:\test\ /SourceKey:key /s

             Note: empty folders will not be downloaded.

• Download one file from the file share to your local disk

             AzCopy https://myaccount.file.core.windows.net/myfileshare/myfolder1/ d:\test\ /SourceKey:key abc.txt

            Note: when specifying the file pattern ‘abc.txt’, AzCopy will try to find exactly the same name file under ‘myfileshare/myfolder1’, not including its subfolders.

The below scenarios are NOT supported in AzCopy during preview.

• Copying files between Azure Blobs and Azure Files, as well as between Azure File shares.
• Reading Azure File data from Geo Redundant Account’s Secondary Region.
• Using Share Access Signature (SAS) to connect with Azure Files.

Storage Client Library 4.0 (or REST APIs version 2014-02-14)

Storage Client library 4.0 supports files and you can write a quick application to upload files into the service.

Remote desktop to the VM which has mapped the share

Using file explorer: There are two ways to copy files using file explorer.

        a. Copy and paste files from your local machine into the file share in explorer in a remote session.

        b. Using xcopy with local drives shared in a remote session

                 i. Remote desktop into an Azure VM and opt to choose the local drive to be accessible in remote session as shown in the below picture.

                 ii. Create a mapping to the Azure file share using net use as we mentioned above.

                 iii. Use xcopy:

                       xcopy \\tsclient\c\software\demo\data\* \\<accountname>.file.core.windows.net\myshare\data\.

Interaction between SMB and REST Interfaces

As mentioned above, an Azure File share can be accessed via two interfaces at the same time – SMB 2.1 or REST.

When a REST operation is performed on a file that is concurrently open on an SMB client, the REST operation must respect the share mode specified by the SMB client.

This section briefly describes how the two interfaces interact but more details can be found in the MSDN documentation.

The following SMB file access mode is used to determine whether the REST operation can be completed:

NOTE: List Files (REST API), Get File Properties (REST API), and Get File Metadata (REST API) do not operate on SMB file content and do not require read access to the file (i.e. these operations will still succeed even if an SMB client has the file open for exclusive read access).

Here is an example of a sharing violation for REST Get File:

• An SMB Client opens a file with FileShare.Write (but not FileShare.Read which would mean other clients are not allowed to read the file at the same time).
• A REST Client then performs a Get File (REST API) operation on the file (thereby using FileAccess.Read as specified in the table above).
• Result: The REST Client’s request fails with status code 409 (Conflict) and error code SharingViolation since SMB Client still has the file open while denying Read/Delete access).

When to use Azure Files vs Azure Blobs vs Azure Disks

We have three main ways to store data in the Azure cloud – Files, Blobs and Disks. All three abstractions take advantage of Azure Storage stack and the Azure platform. The following section compares the advantages of each abstraction.

Azure Files: Provides a SMB 2.1 interface in addition to the REST interface to provide access to files. The SMB interface enables applications running in the cloud to use native file system APIs to read and write from files. A file share is best for:

• Lifting & shifting applications to the cloud which already use native file system APIs to share data between pieces of the applications
• Storing development and debugging tools that need to be accessed from many cloud instances
• Applications that want REST access to data from anywhere and SMB access to data from within the region the storage account is located in

Azure Blobs: Provides a REST interface for massively scale out object storage.

• Applications can be written to use REST APIs to store unstructured data into Azure blobs at a massive scale (a single container can be 500TBs in size). .
• Supports streaming scenarios and random access
• Access to data from anywhere via REST

Azure Disks: Provides persistent disks to be attached to Azure virtual machine in which data is made durable by storing the disk as a fixed formatted VHD in a page blob in Azure storage. When used as a VHD, the disk can be attached to a single VM, but not shared across VM instances.

• Lift & Shift applications that use native file system APIs to read and write data to persistent disks, but does not require this data to be available to other cloud VMs. Note that a single disk can be attached to only a single VM at any given time.
• Individual data stored on the disk is not required to be accessed from outside the VM as the data is stored in a VHD formatted with NTFS. However, the VHD can be downloaded from anywhere via REST API and then loaded onto a local virtual machine.
• Disks can be snapshotted to create point in time read-only backups. When recovery needs to take place, a snapshot can then be copied to a different blob and data can be recovered.

When accessing Azure Files via SMB 2.1, the share is available to VMswithin the same region as the storage account. The REST interface on other hand is available from everywhere. This new service enables customers to “lift and shift” legacy applications that rely on file system APIs to the cloud without code change.

The following table compares Azure Files with Azure Blobs. The most significant difference is that Azure Files can be accessed via standard file system APIs from VMs (e.g, Azure Files has true directories and supports file and directory rename). Both Files and Blobs provide a REST APIs that allows global access to the data they store. Another important difference is that Azure Blobs scale massively, up allowing up to 500 TB per container, whereas Azure Files limits capacity to 5 TB per share. Finally, Azure Blobs supports the “copy blob”, and “snapshots” APIs, while the preview of Azure Files does not support those APIs.

Azure Files complement Azure Disks, which can be attached to Azure VMs. A disk can be attached only to a single VM at any given time. Azure disks are fixed format VHDs stored as page blobs in Azure Storage and are used by the VM to persist the data on the disks. However, a disk is an NTFS formatted VHD (or other file system format) and thus it can be accessed from only a single VM. Also, disks do not have a REST interface that understands the VHD or file system format, and thus you cannot access the data (files) within VHD from outside the VM via REST. File shares based on Azure Files can be access from an Azure VM in the same way they access the local disk. In addition, the file share can be shared across many VMs as well as accessible through REST APIs. The below table highlights some of the differences between Azure Disks and Files.

Characteristics and Compatibility of the Azure File service

The following are the preview scalability targets for Azure Files

• Up to 5TB per share
• A file can be up to 1 TB
• Up to 1000 IOPS (of size 8KB) per share
• Up to 60MBps per share of data transfer for large IOs

The version of SMB protocol supported is SMB 2.1, which is available on Windows clients using Windows 7 or up, Windows Server 2008 R2 or up. For Linux, Ubuntu images available on the Azure IaaS images gallery are known to support SMB 2.1, however any other distribution with SMB2.1 support should also work.

Pricing

Pricing for the new SMB service is here. During preview, the capacity will be charged at 50% of the GA price.

SMB Compatibility

The SMB 2.1 protocol has many features; some of them are not applicable to the cloud (e.g, named pipes, etc), while others are used so rarely that we do not support them (alternative data streams). Please see here for the list of these features not supported by Azure Storage Files.

FAQ

1. Is SMB 3.0 supported?

We currently support SMB 2.1. We do have it in our list of feature requests to support SMB 3.0 but we do not have a timeline to share yet. We found that SMB 2.1 worked well with most popular systems and tools.

2. Is Active Directory based authentication supported?

We currently do not support AD based authentication or ACLs but do have it in our list of feature requests to support it. For now, the Azure Storage account keys are used to provide authentication and authorized access to the file share.

3. Would a File Share be accessible from outside the region that hosts the storage account?

SMB 2.1 protocol is available only from within the same region as storage account. REST APIs are available for accessing these files concurrently from anywhere.

4. How do I make the share visible to my VM?

You map (or mount) the share as you would any other SMB share in your OS. For Windows, you can use the “net use” command on the command-line, or use the File Explorer to mount a share. The storage account name is the username, and the password is the storage account key:

net use * \\myaccountname.file.core.windows.net\<sharename> /u:myaccountname StorageAccountKeyEndingIn==

5. Can I mount a share from Linux?

Yes. Today, it looks like only the latest 2 Ubuntu images in the portal can support Azure Files. To mount the share from linux, you first need to install some client tools. One choice is cifs-utils. This is the command from Ubuntu to install cifs-utils:

     sudo apt-get install cifs-utils

        Next, you need to make a mount point (mkdir mymountpoint), and then issue the mount command that is similar to this:

     sudo mount -t cifs //myaccountname.file.core.windows.net/mysharename ./mymountpoint -o vers=2.1,username=myaccountname,password=StorageAccountKeyEndingIn==,dir_mode=0777,file_mode=0777

        You can also add settings in your /etc/fstab to mount the share.

6. How do I access the data on the share?

Once the share is mounted, accessing it is no different than accessing any file on your local hard drive. Just use the standard file system APIs to create/open/read/modify/delete the files or directories you need.

7. How do I install the PowerShell Cmdlets for Files?

To work with Azure file service in PowerShell, you first need to download and install the preview module following these steps:

1. Download the zip format package and unzip it.
2. Copy the AzureStorageFile folder to %UserProfile%\Documents\WindowsPowerShell\Modules. You need to create this path if it doesn’t exist yet.

     Note:  You could also put this module folder in any other directory but need to make sure the directory path is added to $env: PSModulePath variable. Find more information here.

1. Double-click the shortcut to open up a PowerShell console with AzureStorageFile module automatically loaded. You could also manually import the module by using import-module cmdlet as shown above in a sample.

     Note:  AzureStorageFile module is in preview and developed on a new version of storage client library which is different from Microsoft azure module. Please import them in different PowerShell sessions. These will be unified in a future SDK release.

8. Does the new emulator support Azure Files?

The new emulator does not support SMB or REST APIs for Azure Files.

Summary and Links

To summarize, we are very excited to announce a new file sharing PaaS service that enables our users to easily share data between application instances using standard file system APIs. As we always do, we would love to hear feedback via comments on this blog, MSDN forum, stackoverflow, or send email to mastoragequestions@microsoft.com.

Please see these links for more information:

Azure Files 2014-04-14 version

AzCopy

Azure File PowerShell Cmdlets (CTP)

Storage .NET Client Library 4.0 for 2014-04-14 version

Brad Calder, Andrew Edwards, Jai Haridas, Atul Sikaria and Jean Ghanem