A recent update for .NET 4.5 introduced a regression to HttpWebRequest that may affect high scale applications. This blog post will cover the details of this change, how it impacts clients, and mitigations clients may take to avoid this issue altogether.
What is the effect?
Client would observe long latencies for their Blob, Queue, and Table Storage requests and may find either that that their requests to storage are dispatched after a delay or it is not dispatching requests to storage and instead see System.Net.WebException being thrown from their application when trying to access storage. The details about the exception is explained below. Running a netstat as described in the next section would show that the process has consumed many ports causing port exhaustion.
Who is affected?
Any client that is accessing Windows Azure Storage from a .NET platform with KB2750149 or KB2805227 installed that does not consume the entire response stream will be affected. This includes clients that are accessing the REST API directly via HttpWebRequest and HttpClient, the Storage Client for Windows RT, as well as the .NET Storage Client Library (2.0.6.0 and below provided via NuGet, GitHub, and the SDK). You can read more about the specifics of this update here.
In many cases the Storage Client Libraries do not expect a body to be returned from the server based on the REST API and subsequently do not attempt to read the response stream. Under previous behavior this “empty” response consisting of a single 0 length chunk would have been automatically consumed by the .NET networking layer allowing the socket to be reused. To address this change proactively we have added a fix to the .NET Client library in version 2.0.6.1 to explicitly drain the response stream.
A client can use the netstat utility to check for processes that are holding many ports open in the TIME_WAIT or ESTABLISHED states by issuing a nestat –a –o ( The –a will show all connections, and the -o option will display the owner process ID).
Running this command on an affected machine shows the following:
You can see above that a single process with ID 3024 is holding numerous connections open to the server.
Description
Users installing the recent update (KB2750149 or KB2805227) will observe slightly different behavior when leveraging the HttpWebRequest to communicate with a server that returns a chunked encoded response. (For more on Chunked encoded data see here).
When a server responds to an HTTP request with a chunked encoded response the client may be unaware of the entire length of the body, and therefore will read the body in a series of chunks from the response stream. The response stream is terminated when the server sends a zero length “chunk” followed by a CRLF sequence (see the article above for more details). When the server responds with an empty body this entire payload will consists of a single zero-length chunk to terminate the stream.
Prior to this update the default behavior of the HttpWebRequest was to attempt to “drain” the response stream whenever the users closes the HttpWebResponse. If the request can successfully read the rest of the response then the socket may be reused by another request in the application and is subsequently returned back to the shared pool. However, if a request still contains unread data then the underlying socket will remain open for some period of time before being explicitly disposed. This behavior will not allow the socket to be reused by the shared pool causing additional performance degradation as each request will be required to establish a new socket connection with the service.
Client Observed Behavior
In some cases older versions of the Storage Client Library will not retrieve the response stream from the HttpWebRequest (i.e. PUT operations), and therefore will not drain it, even though data is not sent by the server. Clients with KB2750149 or KB2805227 installed that leverage these libraries may begin to encounter TCP/IP port exhaustion. When TCP/IP port exhaustion does occur a client will encounter the following Web and Socket Exceptions:
- or -
System.Net.WebException: Unable to connect to the remote server
System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted.
Note, if you are accessing storage via the Storage Client library these exceptions will be wrapped in a StorageException:
Microsoft.WindowsAzure.Storage.StorageException: Unable to connect to the remote server
System.Net.WebException: Unable to connect to the remote server
System.Net.Sockets.SocketException: Only one usage of each socket address (protocol/network address/port) is normally permitted
Mitigation
We have been working with the .NET team to address this issue. A permanent fix is now available which reinstates this read ahead semantic in a time bounded manner.
Install KB2846046 or .NET 4.5.1 Preview
Please consider installing the hotfix (KB2846046) from the .NET team to resolve this issue. However, please note that you need to contact Microsoft Customer Support Services to obtain the hotfix. For more information, please visit the corresponding KB article.
You can also install .NET 4.5.1 Preview that already contains this fix.
Upgrade to latest version of the Storage Client (2.0.6.1)
An update was made for the 2.0.6.1 (NuGet, GitHub) version of the Storage Client library to address this issue. If possible please upgrade your application to use the latest assembly.
Uninstall KB2750149 and KB2805227
We also recognize that some clients may be running applications that still utilize the 1.7 version of the storage client and may not be able to easily upgrade to the latest version without additional effort or install the hotfix. For such users, consider uninstalling the updates until the .NET team releases a publicly available fix for this issue. We will update this blog, once such fix is available.
Another alternative is to pin Guest OS for your Windows Azure cloud services as this prevents getting updates. This involves explicitly setting your OS to a version released before 2013.
More information on managing Guest OS updates can be found at Update the Windows Azure Guest OS from the Management Portal.
Update applications that leverage the REST API directly to explicitly drain the response stream
Any client application that directly references the Windows Azure REST API can be updated to explicitly retrieve the response stream from the HttpWebRequest via [Begin/End]GetResponseStream() and drain it manually i.e. by calling the Read or BeginRead methods until end of stream
Summary
We apologize for any inconvenience this may have caused. Please feel free to leave questions or comments below,
Joe Giardino, Serdar Ozler, Jean Ghanem, and Marcus Swenson
Resources
Windows Azure Storage Client library 2.0.6.1 (NuGet, GitHub)
Original KB article #1: http://support.microsoft.com/kb/2750149
Original KB article #2: http://support.microsoft.com/kb/2805227
Hotfix KB article: http://support.microsoft.com/kb/2846046
.NET 4.5.1 Preview: http://go.microsoft.com/fwlink/?LinkId=309499