In this post, these status codes have to do with ASP.NET Core running on IIS. ASP.NET Core uses a module to direct the request to Kestrel which can be referred to as a proxy. When there is a timeout in the ASP.NET Core application it will return a 408 to the IIS proxy. An HTTP status code of 408 is a Request Timeout, it is designed after RFC 2068 and RFC 2616. The description of a 408 is “the client did not produce a request within the time that the server was prepared to wait. The client MAY repeat the request without modifications at a later time.” If you wanted to update the requestTimeout for an ASP.NET Core application you can use the property named UseSetting from within the CreateHostBuilder() method of the Program class inside the Program.cs file. The code snippet would look something like this.
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.UseStartup<Startup>();
webBuilder.UseSetting("requestTimeout", "00:03:00");
webBuilder.UseSetting("startupTimeLimit", "180");
webBuilder.UseSetting("stdoutLogEnabled", "true");
webBuilder.UseSetting("stdoutLogFile", @"\\?\%home%\LogFiles\stdout");
});
You can see the list of all ASP.NET Core attributes here Attributes of the aspNetCore element
Don’t get confused with the KestrelServerLimits.KeepAliveTimeout which I found mentioned in a few places as a means to extend the timeout.
webBuilder.ConfigureKestrel(options =>
{
options.Limits.KeepAliveTimeout = TimeSpan.FromMinutes(3);
});
Although the property KeepAliveTimeout includes “Timeout” it has to do with TCP Keep-Alive connections and not HTTP requests. I wrote an article about Keep-Alive here some time ago, perhaps this might be of interest if you want to pursue the Keep-Alive and TCP Session area a bit more. The same goes for IIS connectionTimeout show here, which is also associated to Keep-Alive.
Back to the original topic now. You will find that the requestTimeout property has a default value of 2 minutes (00:02:00) and when that threshold is breached a 408 is returned to IIS. If you wanted to increase the timeout value to more than 2 minutes, for some reason, you can do this as shown earlier. I will interject a personal opinion that if you have a request that takes over 2 minutes, you need to optimize this by decoupling or performing the process offline. Just because you can increase the timeout, doesn’t necessarily mean you should. You also do want to look out for long running outbound HTTP connections, this can also cause timeouts if the remote system doesn’t respond in time or the CPU utilization of the machine on which the code is running.
HTTP also has a default connection limit timeout of 2 minutes as well, as per the NETSH HTTP SHOW TIMEOUT shown in Figure 1.
Figure 1, HTTP 2 minute request timeout
You wouldn’t be able to change this on an App Service, but if you are running on a VM or on-premise you can, but again, just because you can increase the timeout, doesn’t mean you should. Since both ASP.NET Core and HTTP and IIS have the same or very similar default timeouts of 2 minutes leads me to believe there might exist a race condition of some kind when it comes to determining which status code actually gets returned to client. Which one gets there first, the 408 or the 502.3? The point is that if IIS and/or HTTP are configured to timeout before the ASP.NET Core application does, then a 502 gets returned to the client instead of the 408. The timeout is specifically a 502.3. HTTP sub status codes are super important because a 502 in itself is relatively ambiguous. A textbook definition of a 502 is “the server, while acting as a gateway or proxy, received an invalid response from the upstream server it accessed in attempting to fulfill the request.” Which is true because IIS here is simply a proxy which sent/forwarded the request using the logic existing within the AspNetCoreModule module. The 502.3 can be described as “CGI error / gateway timedout error”, however I have never seen anything officially stating that.
A 502.5 I have seen described as “failed to start process” or “Process Failure” which reads very close to 500.37 which has a description of “ANCM Failed to Start Within Startup Time Limit”. This 500.37 fails after 120000 milliseconds which is 2 minutes as well. I wrote this article some years ago about 502.5 which does still match the symptoms, I.e. the process couldn’t start, the solution however doesn’t apply so well.
If I were going to troubleshoot any any of these errors, I would use this approach, so long as the issue is reproducible. The fact that in all cases, you typically have a 2 minute time window to trigger the request and then capture the dump while the slowness is happening makes the approach feasible. The mentioned debugging approach is focused on the out of process dotnet core application, if you are running in-process now then you simply need the W3WP process. In-process is now the default. There are actually a lot of differences when running in-process versus out-of-process. You can use the AspNetCoreHostingModel described here to make you dotnet application run OutOfProcess.
In addition to HTTP status code and HTTP sub status codes there are also win32 status error codes. Three common win32 status codes in this scenario are 12029, 12030 and 1292.
- ERROR_WINHTTP_CANNOT_CONNECT 12029 Returned if connection to the server failed.
- ERROR_WINHTTP_CONNECTION_ERROR 12030 The connection with the server has been reset or terminated, or an incompatible SSL protocol was encountered. For example, WinHTTP version 5.1 does not support SSL2 unless the client specifically enables it.
- ERROR_IMPLEMENTATION_LIMIT 1292 (0x50C) An operation attempted to exceed an implementation-defined limit.
You can sometimes also find out what those mean using net helpmsg, as seen in Figure 2.
Figure 2, using NET HELPMSG to find sc-win32-status code descriptions
I have also seen a 502.8 which I’ve seen happen when TLS 1.0 is disabled or that there are no available workers yet available to service the request.
In all cases I recommend you have a look at ASP.NET Core best practices and make sure you are following the guidelines.
Lastly, I wanted to make sure the updates to the attributes where actually being applied, so I ran the ASP.NET Core application and took a memory dump. I wrote an article about MEX here and I ran !mex.AspxCoreConfiguration which dumped out the following as shown in Figure 3.
Figure 3, using WinDbg to check the custom configurations of an ASP.NET Core application
When you compare the values dumped out of the memory dump and the settings shown in the code at the beginning, you can see they match. I did not test if they were actually honored, but I have not reason to believe that are not. You should confirm if they are honored based on the value you set for AspNetCoreHostingModel.