[QUERY] Authentication exceptions attempting to read secrets with new 4.1.0 package

[PunzunLtd]

Query/Question
Our issue is very similar to a (closed) issue from last spring. When attempting to read a secret using DefaultAzureCredentials in Visual Studio we get one exception, but using VisualStudioCredentials we get a different exception. In both cases we're blocked. We haven't tried to run the code in our devops pipeline, which uses an Azure managed identity and therefore (we think) would not be affected by this problem.

What are we doing wrong? Case 11900 was closed, implying it was solved. That case referenced an older, deprecated version of the KeyVault package. In fact, our code worked fine with Microsoft.Azure.KeyVault 3.0.5 and failed after upgrading and refactoring to the new Azure.Security.KeyVault.Secrets 4.1.0 package.

If we can't get this to work in Visual Studio, we'll have to go back to the deprecated package.

Environment:

• Name and version of the Library package used: Azure.Security.KeyVault.Secrets 4.1.0
• Hosting platform: .NET 5.0.0
• IDE and version : Visual Studio 16.8.2
• Using JetBrains ReSharper 2020.3 EAP 9 unit test runner
• AZURE_TENANT_ID variable set to our tenant ID in the unit test assembly config, but may not be picked up by ReSharper

User account
The Visual Studio identity is both the logged-in Windows identity and the Azure service administrator identity, created as a live.com ID back in the mid-2000s. Elsewhere in the universe there is a Microsoft Office identity created in the same era with the same email address, which I mention because other issues suggest this might be a problem.

Visual Studio is configured only to use the Windows (i.e., Azure administrator) identity, filtered to the subscription containing the vault.

The user account has an access policy with all 8 Secrets permissions on the affected KeyVault, and is in the role of Owner of the KeyVault.

Details
Code

The failing code in both cases looks like this:

using var listener = AzureEventSourceListener.CreateConsoleLogger();
var options = new DefaultAzureCredentialOptions()
{
	Diagnostics =
	{
		LoggedHeaderNames = { "x-ms-request-id" },
		LoggedQueryParameters = { "api-version" },
		IsLoggingContentEnabled = true
	}
};
var credential = new DefaultAzureCredential(options); //new VisualStudioCredential(options);
var secretClient = new SecretClient(new Uri("https://contoso-dev.vault.azure.net/"), credential);
var secret = secretClient.GetSecret(key);
return secret.Value?.Value;

Using DefaultAzureCredentials
Throws Azure.Identity.AuthenticationFailedException:

Azure.Identity.AuthenticationFailedException : SharedTokenCacheCredential authentication failed: A configuration issue is preventing authentication - check the error message from the server for details.You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS70002: The client does not exist or is not enabled for consumers. If you are the application developer, configure a new application through the App Registrations in the Azure Portal at https://go.microsoft.com/fwlink/?linkid=2083908.
Trace ID: ***
Correlation ID: ***
Timestamp: 2020-11-30 18:00:53Z
  ----> Microsoft.Identity.Client.MsalServiceException : A configuration issue is preventing authentication - check the error message from the server for details.You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS70002: The client does not exist or is not enabled for consumers. If you are the application developer, configure a new application through the App Registrations in the Azure Portal at https://go.microsoft.com/fwlink/?linkid=2083908.

{snip}

[Informational] Azure-Identity: DefaultAzureCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: b267ca44-fc3b-4ace-b0c1-ea7f559d430a
[Informational] Azure-Identity: EnvironmentCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: b267ca44-fc3b-4ace-b0c1-ea7f559d430a
[Informational] Azure-Identity: EnvironmentCredential.GetToken was unable to retrieve an access token. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: b267ca44-fc3b-4ace-b0c1-ea7f559d430a Exception: Azure.Identity.CredentialUnavailableException (0x80131500): EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: b267ca44-fc3b-4ace-b0c1-ea7f559d430a
[Informational] Azure-Identity: Probing IMDS endpoint for availability. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token
[Informational] Azure-Identity: IMDS endpoint is did not respond. Endpoint: http://169.254.169.254/metadata/identity/oauth2/token
[Informational] Azure-Identity: ManagedIdentityCredential.GetToken was unable to retrieve an access token. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: b267ca44-fc3b-4ace-b0c1-ea7f559d430a Exception: Azure.Identity.CredentialUnavailableException (0x80131500): ManagedIdentityCredential authentication unavailable. No Managed Identity endpoint found.
[Informational] Azure-Identity: SharedTokenCacheCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: b267ca44-fc3b-4ace-b0c1-ea7f559d430a
[Informational] Azure-Core: Request [f5da76b0-2a50-4959-9aa0-e23e6210c187] GET https://login.microsoftonline.com/common/discovery/instance?api-version=1.1&authorization_endpoint=REDACTED

The log on the Azure Key Vault looks like this:

{ 
"time": "2020-11-30T18:00:52.3980962Z", 
"category": "AuditEvent", "operationName": 
"Authentication", "resultType": "Success", 
"correlationId": "12345678-a41b-470b-ad18-123456789ab", 
"callerIpAddress": "8.8.8.8",
"identity": {}, 
"properties": {"clientInfo":"azsdk-net-Security.KeyVault.Secrets/4.1.0 (.NET 5.0.0; Microsoft Windows 10.0.19042)","httpStatusCode":401,"requestUri":"https://contoso.vault.azure.net/secrets/unitTestSecret/?api-version=7.1"}, 
"resourceId": "/SUBSCRIPTIONS/7FE1B682-83B4-4AAC-B848-9196A7D401A9/RESOURCEGROUPS/CONTOSO-RG/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSO-DEV", 
"operationVersion": "7.1", 
"resultSignature": "Unauthorized", 
"durationMs": "39"
}

Using VisualStudioCredentials

Exception is:

Azure.RequestFailedException : Service request failed.
Status: 403 (Forbidden)

Content:
{"error":{"code":"Forbidden","message":"Access denied to first party service.\r\nCaller: name=from-infra;tid=f8cdef31-a31e-4b4a-93e4-5f571e91255a;appid=872cd9fa-d31f-45e0-9eab-6e460a02d1f1;iss=https://sts.windows.net/f8cdef31-a31e-4b4a-93e4-5f571e91255a/\r\nVault: contoso-dev;location=eastus","innererror":{"code":"AccessDenied"}}}

Headers:
Cache-Control: no-cache
Pragma: no-cache
x-ms-keyvault-region: eastus
x-ms-request-id: REDACTED
x-ms-keyvault-service-version: 1.2.80.0
x-ms-keyvault-network-info: conn_type=Ipv4;addr=8.8.8.8;act_addr_fam=InterNetwork;
X-Powered-By: REDACTED
Strict-Transport-Security: REDACTED
X-Content-Type-Options: REDACTED
Date: Mon, 30 Nov 2020 20:15:13 GMT
Content-Length: 336
Content-Type: application/json; charset=utf-8
Expires: -1

{snip}

[Informational] Azure-Identity: VisualStudioCredential.GetToken invoked. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: 23c290b4-dd25-45d0-b2f3-c47ff8b631a0
[Informational] Azure-Identity: VisualStudioCredential.GetToken succeeded. Scopes: [ https://vault.azure.net/.default ] ParentRequestId: 23c290b4-dd25-45d0-b2f3-c47ff8b631a0 ExpiresOn: 2020-11-30T21:15:12.0000000+00:00
[Informational] Azure-Core: Request [23c290b4-dd25-45d0-b2f3-c47ff8b631a0] GET https://contoso-dev.vault.azure.net/secrets/unitTestSecret/?api-version=7.1

Azure Key Vault log contains:

{ "time": "2020-11-30T20:15:11.5468759Z", "category": "AuditEvent", "operationName": "Authentication", "resultType": "Success", "correlationId": "6c71169d-f9bb-49ae-a3e2-3a696e4020cf", "callerIpAddress": "8.8.8.8", "identity": {}, "properties": {"clientInfo":"azsdk-net-Security.KeyVault.Secrets/4.1.0 (.NET 5.0.0; Microsoft Windows 10.0.19042)","httpStatusCode":401,"requestUri":"https://contoso-dev.vault.azure.net/secrets/unitTestSecret/?api-version=7.1"}, "resourceId": "/SUBSCRIPTIONS/7FE1B682-83B4-4AAC-B848-9196A7D401A9/RESOURCEGROUPS/CONTOSO-RG/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSO-DEV", "operationVersion": "7.1", "resultSignature": "Unauthorized", "durationMs": "34"}
{ "time": "2020-11-30T20:15:13.3719179Z", "category": "AuditEvent", "operationName": "SecretGet", "resultType": "Success", "resultDescription": "Access denied to first party service.\nCaller: name=from-infra;tid=f8cdef31-a31e-4b4a-93e4-5f571e91255a;appid=872cd9fa-d31f-45e0-9eab-6e460a02d1f1;iss=https://sts.windows.net/f8cdef31-a31e-4b4a-93e4-5f571e91255a/\nVault: contoso-dev;location=eastus", "correlationId": "ce39bd97-25a1-44b6-a12c-123456789ab", "callerIpAddress": "8.8.8.8", "identity": {"claim":{"appid":"872cd9fa-d31f-45e0-9eab-6e460a02d1f1","http://schemas.microsoft.com/identity/claims/scope":"user_impersonation","ipaddr":"8.8.8.8","http://schemas.microsoft.com/claims/authnmethodsreferences":"pwd"}}, "properties": {"id":"https://contoso-dev.vault.azure.net/secrets/unitTestSecret","clientInfo":"azsdk-net-Security.KeyVault.Secrets/4.1.0 (.NET 5.0.0; Microsoft Windows 10.0.19042)","httpStatusCode":403,"requestUri":"https://contoso-dev.vault.azure.net/secrets/unitTestSecret/?api-version=7.1","isAccessPolicyMatch":false}, "resourceId": "/SUBSCRIPTIONS/7FE1B682-83B4-4AAC-B848-9196A7D401A9/RESOURCEGROUPS/CONTOSO-RG/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSO-DEV", "operationVersion": "7.1", "resultSignature": "Forbidden", "durationMs": "98"}
{ "time": "2020-11-30T20:17:42.1404802Z", "category": "AuditEvent", "operationName": "Authentication", "resultType": "Success", "correlationId": "ec4dd886-8cc8-4c73-b72c-92ada31f3155", "callerIpAddress": "8.8.8.8", "identity": {}, "properties": {"clientInfo":"azsdk-net-Security.KeyVault.Secrets/4.1.0 (.NET 5.0.0; Microsoft Windows 10.0.19042)","httpStatusCode":401,"requestUri":"https://contoso-dev.vault.azure.net/secrets/unitTestSecret/?api-version=7.1"}, "resourceId": "/SUBSCRIPTIONS/7FE1B682-83B4-4AAC-B848-9196A7D401A9/RESOURCEGROUPS/CONTOSO-RG/PROVIDERS/MICROSOFT.KEYVAULT/VAULTS/CONTOSO-DEV", "operationVersion": "7.1", "resultSignature": "Unauthorized", "durationMs": "98"}

[jsquire]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[dondeetan]

I'm having the same problem and here's the code I use using azure.identity:

Exception has occurred: CLR/Microsoft.Extensions.Configuration.AzureAppConfiguration.KeyVaultReferenceException
An unhandled exception of type 'Microsoft.Extensions.Configuration.AzureAppConfiguration.KeyVaultReferenceException' occurred in Microsoft.Extensions.Configuration.AzureAppConfiguration.dll
Inner exceptions found, see $exception in variables window for more details.
Innermost exception Azure.RequestFailedException : Service request failed.
Status: 403 (Forbidden)

Content:
{"error":{"code":"Forbidden","message":"Access denied to first party service.\r\nCaller: name=from-infra;tid=f8cdef31-a31e-4b4a-93e4-5f571e91255a;appid=872cd9fa-d31f-45e0-9eab-6e460a02d1f1;iss=https://sts.windows.net/f8cdef31-a31e-4b4a-93e4-5f571e91255a/\r\nVault: secopsreporting-kv;location=westus2","innererror":{"code":"AccessDenied"}}}

public static IHostBuilder CreateHostBuilder(string[] args) =>
Host.CreateDefaultBuilder(args)
.ConfigureWebHostDefaults(webBuilder =>
{
webBuilder.ConfigureAppConfiguration((hostingContext, config) =>
{
var settings = config.Build();
var appconfigurl = settings["AppSettings:AppConfigEndpoint"];
config.AddAzureAppConfiguration(options =>
{
//Used for Getting App ConfigurationUrl but using Managed Identities
//Note: See this bug:
//#11509
options.Connect(new Uri(appconfigurl), new DefaultAzureCredential(new DefaultAzureCredentialOptions{
ExcludeSharedTokenCacheCredential = true
})).UseFeatureFlags()
.ConfigureKeyVault(kv =>
{
kv.SetCredential(new DefaultAzureCredential(new DefaultAzureCredentialOptions{
ExcludeSharedTokenCacheCredential = true
}));
});
});
})
.UseUrls("http://0.0.0.0:8081")
.UseStartup();
});

Forgot to add that I do have the access policy setup in Azure

[schaabs]

@PunzunLtd Thanks for filing this issue. I'm sorry for the delayed response. The original issue you hit with the DefaultAzureCredential throwing AuthenticationFailed

Azure.Identity.AuthenticationFailedException : SharedTokenCacheCredential authentication failed: A configuration issue is preventing authentication - check the error message from the server for details.You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS70002: The client does not exist or is not enabled for consumers. If you are the application developer, configure a new application through the App Registrations in the Azure Portal at https://go.microsoft.com/fwlink/?linkid=2083908.
Trace ID: ***
Correlation ID: ***
Timestamp: 2020-11-30 18:00:53Z
  ----> Microsoft.Identity.Client.MsalServiceException : A configuration issue is preventing authentication - check the error message from the server for details.You can modify the configuration in the application registration portal. See https://aka.ms/msal-net-invalid-client for details.  Original exception: AADSTS70002: The client does not exist or is not enabled for consumers. If you are the application developer, configure a new application through the App Registrations in the Azure Portal at https://go.microsoft.com/fwlink/?linkid=2083908.

This is caused by a known issue with MSA accounts and the SharedTokenCacheCredential described in issue #16306 (although the title is a bit missleading because in this case the issue was blocking the DefaultAzureCredential from using the AzureCliCredential). The issue unfortunately isn't fixable at this time, and so we are will actually be disabling the SharedTokenCacheCredential from the DefaultAzureCredential authentication flow in the next release of the Azure.Identity library. The end behavior of the DefaultAzureCredential will still be the same to the user as we have essentially replaced the SharedTokenCacheCredential with the VisualStudioCredential. You can work around this issue either in the way you have by using the VisualStudioCredential, or you could go back to using the DefaultAzureCredential and simply disable the SharedTokenCacheCredential.

var cred = new DefaultAzureCredential(new DefaultAzureCredentialOptions { ExcludeSharedTokenCacheCredential = true });

Unfortunately this still doesn't explain the 403 from the Key Vault service once you were able to acquire a token, @heaths should be able to help out more here.