“Internal connection fatal error” from TdsParser when calling native compiled stored procedure in SQL Server 2019

[Genmutant]

Describe the bug

I am having huge problems with the current SQL Server 2019 (15.0.2070.41) and native compiled stored procedures when calling them from the Microsoft.Data.SqlClient (1.1.0 and lower).

It is working when either using SQL Server 2017 or System.Data.SqlClient, or just not using a native compiled procedure.

Apparently the TdsParser is receiving data it doesn't expect from the SQL Server connection and is throwing an exception during the parsing.

Exception message: System.InvalidOperationException: Internal connection fatal error.
Stack trace: at Microsoft.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at Microsoft.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at Microsoft.Data.SqlClient.SqlDataReader.get_MetaData()
at Microsoft.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReaderTds(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, Boolean isAsync, Int32 timeout, Task& task, Boolean asyncWrite, Boolean inRetry, SqlDataReader ds, Boolean describeParameterEncryptionRequest)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, TaskCompletionSource`1 completion, Int32 timeout, Task& task, Boolean& usedCache, Boolean asyncWrite, Boolean inRetry, String method)
at Microsoft.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method)
at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior)
at Microsoft.Data.SqlClient.SqlCommand.ExecuteReader()

To reproduce

using System;
using System.Data;
using Microsoft.Data.SqlClient;

namespace SqlDbTest
{
    class Program
    {
        static void Main()
        {
            const string connString = "Server=localhost;Database=TestDb;Trusted_Connection=True";    
            const string prod = @"CREATE OR ALTER PROCEDURE dbo.NativeProdForTesting WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNER
                AS BEGIN ATOMIC WITH ( TRANSACTION ISOLATION LEVEL = SNAPSHOT,  LANGUAGE = N'us_english')
                    SELECT NULL AS testcol
                RETURN 0
                END";

            using (var con = new SqlConnection(connString))
            {
                using var createSp = con.CreateCommand();
                createSp.CommandText = prod;
                con.Open();
                createSp.ExecuteNonQuery();
            }

            try
            {
                using var con = new SqlConnection(connString);
                using var cmd = con.CreateCommand();
                cmd.CommandType = CommandType.StoredProcedure;
                cmd.CommandText = "[dbo].[NativeProdForTesting]";
                cmd.Connection.Open();
                using var reader = cmd.ExecuteReader();
                Console.WriteLine("NO ERROR!");
            }
            catch (InvalidOperationException ex)
            {
                Console.WriteLine(ex);
            }
        }
    }
}

Expected behavior

No exception and the reader should provide the result of the query.

Further technical details

Microsoft.Data.SqlClient version: 1.1.0 and lower
.NET target: Core 3.1, Framework 4.7.2
SQL Server version: SQL Server 2019 (15.0.2070.41)
Operating system: Windows 10, Windows Server 2016 Standard

Additional context
I have the same problem with JDBC and it's TDSParser (7.4.1.jre8), see here. I also posted the problem on Stackoverflow

[cheenamalhotra]

Hi @Genmutant

I'm able to reproduce issue with Microsoft.Data.SqlClient and am looking into it.
Thanks for reporting this with us.

[Genmutant]

The token which can't be parsed seems to be the same as the one which JDBC has a problem with, and that also doesn't get recogniced by Wireshark: The 0xA3 token.

[Genmutant]

Introduced in TDS 7.4, the DATACLASSIFICATION token SHOULD<39> describe the data
classification of the query result set.
Token Stream Comments:
 The token value is 0xA3.
 This token only be sent by the server if the client sends a DATACLASSIFICATION FeatureExt in the
Login message and the server responds with a DATACLASSIFICATION FeatureExtAck. When this
token is used, the token is sent by the server in response to every SQLBatch request.

[cheenamalhotra]

Hi @Genmutant

Yes you're right, this token is received because both JDBC and Microsoft.Data.SqlClient drivers support Data Classification and we send DATACLASSIFICATION FeatureExt token during login. System.Data.SqlClient does not support this feature hence is not impacted.

The glitch is this token is not expected to be in the stream at a location where it's currently being received for this use-case, as per feature design. We're following up with product team internally to gather more details on the same. Will keep you posted!

[cheenamalhotra]

Client sends a 0x09 (DATACLASSIFICATION) as FeatureExt during Login (link), server sends acknowledgement with 0x09 (DATACLASSIFICATION) as FeatureExtAck.

Wireshark may not translate TDS login packets from client correctly.

[cheenamalhotra]

Hi @Genmutant

I'd like to confirm this was identified as a bug in the SQL Server 2019 and we have received confirmation on the fix made to address the same.

The upcoming SQL Server 2019 CU2 release will address this issue.
There will be no changes required in SqlClient and JDBC drivers.

[DaEkstrim]

I know this is not the right place to ask this kind of question, but I'm experiencing the same issue once I migrated from System.Data.SqlClient to Microsoft.Data.SqlClient while trying to execute a native compiled stored procedure on an SQL 2019 compatiblity Azure SQL Database through a linux-based dotNet Core 2.2 application on AKS, but I couldn't find any other information on the internet.

The question is: Do you have an information when this fix (CU2) would be released for Azure SQL Database (PAAS)?

[Genmutant]

Fixed in CU2: https://support.microsoft.com/en-us/help/4538342/fix-latest-drivers-fail-to-work-with-in-memory-compiled-queries-in-sql

[cheenamalhotra]

@Genmutant thanks for linking the release update, we'll close the issue!

[augustolp]

Hello @cheenamalhotra! How are you?
Do you know if this issue was also fixed in Azure SQL service?
Best,
Augusto.

[cheenamalhotra]

Hi @augustolp

The fix is currently not yet deployed in Azure instances.
Estimate of deployment is in April.

[simader]

Hello,
@cheenamalhotra:
Can anybody confirm that this is roled out on every azure DB instance? I have a strange behaviour that the same code is running on nearly all DBs, but on one database i get this excecption.

Or can be there an ohter issue?