[DataStore] Optional fields related to another table using @belongsTo directive are throwing errors

[srimalaya]

Describe the bug

This is the schema that I am using:

GraphQL Schema

type Blog @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  customs: [MyCustomModel]
  notes: [String]
  post: [Post] @hasMany(indexName: "postByBlog", fields: ["id"])
}

type Post @model @auth(rules: [{allow: public}]) {
  id: ID!
  name: String!
  blog_id: ID @index(name: "postByBlog")
  random_id: String @index(name: "byRandom")
  blog: Blog @belongsTo(fields: ["blog_id"])
}

type MyCustomModel {
  id: ID!
  name: String!
  desc: String
  children: [MyNestedModel]
}

type MyNestedModel {
  id: ID!
  nestedName: String!
  notes: [String]
}

I already have this schema running and working perfectly on a test app for Android and JS. Now, I am trying to test DataStore on iOS (Swift + SwiftUI). With the above schema, I am successfully able to query all entries from the Blog table. I am also able to query from the Post table ONLY IF a value for "blog" field exists for all entries inside the Post table on DynamoDB.

The issue (specific to iOS only) arises when:

1. There is no value for "blog" for a particular entry and that entry is queried or is part of the result from a query.
2. When I try to create a Post and deliberately leave out "blog" OR set "blog" to nil

I have also tried setting no value or nil for the "random_id" field but that does not cause any issues. Only "blog" field causes issues.

I see this in logs whenever it happens in case of both query and save operations:

Error creating post - DataStoreError: The data couldn’t be read because it is missing.
Recovery suggestion: The data couldn’t be read because it is missing.
Caused by:
valueNotFound(Swift.String, Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "elements", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "blog", intValue: nil), CodingKeys(stringValue: "id", intValue: nil)], debugDescription: "Expected String value but found null instead.", underlyingError: nil))

There was an issue with the CLI when specifying "null" value for optional Index fields. However, that issue was fixed in CLI version 8.0.2 and works perfectly on Android.

Here are the queries that I am using:

Query for Blog and Save operation for Post

    func queryItem() {
        var blog: Blog? = nil
        Amplify.DataStore.query(Blog.self) {
            switch $0 {
            case .success(let result):
                print("Items: \(result)")
                blog = result[0]
                createItem(item: blog!)
            case .failure(let error):
                print("Error listing items - \(error)")
            }
        }
    }
    
    func createItem(item: Blog) {
        Amplify.DataStore.save(Post(name: "Test-2")) {
            switch $0 {
            case .success:
                print("Created a new post successfully")
            case .failure(let error):
                print("Error creating post - \(error)")
            }
        }
    }

Steps To Reproduce

Steps to reproduce the behavior:
1. Set up app backend with the provided schema
2. Populate data
4. Populate some entries inside `Post` model without specifying value for blog or set blog = nil
5. See error when creating , updating, or querying any entry with missing blog.

Expected behavior

I should be able to list or update all Posts irrespective of whether or not a value for "blog" exists.

Amplify Framework Version

1.24.0

Amplify Categories

DataStore

Dependency manager

Swift PM

Swift version

5.6

CLI version

8.2.0

Xcode version

13.3.1

Relevant log output

Error creating post - DataStoreError: The data couldn’t be read because it is missing.
Recovery suggestion: The data couldn’t be read because it is missing.
Caused by:
valueNotFound(Swift.String, Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "elements", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "blog", intValue: nil), CodingKeys(stringValue: "id", intValue: nil)], debugDescription: "Expected String value but found null instead.", underlyingError: nil))

Is this a regression?

No

Regression additional context

No response

Device

iPhone SE (3rd Generation)

iOS Version

15.4

Specific to simulators

No response

Additional context

No response

[srimalaya]

An update, in case it helps:
I tried using GraphQL Mutations (with the same code-generated models). GQL mutations work perfectly even when "blog" is null. I am able to both query and create a post without specifying a blog and it works as expected.

So, there's probably nothing wrong with the models, but a bug on the DataStore end is causing this issue. This is a serious constraint on our end. On our actual app, the schema has several fields which are optional, which are then connected via a one-to-many relationship with another table. It will be a headache to accommodate this constraint which for some reason is specific only to iOS, because of which we didn't catch it earlier.

[lawmicha]

Hi @shri-onecup , i can confirm with the following steps that I am able to reproduce errors related to your schema.

1. On AppSync console, send a create mutation for a Post without a Blog. The post is created successfully.
2. In the sample app, pull the data down via Amplify.DataStore.start, no errors and the single post should be in the local store now.
3. Perform Amplify.DataStore.query(Post.self) and see the error

2022-05-19 10:05:50.739374-0400 amplify1792[63462:867886] [SQLiteStorageEngineAdapter] select
  "root"."id" as "id", "root"."createdAt" as "createdAt", "root"."name" as "name",
  "root"."random_id" as "random_id", "root"."updatedAt" as "updatedAt", "root"."blog_id" as "blog_id",
  "blog"."id" as "blog.id", "blog"."createdAt" as "blog.createdAt", "blog"."customs" as "blog.customs",
  "blog"."name" as "blog.name", "blog"."notes" as "blog.notes", "blog"."updatedAt" as "blog.updatedAt"
from "Post" as "root"
left outer join "Blog" as "blog"
  on "blog"."id" = "root"."blog_id"
2022-05-19 10:05:50.742229-0400 amplify1792[63462:867886] [SQLiteStorageEngineAdapter] select
  "root"."id" as "id", "root"."createdAt" as "createdAt", "root"."name" as "name",
  "root"."random_id" as "random_id", "root"."updatedAt" as "updatedAt", "root"."blog_id" as "blog_id",
  "blog"."id" as "blog.id", "blog"."createdAt" as "blog.createdAt", "blog"."customs" as "blog.customs",
  "blog"."name" as "blog.name", "blog"."notes" as "blog.notes", "blog"."updatedAt" as "blog.updatedAt"
from "Post" as "root"
left outer join "Blog" as "blog"
  on "blog"."id" = "root"."blog_id"
2022-05-19 10:05:50.742564-0400 amplify1792[63462:867886] [DataStore] A column named blog_id was found in the result set but no field on
Post could be found with that name and it will be ignored.
2022-05-19 10:05:50.742806-0400 amplify1792[63462:867886] [DataStore] A column named blog_id was found in the result set but no field on
Post could be found with that name and it will be ignored.
DataStoreError: The data couldn’t be read because it is missing.
Recovery suggestion: The data couldn’t be read because it is missing.
Caused by:
valueNotFound(Swift.String, Swift.DecodingError.Context(codingPath: [CodingKeys(stringValue: "elements", intValue: nil), _JSONKey(stringValue: "Index 0", intValue: 0), CodingKeys(stringValue: "blog", intValue: nil), CodingKeys(stringValue: "id", intValue: nil)], debugDescription: "Expected String value but found null instead.", underlyingError: nil))

I believe this is related to queries performing an eager load on the associated model (the Blog of the Post). I believe the expected behavior should be to return the Post successful with nil blog. We'll have to look into this more closely in where exactly it's failing, how the SQL statement is created, and figure out a fix that decodes the data properly

[srimalaya]

@lawmicha Thanks for the update! Since, this is actually a bug with DataStore, can you please give me an estimate on when I can expect a fix. This bug is causing serious setbacks on our iOS app's production pipeline because all our other platforms (JS and Android) rely solely on DataStore.

[srimalaya]

@lawmicha can you please provide an update? We are really stalled on our timeline to launch the app into production. It is especially troublesome because I am forced to use GraphQL for now while both our other platforms are using DataStore, because of which we are facing issues with offline use, and updates to existing entries.

[lawmicha]

Hi @shri-onecup, sorry for the delays, the branch for this fix is located at fix/optional-associations please give it a try and let us know if you are seeing any issues. The PR is currently in draft as I work through more testing: #1849

The following use cases i've tested:

• save a post and query for it
• save a blog and query for it
• save a post, save a blog, update the post, query the post and see that the blog is eager loaded, query the blog and lazy load the posts.
• query all post, can see that some posts belong to a blog and some do not

[srimalaya]

Hi @lawmicha, I just got a chance to do some quick tests and it seems like the fix is working perfectly on my end. Thanks for it. Any ideas on when it might be released officially?

[srimalaya]

Hi @lawmicha, any estimates on when the fix will be made official? I am really looking forward to using it as part of an official release ASAP.

[srimalaya]

@lawmicha I think ObserveQuery is still facing this problem. The initial sync runs fine. But when fetching a real-time change from cloud, I get this error and the subscriptions just stop and don't start again even when cancelled and subscribed to again via another observeQuery.
[IncomingAsyncSubscriptionEventToAnyModelMapper] The operation couldn’t be completed. (Amplify.GraphQLResponseError<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>> error 1.)

Here's a bit detailed log (I cut chunks off it. Let me know if you want the entirety):

[IncomingAsyncSubscriptionEventPublisher] onUpdateValueListener: data(Swift.Result<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>, Amplify.GraphQLResponseError<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>>>.failure(GraphQLResponseError<MutationSync<AnyModel>>: GraphQL service returned a partially-successful response containing errors: [Amplify.GraphQLError(message: "Cannot return null for non-nullable type: \'Sex\' within parent \'Animal\' (/onUpdateNote/animal/animal_sex)", locations: nil, path: Optional([Amplify.JSONValue.string("onUpdateNote"), Amplify.JSONValue.string("animal"), Amplify.JSONValue.string("animal_sex")]), extensions: nil), .........same message for other fields......]

[IncomingAsyncSubscriptionEventToAnyModelMapper] dispose(of graphQLResponse): failure(GraphQLResponseError<MutationSync<AnyModel>>: GraphQL service returned a partially-successful response containing errors: [Amplify.GraphQLError(message: "Cannot return null for non-nullable type: \'Sex\' within parent \'Animal\' (/onUpdateNote/animal/animal_sex)", locations: nil, path: Optional([Amplify.JSONValue.string("onUpdateNote"), Amplify.JSONValue.string("animal"), Amplify.JSONValue.string("animal_sex")]), extensions: nil), .........same message for other fields......]

Recovery suggestion: The list of `GraphQLError` contains service-specific messages.)
2022-06-17 09:11:18.026604-0700 user-app[16800:26825656] [IncomingAsyncSubscriptionEventToAnyModelMapper] The operation couldn’t be completed. (Amplify.GraphQLResponseError<AWSPluginsCore.MutationSync<AWSPluginsCore.AnyModel>> error 1.)

For your reference: I was observing a Note model which has an optional field animal linked with @belongsTo directive.