Networking

Goal

Learn how to send and retrieve data on the network.

Updating our Models

To better support this functionality, we will update our existing application to have IDs. Since we have broken our nested json into multiple models, it will make it easier to follow.

First, download this file to replace your current app/src/main/assets/BooksAndAuthors.json. Make sure to overwrite the original file. This version of the data has id fields embedded for each list, book and author.

Clear your application cache and deploy your application.

Although we changed the json source, the models don't yet know about the change.

Add the optional id parameter to the ReadingList model. We'll also want to add a bookIds field, for talking with the server.

package com.aboutobjects.curriculum.readinglist.model

data class ReadingList(
    val title: String? = null,
    val books: List<Book> = emptyList(),
    val bookIds: List<Int> = emptyList(),
    val id: Int? = null
)

Why are these values optional?

If we were to create a new ReadingList, we don't yet know what any of the values should be.

For the Book model, we'll add both the id and authorId parameters.

package com.aboutobjects.curriculum.readinglist.model

data class Book(
    val title: String? = null,
    val author: Author? = null,
    val authorId: Int? = null,
    val year: String? = null,
    val id: Int? = null
)

And for the Author model, just add the id

val id: Int? = null

Deploy your application. Everything still OK?

Let's open your BookListService.edit function and change our filterNot criteria. Since we have an identifier now, we no longer need all of that other criteria. There should only ever be one match.

fun edit(source: Book?, edited: Book): Completable {
    return readingList.value?.let { oldReadingList ->
        // Create a new ReadingList
        val newReadingList = ReadingList(
            title = oldReadingList.title,
            books = oldReadingList.books
                .toMutableList()
                .filterNot { it.id == source?.id}
                .plus(edited)
                .toList()
        )
        // Save the results
        save(newReadingList)
    } ?: error(IllegalArgumentException("Reading List not found"))
}

This function will change a bit when we start talking to the network.

Clear your app storage, re-deploy your app and make sure that you can still edit books.

RESTful Server

At this point, the instructor will start up this RESTful server for us to talk to.

We will want to add a buildConfigField to represent the server we are going to talk to.

Open your module-level build.gradle and add a new buildConfigField. Note that although we are putting ours inside the default block, you could easily point to one server for production and another for your debug application.

buildConfigField "String", "CURRICULUM_SERVER", '"http://192.168.122.1:4567"'

The specific address you use will come from the instructor.

If you are running the ao-android-curriculum-server yourself, launch http://localhost:4567/hello and it will tell you what address to use.

Sync your project.

Child IDs vs Full Population

It is quite common for a RESTful server to only return the IDs of children so as to avoid sending unnecessary information. For an example of this, visit <CURRICULUM_SERVER>/lists in your browser.

IE using the IP address in the example above, http://192.168.122.1:4567/lists

You notice how each book is represented by only an ID? Let's look at a specific book. Visit <CURRICULUM_SERVER>/books/12.

Here we see that the author is represented by only an ID. <CURRICULUM_SERVER>/authors/10

If you were to retrieve the entire Reading List for display, you would make 1 call for the Reading List, 18 more calls for the Books, and 15 more calls for the Authors.

34 network calls for a single UI display is a bit much, don't you think?

When you encounter situations like that, make sure to call attention to it. The backend teams can likely add functionality for specific use cases.

For example, visit <CURRICULUM_SERVER>/books/12?full=true and notice how the Author is populated?

Similarly, you can visit <CURRICULUM_SERVER>/lists?full=true to view everything.

Obviously, if you have millions of Reading Lists, that would not be acceptable either. You have to draw a balance between low bandwidth / high performance calls and the number of calls required.

The main API page located at <CURRICULUM_SERVER>/hello will indicate which API endpoints support the ?full=true method.

Retrofit

While Java and Android both have built-in libraries for doing HTTP and network calls, using them to make RESTful calls can be quite tedious as you have to implement a lot of boilerplate code.

Wouldn't it be nice if you could skip all that and just define the API contract?

That's exactly what Retrofit allows you to do.

In your project-level build.gradle, let's add a new retrofitVersion.

project.ext {
    sampleVariable = "This is a sample variable"
    retrofitVersion = '2.5.0'
}

Then in your module-level build.gradle add the dependencies.

implementation "com.squareup.retrofit2:retrofit:${rootProject.ext.retrofitVersion}"
implementation "com.squareup.retrofit2:adapter-rxjava2:${rootProject.ext.retrofitVersion}"
implementation "com.squareup.retrofit2:converter-gson:${rootProject.ext.retrofitVersion}"

Sync your project.

AndroidManifest

For our application to be able to talk to the internet, we need to ask for permission.

Open your AndroidManifest.xml and add a new permission inside the manifest tag, before the application tag. Also add the android:usesCleartextTraffic to the application tag since our demo is not using HTTPS.

<manifest xmlns:android="http://schemas.android.com/apk/res/android"
          package="com.aboutobjects.curriculum.readinglist">

    <uses-permission android:name="android.permission.INTERNET" />
    
    <application
            android:usesCleartextTraffic="true">

Defining our API

CurriculumAPI

Right-click on your service package and add a new CurriculumAPI with a Kotlin type of interface.

This class will just define the interface of our API. More information about the API can be found here.

Simple GETs

First, let's define our simplest GET calls.

@GET("/lists")
fun getReadingLists( @Query("full") full: String = "true" ): Single<List<ReadingList>>

@GET("/books")
fun getBooks( @Query("full") full: String = "true" ): Single<List<Book>>

@GET("/authors")
fun getAuthors(): Single<List<Author>>

These methods take no required parameters (you can choose to turn off the full ID conversion we discussed earlier) and return a Single<List<MODEL>> of our models. The Single contract specifies that there can only be one result; which in our case is a List of models.

You'll notice that the full parameter is not an option on the /authors endpoint, per the specification of our server.

GET by id

Next up, we'll look at the GET calls that require an identifier to be passed.

@GET("/lists/{id}")
fun getReadingList(
    @Path("id") id: Int,
    @Query("full") full: String = "true"
): Single<ReadingList>

@GET("/books/{id}")
fun getBook(
    @Path("id") id: Int,
    @Query("full") full: String = "true"
): Single<Book>

@GET("/authors/{id}")
fun getAuthor(@Path("id") id: Int): Single<Author>

These look the same as before, but we are passing in the extra @Path element to replace the named parameter in the url. We are also only returning a single model, rather than a list of them.

Create with PUT

Creating objects is a little different.

@PUT("/lists/create")
fun createReadingList(
    @Body model: ReadingList,
    @Query("full") full: String = "true"
): Single<ReadingList>

@PUT("/books/create")
fun createBook(
    @Body model: Book,
    @Query("full") full: String = "true"
): Single<Book>

@PUT("/authors/create")
fun createAuthor(@Body model: Author): Single<Author>

Here, we pass our source model, which Gson converts and uses as the body of the HTTP PUT call to the server. Once again, the /authors/create endpoint does not support the full parameter.

These endpoints are the reason we added ReadingList.bookIds and Book.authorId to our models.

Update with POST

Similarly, we might want to update an existing model on the server.

For these, the server asks that we use POST. If the server were to require PATCH, we would have to make the client match.

@POST("/lists/update/{id}")
fun updateReadingList(
    @Path("id") id: Int,
    @Body model: ReadingList,
    @Query("full") full: String = "true"
): Single<ReadingList>

@POST("/books/update/{id}")
fun updateBook(
    @Path("id") id: Int,
    @Body model: Book,
    @Query("full") full: String = "true"
): Single<Book>

@POST("/authors/update/{id}")
fun updateAuthor(
    @Path("id") id: Int,
    @Body model: Author
): Single<Author>

This time we are using the all the different parts from above together into a single HTTP POST call.

Remove with DELETE

What if we wanted to delete an item?

@DELETE("/lists/delete/{id}")
fun deleteReadingList(@Path("id") id: Int): Completable

@DELETE("/books/delete/{id}")
fun deleteBook(@Path("id") id: Int): Completable

@DELETE("/authors/delete/{id}")
fun deleteAuthor(@Path("id") id: Int): Completable

This time we just use the id for an HTTP DELETE call. You might notice that we are using a Completable instead of a Single? That's because the server does not return any models to us. Actually, it just returns the word "ok".

Find with Forms

We have one last method

@FormUrlEncoded
@POST("/authors/find")
fun findAuthor(
    @Field("firstName") firstName: String?,
    @Field("lastName") lastName: String?
): Single<Author>

This time we are doing an HTTP POST call, but as application/x-www-form-urlencoded. This would be the same as if you had a form on a webpage and those fields were being populated by the user.

You might also notice that both of them are specified as String? instead of String this time. The server endpoint specifies that the parameters on this call are each optional; whereas on the other calls they are required.

As a side note, you would not normally implement every single method on your REST server. We are only showing these for completeness.

CurriculumService

Right-click on your service package and add a new CurriculumService with a Kotlin type of class. Add our ReadingListApp as a parameter to it.

package com.aboutobjects.curriculum.readinglist.service

import com.aboutobjects.curriculum.readinglist.ReadingListApp

class CurriculumService(val app: ReadingListApp) {
}

This class will provide the concrete implementation of our API.

Let's add a couple variables to tie things together.

private val retrofit = Retrofit.Builder()
    .baseUrl(BuildConfig.CURRICULUM_SERVER)
    .addConverterFactory(GsonConverterFactory.create(app.gson))
    .addCallAdapterFactory(RxJava2CallAdapterFactory.createWithScheduler(Schedulers.io()))
    .build()

private val backend = retrofit.create(CurriculumAPI::class.java)

You'll use the com.aboutobjects.curriculum.readinglist.BuildConfig import.

Here, we build a new private Retrofit instance using our previously defined server endpoint. We also specify that POJOs should be converted using our Gson configuration and that all network calls should be bound to the io thread.

We then create a backend variable that matches our API contract we built, using Retrofit.

You might be wondering why we aren't implementing the API directly. Maybe we want to add debug logging, don't want to expose some of the methods, or maybe we want change the parameters...

That being said, you could make that backend variable publicly available directly and remove the need for this class entirely. It just removes your ability to tweak or mock behavior.

Let's add a function to retrieve our ReadingList.

fun getReadingList(id: Int): Single<ReadingList> {
    return backend.getReadingList(id = id)
}

For creating the Author we create a local Author as a model; which the server will return with a populated id.

private fun createAuthor(author: Author): Single<Author> {
    return backend.createAuthor(
        model = Author(
            firstName = author.firstName,
            lastName = author.lastName
        )
    )
}

As a reminder, this is why the id is optional.

But, what if that author already exists? Let's add another method to double-check that first.

private fun findOrCreateAuthor(source: Author): Single<Author> {
    return backend.findAuthor(
        firstName = source.firstName,
        lastName = source.lastName
    ).onErrorResumeNext{
        createAuthor(author = source)
    }
}

We will attempt to find the author by name. If that fails, then we will call our createAuthor function.

Now we can add a function to create a new book.

fun createBook(source: Book): Single<Book> {
    return findOrCreateAuthor(source = source.author ?: throw IllegalArgumentException("Source Author missing"))
        .flatMap { author ->
            backend.createBook(
                model = Book(
                    title = source.title,
                    year = source.year,
                    authorId = author.id ?: throw IllegalArgumentException("Source Author ID missing")
                )
            )
        }
}

We will rely on our findOrCreateAuthor to ensure that the Author already exists on the backend before we make the backend call to create the book with the Author's ID. If there is no Author specified in our source book, we'll just bail as we can't do anything without that required parameter.

If you recall, our UI currently allows the user to change the Author's name in the same screen as the Book's title and year. A more robust design might require those to be edited on completely different screens. For our current design, we need to take additional precautions.

Let's add a method to our Author model.

fun isSameAs(other: Author?): Boolean {
    return displayName().equals(other?.displayName())
}

Back in CurriculumService, we can now write the updateBook function and take advantage of the new isSameAs.

fun updateBook(source: Book, edited: Book): Single<Book> {
    if (source.id == null) {
        throw IllegalArgumentException("Source Book ID missing")
    }

    if (source.author?.id == null) {
        throw IllegalArgumentException("Source Author missing")
    }

    return when {
        // source author HAS id, but edited author does NOT
        source.author.isSameAs(edited.author) -> Single.just(source.author)
        // maybe we typed a different existing authors' name?
        edited.author != null -> findOrCreateAuthor(edited.author)
        // fail-safe
        else -> Single.just(source.author)
    }.flatMap { author ->
        backend.updateBook(
            id = source.id,
            model = Book(
                id = source.id,
                title = edited.title,
                year = edited.year,
                authorId = author.id ?: throw IllegalArgumentException("Source Author ID missing")
            )
        )
    }
}

We start off by doing some sanity checks. If the source id or source author id are missing, we don't make any backend calls. We could have done those both using let, but in this case doing the fail-fast first is much cleaner.

We then try to determine who the Author is. If the source and edited books both list the same author (remember, same display name) - then we will use the source version since it has an id. The edited version would not have one yet.

If they do not match, we will call our findOrCreateAuthor function using the new edited author information. If the Author exists on the server, we will use it. If not, we will create it.

As a final sanity check, and because you should ALWAYS have an else/default clause, we will default to using the original author if the new one is missing.

Once we have determined the author, then we will make a call to the server to update the book. There is another throw in there; but it is just a sanity check. Our authorId should not be null at this stage.

We're also going to need a way to update our ReadingList with the new Books. The backend call requires that we pass a list of bookIds, so we need to iterate over our book list and replace the Book entries with an Int id.

fun addBookToReadingList(readingList: ReadingList, book: Book): Single<ReadingList> {
    if (book.id == null) {
        throw IllegalArgumentException("Book ID missing")
    }

    return backend.updateReadingList(
        id = readingList.id ?: throw IllegalArgumentException("Reading List ID missing"),
        // Send a version with just the IDs
        model = ReadingList(
            id = readingList.id,
            title = readingList.title ?: "",
            bookIds = readingList.books
                .mapNotNull { it -> it.id }
                .toMutableList()
                .plus(book.id)
                .distinct()
                .sorted()
                .toList()
        )
    )
}

Once we have converted the existing List<Book> to a List<Int>, we make the list mutable (ie: MutableList<Int>) and add the requested Book to the list as well. We make sure there are no duplicates (again, another sanity check) and re-sort them. Finally, we convert it back to a non-mutable List<Int> that the backend API requires.

As you can see, you can customize how the backend calls are made rather than a 1:1 match.

Calling our API

Let's hook our new service up. If we were using dependency injection (like Dagger) we would be doing this a bit different; but for now, let's add it to our ReadingListApp.

val curriculumService: CurriculumService by lazy { 
    CurriculumService(app = this)
}

Loading from the Server

Open our BookListService and add a new method

private fun loadFromServer(): Maybe<ReadingList> {
    return app.curriculumService
        .getReadingList(id = 0)
        .toMaybe()
        .onErrorComplete()
}

Since we aren't using any FileReader this time, it won't throw any Exceptions that we can catch. If there is a problem, it will come back in the onError callback.

And since we want to gracefully fall-over to something else rather than bailing out, we tell it to treat a failure to read the server the same way it would treat reading an empty cache file.

Now we can change our init block... maybe like:

init {
    val loadDisposable = loadFromServer()
        .switchIfEmpty(loadFromFiles())
        .switchIfEmpty(loadFromAssets())
        .subscribeOn(Schedulers.newThread())
        .observeOn(AndroidSchedulers.mainThread())
        .subscribeBy(
            onSuccess = { readingList -> save(readingList) },
            onError = { Log.w(ReadingListApp.TAG, "Error loading books", it) },
            onComplete = { Log.w(ReadingListApp.TAG, "No books found") }
        )
}

So here, we will try to load from the server first. If that fails, we will try to load our cached copy. If that ALSO fails, we will try to load the original version shipped with the application.

To understand what this looks like, I added a couple additional logs, cleared the cache and started the application without the server running:

12-04 08:39:59.442 D/ReadingListApp( 6240): Reading from SERVER
12-04 08:39:59.497 D/ReadingListApp( 6240): Reading from FILES
12-04 08:39:59.499 D/ReadingListApp( 6240): Failed to load Json from files/BooksAndAuthors.json
12-04 08:39:59.499 D/ReadingListApp( 6240): java.io.FileNotFoundException: /data/user/0/com.aboutobjects.curriculum.readinglist/files/BooksAndAuthors.json (No such file or directory)
12-04 08:39:59.499 D/ReadingListApp( 6240): 	at java.io.FileInputStream.open0(Native Method)
12-04 08:39:59.499 D/ReadingListApp( 6240): 	at java.io.FileInputStream.open(FileInputStream.java:231)
12-04 08:39:59.499 D/ReadingListApp( 6240): 	at java.io.FileInputStream.<init>(FileInputStream.java:165)
12-04 08:39:59.499 D/ReadingListApp( 6240): 	at java.io.FileReader.<init>(FileReader.java:72)
12-04 08:39:59.499 D/ReadingListApp( 6240): 	at com.aboutobjects.curriculum.readinglist.service.BookListService.loadFromFiles(BookListService.kt:62)
12-04 08:39:59.499 D/ReadingListApp( 6240): 	at com.aboutobjects.curriculum.readinglist.service.BookListService.<init>(BookListService.kt:29)
...clip...
12-04 08:39:59.499 D/ReadingListApp( 6240): Reading from ASSETS
12-04 08:39:59.768 I/ReadingListApp( 6240): 18 books loaded

We can see that it tried to read from the (temporarily down) server, then from the (not yet written) cache then finally from the assets copy.

You might be asking why we'd go through so much effort? Why not just rely on the server copy? This is just to give you an idea of what might be required to support "Offline Mode". You need the local cache in order to read/write changes when the app is offline. You need the asset copy in order to seed the app if it is offline when they first launch it. It could also be useful if they chose to 'factory reset' your application.

Saving to the Server

Currently, our application does have the concept of creating new reading lists. The API supports it, so that is functionality you could add (maybe with some additional UI elements). For now, let's focus on what we need to do to update a single book entry on the server.

Our application currently supports functionality to both "Add Book" and "Edit Book". Both functions do their processing through BookListService.edit. The "Add Book" scenario just has the source set to null.

As such, that function is the one we will hijack to change the behavior.

To re-cap, this is the current offline-only function (from above):

fun edit(source: Book?, edited: Book): Completable {
    return readingList.value?.let { oldReadingList ->
        // Create a new ReadingList
        val newReadingList = ReadingList(
            title = oldReadingList.title,
            books = oldReadingList.books
                .toMutableList()
                .filterNot { it.id == source?.id}
                .plus(edited)
                .toList()
        )
        // Save the results
        save(newReadingList)
    } ?: error(IllegalArgumentException("Reading List not found"))
}

Let's replace it with a new online-only function:

fun edit(source: Book?, edited: Book): Single<Book> {
    return when (source) {
        null -> app.curriculumService.createBook(source = edited)
        else -> app.curriculumService.updateBook(source = source, edited = edited)
    }.flatMap { book ->
        app.curriculumService
            .addBookToReadingList(
                readingList = readingList.value ?: throw IllegalArgumentException("Reading List not loaded"),
                book = book
            )
            .map { it -> Pair(it, book) }
            .doOnSuccess { (readingList, _) -> save(readingList) }
    }.map { (_, book) ->
        book
    }
}

If the source book is null, we call our createBook method. If it is not, we call updateBook. Both of those methods return a Single<Book>.

We take that result and tell the server to add the book to the reading list. Without this step, the new book would exist, but not be associated with our reading list.

Once we get a response on updating the reading list, we make sure to save it to our local cache. Finally, we return that book to the caller.

You might be wondering about the Pair(it, book), (readingList, _) and (_, book). Pair(it aka readingList, book) is being used to return 2 values instead of just 1. Normally, only the readingList would get carried forward, but we need the book on a future step. Whenever we don't need to use a particular value, it can be replaced with _ to represent "don't know, don't care". In this case, save only cares about the readingList and the map only cares about the book.

As a final change, since our old function returned Completable and the new one returns Single<Book>, let's update our EditBookFragment.onOptionsItemSelected. Just change the word onComplete to onSuccess.

Deploy your application and edit some books.

The other students in the classroom are currently sharing the same reading list as you. Are you seeing each others books?

Improvements

What improvements could be made to the application?

Maybe a periodic refresh in case you didn't edit any books?
Maybe a personalized reading list rather than a shared one?
Maybe a share button so you could email the list to a friend?
How about a separate page to edit the Author directly?
Maybe you would like to specify a sort order or filtering rules?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Networking

Goal

Table of Contents

Updating our Models

RESTful Server

Child IDs vs Full Population

Retrofit

AndroidManifest

Defining our API

CurriculumAPI

Simple GETs

GET by id

Create with PUT

Update with POST

Remove with DELETE

Find with Forms

CurriculumService

Calling our API

Loading from the Server

Saving to the Server

Improvements

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally