Async iterators

[dom96]

This is related to FutureStreams. It sure would be neat to be able to do something like this:

import asyncdispatch, httpclient
let client = newAsyncHttpClient()

let resp = client.get("http://google.com")
for data in resp.bodyStream:
  # Do something with data.

This might not be possible.

[zielmicha]

reactor.nim has asyncFor: https://github.com/zielmicha/reactor.nim/blob/ef6afa3de8946b4959d5525de04ea42202541892/reactor/async/asyncmacro.nim#L197

Example:

asyncFor item in src:
  echo "piping ", item
  await dst.send(item)

It's implemented as a very simple macro (see source).

With support from {.async.} macro (e.g. translate stmtFor into call to asyncFor) it's possible to achieve for syntax you want. Of course then stdlib version of asyncFor macro would also need to handle normal iterators.

[jackhftang]

FYI, a working example as inspired by reactor.nim.

import asyncdispatch, httpclient, macros

macro asyncFor*(iterClause: untyped, body: untyped): untyped =
  if iterClause.kind != nnkInfix or iterClause[0] != ident("in"):
    error("expected `x in y` after for")

  let stream = iterClause[2]
  let itemName = iterClause[1]
  result = quote do:
    while true:
      let future = read(`stream`)
      yield future
      if future.failed: 
        raise future.readError()
      let tu = future.read()
      if not tu[0]: break
      let `itemName` {.inject.} = tu[1]
      `body`

proc main() {.async.} = 
  let client = newAsyncHttpClient()
  let resp = await client.get("http://google.com")
  asyncFor s in resp.bodyStream:
    echo s[1..20]

waitFor main()

The may not be possible part is that there is no true await (which is a macro transformed to future iterator), so it is not possible to have proc items() on FutureStream[T], which is sync, with async effect. Here is an example with implemented with waitFor, it is not so right, but it works.

import asyncdispatch, httpclient

iterator items[T](stream: FutureStream[T]): T =
  while true:
    let future = stream.read()
    let tu = waitFor future
    if not tu[0]: break
    yield tu[1]

proc main() {.async.} =
  let client = newAsyncHttpClient()
  let resp = await client.get("http://google.com")
  for data in resp.bodyStream:
    echo data[..20]
    
waitFor main()

[ringabout]

Related: https://github.com/SirNickolas/asyncIters-Nim