Great work! Do you have uasyncio/non-blocking feature in the roadmap?

[beyonlo]

Hi, I talked to you in the past on this issue mcauser/awesome-micropython#38

Do you think do support the uasyncio/non-blocking on the Slave and Master?

Thank you in advance!

[GimmickNG]

I'm not sure about async support, but I have managed to get good success in my fork by using select.select(), disabling all socket timeouts and using the timeout parameter in the call to select():

https://github.com/GimmickNG/pycopy-modbus/blob/c0f6622ad3d1bd462846b9b9d9ac714e45b66569/umodbus/tcp.py#L288-L295

My version is primarily intended for Pycopy, not Micropython, so additional libraries like typing and selectors may not be available in the latter. However, it should be easy to remove those and use plain uselect.poll() instead.

[beyonlo]

I'm not sure about async support, but I have managed to get good success in my fork by using select.select(), disabling all socket timeouts and using the timeout parameter in the call to select():

https://github.com/GimmickNG/pycopy-modbus/blob/c0f6622ad3d1bd462846b9b9d9ac714e45b66569/umodbus/tcp.py#L288-L295

My version is primarily intended for Pycopy, not Micropython, so additional libraries like typing and selectors may not be available in the latter. However, it should be easy to remove those and use plain uselect.poll() instead.

Hello!

I read a bit about the select and what I understand is that it is nonblocking too, is that correct? The results is the same than using a sock.setblocking(False)? Or am I wrong?

That select help to non blocking just for Modbus TCP. And about the Modbus RTU, did you think anything to manage a better non blocking? The uasyncio can to do nonblocking UART with asyncio.StreamReader() and asyncio.StreamWriter() instead uart.read() and uart.write(). I think will be enough to non blocking on the RTU.

I am curious. Why do you use pycopy instead Micropython? Any new/different feature, or your project already running old application with pycopy, or what?

Thank you so much.

[GimmickNG]

I'm not sure about async support, but I have managed to get good success in my fork by using select.select(), disabling all socket timeouts and using the timeout parameter in the call to select():
https://github.com/GimmickNG/pycopy-modbus/blob/c0f6622ad3d1bd462846b9b9d9ac714e45b66569/umodbus/tcp.py#L288-L295
My version is primarily intended for Pycopy, not Micropython, so additional libraries like typing and selectors may not be available in the latter. However, it should be easy to remove those and use plain uselect.poll() instead.

Hello!

I read a bit about the select and what I understand is that it is nonblocking too, is that correct? The results is the same than using a sock.setblocking(False)? Or am I wrong?

I'm not sure. Initially I tried using socket with a 0 timeout but I would never be able to read any messages despite sending them to the server, so I switched to select and it seemed to work after that.

That select help to non blocking just for Modbus TCP. And about the Modbus RTU, did you think anything to manage a better non blocking? The uasyncio can to do nonblocking UART with asyncio.StreamReader() and asyncio.StreamWriter() instead uart.read() and uart.write(). I think will be enough to non blocking on the RTU.

You're most likely correct, I don't use Modbus RTU and select() probably wouldn't work there. For my usecase it seemed that select() did the job good enough. Implementing asyncio support might be more "correct", but it would also involve making huge changes to my code in addition to the library so I didn't bother with it at the moment.

But now when writing multiple registers, I'm facing an issue where the transaction IDs don't match. I wonder if it'll be worth it to switch over to asyncio or not, but I have really no clue.

I am curious. Why do you use pycopy instead Micropython? Any new/different feature, or your project already running old application with pycopy, or what?
The main feature that I know of is that pycopy had support for minimal versions of the python standard library (via pycopy-lib, e.g. pycopy-selectors) and I couldn't find anything like that for Micropython. In my specific case, I needed support for argparse but I couldn't find anything for Micropython, whereas Pycopy had it so I switched. The two are very similar from what I understand, so I could theoretically switch from one to another just by changing the binary, but I haven't tried that yet.

I mainly work with the UNIX port (my usecase is running hundreds of Modbus clients and servers at the same time, which would have taken ~10GB of memory using CPython+pymodbus, whereas it would probably take <2GB using Pycopy/Micropython), so I didn't work with Modbus RTU, as Pycopy didn't have machine.UART support.

In fact, that's one of the reasons I moved the TCP server to tcp.py in my fork - because I couldn't even import umodbus.modbus without running into ImportErrors, due to the Serial server which I didn't use anyway.

[beyonlo]

I'm not sure about async support, but I have managed to get good success in my fork by using select.select(), disabling all socket timeouts and using the timeout parameter in the call to select():
https://github.com/GimmickNG/pycopy-modbus/blob/c0f6622ad3d1bd462846b9b9d9ac714e45b66569/umodbus/tcp.py#L288-L295
My version is primarily intended for Pycopy, not Micropython, so additional libraries like typing and selectors may not be available in the latter. However, it should be easy to remove those and use plain uselect.poll() instead.

Hello!
I read a bit about the select and what I understand is that it is nonblocking too, is that correct? The results is the same than using a sock.setblocking(False)? Or am I wrong?

I'm not sure. Initially I tried using socket with a 0 timeout but I would never be able to read any messages despite sending them to the server, so I switched to select and it seemed to work after that.

That select help to non blocking just for Modbus TCP. And about the Modbus RTU, did you think anything to manage a better non blocking? The uasyncio can to do nonblocking UART with asyncio.StreamReader() and asyncio.StreamWriter() instead uart.read() and uart.write(). I think will be enough to non blocking on the RTU.

You're most likely correct, I don't use Modbus RTU and select() probably wouldn't work there. For my usecase it seemed that select() did the job good enough. Implementing asyncio support might be more "correct", but it would also involve making huge changes to my code in addition to the library so I didn't bother with it at the moment.

But now when writing multiple registers, I'm facing an issue where the transaction IDs don't match. I wonder if it'll be worth it to switch over to asyncio or not, but I have really no clue.

I am curious. Why do you use pycopy instead Micropython? Any new/different feature, or your project already running old application with pycopy, or what?
The main feature that I know of is that pycopy had support for minimal versions of the python standard library (via pycopy-lib, e.g. pycopy-selectors) and I couldn't find anything like that for Micropython. In my specific case, I needed support for argparse but I couldn't find anything for Micropython, whereas Pycopy had it so I switched. The two are very similar from what I understand, so I could theoretically switch from one to another just by changing the binary, but I haven't tried that yet.

I mainly work with the UNIX port (my usecase is running hundreds of Modbus clients and servers at the same time, which would have taken ~10GB of memory using CPython+pymodbus, whereas it would probably take <2GB using Pycopy/Micropython), so I didn't work with Modbus RTU, as Pycopy didn't have machine.UART support.

In fact, that's one of the reasons I moved the TCP server to tcp.py in my fork - because I couldn't even import umodbus.modbus without running into ImportErrors, due to the Serial server which I didn't use anyway.

Understood.

Well, I read a bit more about select and it is not works like as non blocking. It just multiplexing I/O for modern OS, like as freertos, and so on. That shows that is non blocking, but it still blocking. Change the timeout can help, but will not solve. You need to create the socket as non blocking sock.setblocking(False). And you can does use asyncio.StreamReader() on that non blocking socket created.

I think the best way is to use uasyncio, for three reasons:

1. Really solve problems with non blocking on the Modbus TCP.
2. Solve problem (with the same approach as TCP) with Modbus RTU.
3. Will works as non blocking in Micropython in MCU that has no operating system, like as STM, RPico, etc that are baremetal. I think select works just where's a Operating system.

Uasyncio is really the best and easy way for non blocking solution for Modbus. Other way for non blocking is to use thread. But uasyncio is in my opinion is better.

EDIT:

As a reference, this project is an unfinished project of modbus RTU and TCP (Slave and Master) using uasyncio :
https://github.com/eydam-prototyping/mp_modbus

[GimmickNG]

Understood.

Well, I read a bit more about select and it is not works like as non blocking. It just multiplexing I/O for modern OS, like as freertos, and so on. That shows that is non blocking, but it still blocking. Change the timeout can help, but will not solve. You need to create the socket as non blocking sock.setblocking(False). And you can does use asyncio.StreamReader() on that non blocking socket created.

Interesting, I didn't know how select works. Thanks, I will look into it more.

I think the best way is to use uasyncio, for three reasons:

1. Really solve problems with non blocking on the Modbus TCP.

2. Solve problem (with the same approach as TCP) with Modbus RTU.

3. Will works as non blocking in Micropython in MCU that has no operating system, like as STM, RPico, etc that are baremetal. I think select works just where's a Operating system.

Uasyncio is really the best and easy way for non blocking solution for Modbus. Other way for non blocking is to use thread. But uasyncio is in my opinion is better.

I agree, async support is ideal. Unfortunately I'm rather hard pressed for time at the moment and so I don't know if I can dedicate much time to implementing async support in the library. Perhaps the maintainer of this repo might be able to, or I might be able to later when I am more free.

EDIT:

As a reference, this project is an unfinished project of modbus RTU and TCP (Slave and Master) using uasyncio : https://github.com/eydam-prototyping/mp_modbus

Interesting, I will take a look at this later. Perhaps it may be simpler to add async support than I had initially assumed, but I haven't worked enough with async in python to say.

[beyonlo]

Hello @GimmickNG

Interesting, I will take a look at this later. Perhaps it may be simpler to add async support than I had initially assumed, but I haven't worked enough with async in python to say.

Yes, I think it is simpler than you already done with select :)

For the ModBus RTU (UART) I think that you need just to use the asyncio.StreamReader(uart) and asyncio.StreamWriter(uart) instead the uart.read() and uart.write(). Here a complete simple non-blocking UART communication example using the uasyncio: https://github.com/peterhinch/micropython-async/blob/master/v3/as_demos/auart.py

For the ModBus TCP you need use sock.setblocking(False), in this case you need to works with socket in this mode.

1. For the TCP Client (ModBus TCP Master) you can see how a non blocking TCP Client is implemented on the MQTT client for MicroPython . Look this line https://github.com/peterhinch/micropython-mqtt/blob/master/mqtt_as/mqtt_as.py#L215 shows the self._sock.setblocking(False)

2. For the TCP Server (ModBus TCP Slave) I do not have a example, but I think that will follow the same way as the in TCP Client (ModBus TCP Master), with sock.setblocking(False) and accepting the TCP clients sockets in that mode.

Here is a complete e detailed tutorial to use uasyncio with MicroPython. There has many examples.

I hope that can help you a bit!

Thank you!

[GimmickNG]

@beyonlo Sorry for the huge delay, was sidetracked on my research and so I didn't have the time to update my fork earlier. However, I have now added asyncio support for the TCP client and server, with some attempt at pulling changes from the newest branch of this repository. Maybe it might be worth checking to see if it meets your needs.

I've also added partial async support for the RTU client, but there is no polling mechanism being used right now for the server like with TCP. Since I am not familiar with UART, this part is left open. Perhaps with some changes it could become fully functional as well.

I am not sure whether to make a pull request or not, since there are quite a few incompatibilities because my fork is aimed at Pycopy instead of Micropython. The first thing to change would be the typing module, there are probably some other things as well that may require changing that I am not aware of.

https://github.com/GimmickNG/pycopy-modbus

[beyonlo]

Hello @GimmickNG

@beyonlo Sorry for the huge delay, was sidetracked on my research and so I didn't have the time to update my fork earlier. However, I have now added asyncio support for the TCP client and server, with some attempt at pulling changes from the newest branch of this repository. Maybe it might be worth checking to see if it meets your needs.

I've also added partial async support for the RTU client, but there is no polling mechanism being used right now for the server like with TCP. Since I am not familiar with UART, this part is left open. Perhaps with some changes it could become fully functional as well.

That's a great news, to have uasyncio support for the ModBus TCP (Slave and Master) and ModBus RTU (Slave and Master) is really an excellent feature - no blocking anymore :)

Currently @brainelectronics are doing a excellent work fixing the bugs on this ModBus lib to have it very commitment with ModBus especification and adding new functions in to have this implementation 100% following the Modbus requirements. I helping him with tests: testing and reporting results for each new RC/PR that he are releasing :) But I think after that is all done and stable, add uasyncio support for this lib will be amazing!

I am not sure whether to make a pull request or not, since there are quite a few incompatibilities because my fork is aimed at Pycopy instead of Micropython. The first thing to change would be the typing module, there are probably some other things as well that may require changing that I am not aware of.

I think that even you are using Pycopy instead MicroPython (that this lib aimed), make a PR in this lib with this amazing feature is very grateful in my opinion, I mean, your contribution is really very important!

Thank you very much!

[brainelectronics]

@GimmickNG thank you so much for your contribution. I've created a milestone to implement and add your changes to this lib

[GimmickNG]

Hello @GimmickNG

I think that even you are using Pycopy instead MicroPython (that this lib aimed), make a PR in this lib with this amazing feature is very grateful in my opinion, I mean, your contribution is really very important!

Thank you very much!

@GimmickNG thank you so much for your contribution. I've created a milestone to implement and add your changes to this lib

Thank you. I apologise for making several changes (regarding the naming of the classes and other parts of the package) that would make it more difficult to integrate them, but I can answer any questions that you may have when doing so.

Perhaps over the next week or two I might create a new branch in my repo which is solely for compatibility with this library, which would make merging a lot easier, provided I find the time to do so.