diff --git a/jekyll/_posts/2018-03-12-steam-api-calls-forwarding.md b/jekyll/_posts/2018-03-12-steam-api-calls-forwarding.md new file mode 100644 index 000000000..438cf6b70 --- /dev/null +++ b/jekyll/_posts/2018-03-12-steam-api-calls-forwarding.md @@ -0,0 +1,572 @@ +--- +title: "Using Nim to forward Steam calls between Linux and Wine" +author: Dmitry Fomichev +--- + +*This is an English version of the [Russian article](https://habrahabr.ru/post/349388/) about the SteamForwarder project written in Nim originally posted on [habrahabr](https://habrahabr.ru/).* + +Players on the GNU/Linux platform have a lot of problems. One of them is the necessity to install an another Steam client for each Wine prefix used for Windows Steam games. The situation is getting worse if we consider the necessity to install a native Steam client for ported and cross-platform games also. +![](https://habrastorage.org/webt/h6/ad/8m/h6ad8m2tfoak1abb7kayqozqb9s.png) +But what if we find a way to use one client for all games? As a basis, we can take a native Steam client, and games for Windows will address it just like, for example, to OpenGL or the sound subsystem of GNU/Linux - through the Wine. The implementation of this approach will be discussed below. + +# In Wine veritas + +Wine can handle Windows libraries in two modes: native and built-in. The native library is perceived by Wine as a file with the extension `* .dll`, which it needs to load into memory and work with, as with the Windows entity. Wine handling all libraries, about which it knows nothing, exactly in this mode. The built-in mode implies that Wine must handle the access to the library in a special way and redirect all calls to a pre-created wrapper with the extension `* .dll.so`, which can access the underlying operating system and its libraries. More information about this can be read [here](https://wiki.winehq.org/Wine_Developer's_Guide/Architecture_Overview). + +Fortunately, most of the interaction with the Steam client occurs just through the library `steam_api.dll`. It means our task is reduced to implementing the wrapper for `steam_api.dll.so`, which will access `steam_api.dll`'s GNU/Linux counterpart - `libsteam_api.so`. + +Creation of such wrapper is a well [documented](https://wiki.winehq.org/Winelib_User%27s_Guide#Building_Winelib_DLLs) process. We need to take the original library for Windows, obtain a spec-file for it using `winedump`, write the implementation of all functions mentioned in the spec-file and compile/link all the sources we have written using `winegcc`. Or ask `winemaker` to handle all the routine work for us. + +# The devil is in the detail + +At first glance, the task is not so difficult. Especially considering that `winedump` can create wrappers automatically if there are header files of the original library. In our case header files are published by Valve for game developers on [the official site](https://partner.steamgames.com/). So, after creating a wrapper through `winedump`, enabling the built-in `steam_api.dll` mode via `winecfg` and compiling, we launched our own Steam, then the game itself and... The game is crashed! + + + + +Judging by the log, our wrapper works (!) exactly until the function `SteamInternal_CreateInterface` is called. What is wrong with it? After reading the documentation and correlating it with the header files, we can find that the function retrurns a pointer to an object of the `SteamClient` class. + +I think that those who are familiar with ABI C ++ already understand the crash origin. The root of the problem is the calling conventions. The C ++ standard does not imply the binary compatibility of programs compiled by different compilers, and in our case the Windows game is compiled by MSVC, while native Steam - by GCC. This problem is not observed for all calls to functions in `steam_api.dll` since they follow the calling conventions for the C language. Once a game receives an instance of the `SteamClient` class from the native Steam and tries to invoke its method (which follows the thiscall convention from C++), an error occurs. To fix the problem, we firstly need to identify key differences in the thiscall calling convention for both of compilers. + + + + + + + + +
MSVCGCC
Puts an object pointer to the ECX registerExpects an object pointer on top of the stack
Expects the stack cleanup by calleeExpects the stack cleanup by caller
+ +[[source](http://www.angelcode.com/dev/callconv/callconv.html#thiscall)] + +At this stage, it's worth making a little digression and mentioning that the attempts to solve the problem have already been made, and even quite successfully. There is a project named [SteamBridge](https://github.com/sirnuke/steambridge), which uses two separate Steam libraries - for Windows and for GNU/Linux. The Windows library is built using MSVC and calls the GNU/Linux library using the Wine. The GNU/Linux library is built using GCC and can call `libsteam_api.so` directly. The calling conventions problem is solved by inserting an assembler snippets at the Windows library side and by wrapping each object when it is passed to the MSVC code. This solution is somewhat redundant, since it requires an additional non-crossplatform compiler to build and introduces an extra entity, but the idea of wrapping the returned objects is robust. We will borrow it! + + +Fortunately for us, Wine already knows how to handle calling conventions. It is sufficient to declare a method with the `thiscall` attribute. Thus, we need to create wrappers of all virtual methods of all classes, and in the each method implementation just perform a call of the method from the original class (which pointer should be stored in the wrapper object). The example wrapper implementation will look like this: + + +```c++ +class ISteamClient_ +{ + public: + virtual HSteamPipe CreateSteamPipe() __attribute__((thiscall)); + ... // other methods + private: + ISteamClient * internal; +} +``` +```c++ +HSteamPipe ISteamClient_::CreateSteamPipe() +{ + TRACE("((ISteamClient *)%p)\n", this); + HSteamPipe result = this->internal->CreateSteamPipe(); + TRACE("() = (HSteamPipe)%p\n", result); + return result; +} +``` + + We also need to perform a similar operation but in oposite direction for classes passed from MSVC code to GCC, namely `CCallback` and` CCallResult`. This task is simple and boring, therefore the best solution is to delegate it to the script for code generation. After several attempts to build everything together, the game begins to work. + + + + +It would seem: the fairy tale is over? And here not! + +# Welcome to the versions hell! + +Very soon it turns out that our design is completely viable only for games compiled using the same header files that we have available. And we only have the latest version of the Steam API. Other versions of headers Valve does not publish (even the latest headers were given under a closed license). On the other hand, our Steam is also of the latest version, but it does not prevent it from working with the old versions of the Steam API. How does it do that? + +The answer is hidden behind this line of the log: `trace:steam_api:SteamInternal_CreateInterface_ ((char *)"SteamClient017")`. It turns out that the client stores information about all classes of all versions of the Steam API, and the `steam_api.dll` only asks the client for an instance of the required class of the desired version. It remains only to find where exactly they are stored. First, let's try the straight approach: to find the "SteamClient016" string in the `libsteam_api.so`. Why not the "SteamClient017" string? Because we need to find a place were stored all the versions of Steam API instead of just the version of `libsteam_api.so` we have. + + +```bash +$ grep "SteamClient017" libsteam_api.so +Binary file libsteam_api.so matches +$ grep "SteamClient016" libsteam_api.so +$ +``` + +It seems that there is nothing similar in `libsteam_api.so`. Then we will try to walk through all the libraries of the Steam client. + +```bash +$ grep "SteamClient017" *.so +Binary file steamclient.so matches +Binary file steamui.so matches +$ grep "SteamClient016" *.so +Binary file steamclient.so matches +$ +``` + +And here is what we need! Let's curtain the Gabe Newell's portraint, if you have one, and open `steamclient.so` in the IDA. A quick keyword search shows an interesting set of strings which follow the pattern: `CAdapterSteamClient0XX`, where XX is the version number. What is even more curious, in the file there are lines with pattern `CAdapterSteamYYYY0XX`, where XX is also the version number, and YYYY is the name of the Steam API class for all other interfaces. The analysis of cross-references allows to find a table of virtual methods for each of the classes with such names easily. Thus, the summary scheme for each class will look like this: + +The class memory layout + +The method table is found, but we have absolutely no information about the signatures of these methods. But this problem turned out to be [solvable](https://toster.ru/answer?answer_id=1156239)(the source in Russian) by calculating the maximum depth of the stack the method tries to access. So we can make an utility that will be receiving the `steamclient.so` library to the input, and forms a list of classes of all versions, as well as their methods at the output. It remains only to generate a wrapper code for the method calls conversion based on the list. The task does not look simple, especially considering that the method signature itself is not known to us, we know only the depth of the stack where the method call arguments end. The situation is aggravated by the peculiarities of the in-memory structures return, namely the presence of a hidden argument - pointer to the memory where the structure should be written. This pointer in all call conventions is extracted from the stack by the callee, so we can easily detect it by the `ret $4` instruction in the assembler code of methods in `steamclient.so`. But even so, the amount of non-trivial code generation is huge. + +# The Hero appears + +To any new or just not very popular programming language, the question of its niche first of all arises. Nim is no exception. It is often criticized for trying to "sit on all chairs at once", implying a big amount of features in the absence of one clear direction of development. Among such features it is possible to distinguish two: + +- compilation to C and, as a consequence, easy cross-platform compilation; + +- excellent metaprogramming support (the same language for both of run-time and compile-time code, direct manipulation with AST ). + + +This combination will make the wrapper writing process painless. +First, we create the main file `steam_api.nim` and a file with compilation options: `steam_api.nims` + + + + + + + +It does not look very simple, but it's only because we swung a lot at once. There is cross-compilation, importing functions from C header files and Wine specific compiler options... Despite the seeming complexity, nothing complicated has happened, we just directly implemented some parts of the source code in C that Nim does not know anything about, and at the same time we described for Nim how to call the TRACE macro from the Wine header files (we also told it about these files). + +Now let's go to the most delicious part - to the code generation. Since we do not have complete information about method signatures, we will emulate instances of classes in C code, fortunately we only need to emulate the virtual method table. So, let's imagine that we have a file that describes the methods and classes of the Steam API as follows: + +``` +!CAdapterSteamYYY0XX +[+] +[+] +... +``` + +The `+` sign is optional and will serve as an indicator of the hidden argument for the in-memory return. +Such a file can be obtained by parsing `steamclient.so`. We should get a table from it. The keys of the table are lines following the `CAdapterSteamYYYY0XX` pattern, and the values are arrays of functions that call corresponding methods in the object, which is the field of the wrapper structure implicitly passed to them via the `ECX` register. It is not very convenient to write all this methods in assembler, especially considering that it would be nice to add some kind of journaling, so let's find the minimum assembler fragment: + + + + +{% highlight asm %} +push %ecx # put an object pointer to the stack (it will become the second argument) +push $ # put the method number to the stack (it will be the very first argument) +# the remaining arguments will move to 3 (two previous and the return addresses) +call # call the function written on Nim +add $0x4, %esp # remove the method number from the stack +pop %ecx # remove the object pointer +ret $ # remove the arguments from the stack and return +{% endhighlight %} + + + + + + + +It remains to generate the Nim functions we call in snippet. It is necessary to generate one function for each stack depth encountered in the file and one more for calls with a hidden argument. Further, we call these functions pseudomethods for brevity. + + + + +Lets leave the implementation of the `wrapIfNecessary` function behind the brackets and proceed to the description of the code that generates the fragments described above. First, read the file with class descriptions. We will get the path to the file in the same way as we got the path to the spec-file - via the compiler option. + + + + +Now we've got a class table. Since the `readClasses` function does not use anything that is possible only at runtime, we can safely compute it at compile time and write the result to a constant like that:`const classes = readClasses ()`. Let's create a table of methods-wrappers, consisting of assembler inserts, described above. + + + + +Based on the lists obtained, we construct pseudomethods. The lists enumeration process is left behind. It's also worth noting that all the procedures we used are the usual Nim functions that operate the AST and are called from the body of the macro (which will be described later). The magic of the interpretation of the created ASTs occurs when you leave the body of the macro. + + + + +The hardest part is behind. Its complexity is caused by the necessity to form and insert a dynamic list of arguments into several key points of the pseudomethod declaration. A simple approach with a template and substitution through the quote does not work here, so you have to assemble the nodes of the AST one by one, which negatively affects the amount and readability of the code. It remains to write the macro itself, from which our AST generators will be called. + + + + +Похожим образом создаются объявления основных функций `steam_api.dll`. Для проброса вызовов обратно из GNU/Linux в игру форма уже известна и едина для всех версий Steam API, поэтому нет нужды в кодогенерации. Например, определение первого метода будет выглядеть так: +Similarly, the function declarations from `steam_api.dll` are created. To forward calls back from GNU/Linux into the game, the mechanism is already known and unified for all versions of the Steam API, so there is no need for code generation. For example, the definition of the first method would look like this: + + + + +# Conclusion + +So, we've covered the main key points that allow us to generate a wrapper for the Steam API at compile time. No matter how complex they seem, this approach undoubtedly wins in comparsion to the manual writing of several hundred similar methods. Nim wrote all these methods for us. Someone may ask: "And what about the debugging of all this code?". The issue of debugging the compile time code is really complicated. The only tool available is the good old debugging messages `echo`. Fortunately, Nim has the functions `repr` and` treeRepr`, which turn the AST into lines of code and a string with an AST nodes diagram, respectively, which greatly simplifies debugging. + +Particularly noteworthy is the flexibility of the Nim compiler. Compiling in C, combined with high-end support for metaprogramming, allows it to be considered as both of a super-powerful preprocessor for C and a separate language compiler that is not inferior to C's capabilities in a wrapper of nice python-like syntax. + +Perhaps, the article will seem too chaotic, since it is not easy to describe a complex task and its solution, in which the language is revealed at full power, in a simple and concise manner. Unfortunately, within the framework of this article, it was not possible to describe a few more aspects, namely: + +- the function `wrapIfNeccessary` and the mechanism for determining the name of the object by the pointer; + +- the formation of a wrapper class based on the methods described; + +- interaction with Steam to download the game; + +- details of implementation wrapper functions from `steam_api.dll` (the article was only about virtual methods); + +- utilities for the analysis of `steamclient.so` and` libsteam_api.so`, emulation of the stack behavior; + +- pitfalls and problems that arose when searching for solutions described in the article (garbage collector, ignoring `asmNoStackFrame` pragma, old versions of the compiler). + +Such details, in my opinion, would worsen the perception even more. In addition, the article does not describe the actual course of research and the solution of the problem, but merely represents the reconstruction of the solution in favor of the integrity of the narrative. + +The working solution of the problem identified in the header is presented in the repository on github: +- here you can find [the C++ solution without Nim macros but with codegenerator scripts implemented in Nim](https://github.com/xomachine/SteamForwarder/tree/8f2b8cea17da8718dfd8a87fbd2677d475abb54a), which requires Steam API headers and works with only one version of Steam API; +- here you can find [the Nim solution](https://github.com/xomachine/SteamForwarder/tree/a7dea4b6a87086b93d4165090d85ec8134985962) which described in second half of the article. + +Some names of variables and functions in the original code differ from the examples given in the article. Links are given on the commit of each branch, which is the top at the time of publication, so as not to lose relevance with time. + +I hope the article will give additional interest to the Nim programming language and show readers that it is possible to write something more complicated than `echo "Hello, world!"`. diff --git a/jekyll/assets/news/images/winesteam/ClassLayoutEng.svg b/jekyll/assets/news/images/winesteam/ClassLayoutEng.svg new file mode 100644 index 000000000..728b1bb2f --- /dev/null +++ b/jekyll/assets/news/images/winesteam/ClassLayoutEng.svg @@ -0,0 +1,751 @@ + +image/svg+xmlCAdapterSteamYYY0XX +Class Info +Method 1 +Method 2 +... +0x00000008 +0xXXXXXXXX + \ No newline at end of file