Reference implementation of HERACLES Assembler tools (#75)

* Reference implementation of the HERACLES Assembler tools
---------

Signed-off-by: Flavio Bergamaschi <[email protected]>

Co-authored-by: Flavio Bergamaschi <[email protected]>
Co-authored-by: Ernesto Zamora Ramos <[email protected]>
Co-authored-by: Chris Wilkerson <[email protected]>
Co-authored-by: Jose Rojas Chaves <[email protected]>
Co-authored-by: Kylan Race <[email protected]>
Co-authored-by: Christopher Gutierrez <[email protected]>
Co-authored-by: Hamish Hunt <[email protected]>
Co-authored-by: Jack Crawford <[email protected]>
Co-authored-by: Minxuan Zhou <[email protected]>
Co-authored-by: Wen Wang <[email protected]>
Co-authored-by: Adish Vartak <[email protected]>
Co-authored-by: Michael Steiner <[email protected]>
Co-authored-by: Avinash Alevoor <[email protected]>
Co-authored-by: Rashmi Agrawal <[email protected]>

Author: 15 people

.gitignore

@@ -0,0 +1,205 @@
+#========================
+# Intermediate and Output files
+#========================
+
+*.tmp
+*.temp
+*.mem
+*.minst
+*.cinst
+*.xinst
+*.csv
+*.out
+#*#
+*~
+tmp/
+
+
+# Local files
+#========================
+
+*.yml
+*.pyc
+*.bak
+*.pkl
+*.lock
+*.swp
+tfedlrn.egg-info/
+bin/out
+
+#========================
+# Generated docs
+#========================
+
+*.htm
+*.html
+*.pdf
+html
+latex
+[Dd]ocs/
+
+
+#========================
+# Eclipse & PyDev intermediate files
+#========================
+
+.metadata/
+RemoteSystemsTempFiles/
+.settings
+.project
+.pydevproject
+
+#========================
+# Visual Studio
+#========================
+
+# User-specific files
+*.rsuser
+*.suo
+*.user
+*.userosscache
+*.sln.docstates
+
+# User-specific files (MonoDevelop/Xamarin Studio)
+*.userprefs
+
+# Mono auto generated files
+mono_crash.*
+
+# Visual Studio cache/options directory
+.vs/
+.vscode/
+
+#========================
+# Python general stuff
+#========================
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/

.pre-commit-config.yaml

@@ -38,6 +38,10 @@ repos:
           - HEADER
           - --comment-style
           - //                   # defaults to: #
+        - id: remove-tabs
+          name: remove-tabs
+          files: \.(py)$
+          args: [--whitespaces-count, '4']
     - repo: https://github.com/psf/black-pre-commit-mirror
       rev: 24.4.0 # Updated 2024/04
       hooks:

assembler_tools/empty_file.txt

@@ -0,0 +1,100 @@
+# HERACLES Code Generation Framework (Reference Implementation)
+
+The tools in this directory are the reference implementation of and Assembler codegenerator that takes a pre-generated Polynomial Instruction Set Architecture (P-ISA) program containing instructions that use an abstract, flat memory model for polynomial operations, such as those applied in homomorphic encryption (HE), and maps them to a corresponding set of instructions compatible with the HERACLES architecture, accounting for hardware restrictions, including memory management for the HERACLES memory model.
+
+## Table of Contents
+1. [Dependencies](#dependencies)
+2. [Inputs](#as_inputs)
+3. [Outputs](#as_outputs)
+   1. [Assembler Instruction Specs](#asm_specs)
+4. [Executing the Assembler](#executing_asm)
+   1. [Running for a Pre-Generated Kernel](#executing_single)
+5. [Debug Tools](./debug_tools/README.md)
+
+## Dependencies <a name="dependencies"></a>
+
+This project is Python based. Dependencies for the project are as follows:
+
+- Python 3.10 or newer (Tested with 3.10)
+- Python pip 22.0 or newer
+- Requirements for the Python environment listed in file `requirements.txt`.
+
+  Install with `pip`:
+
+  ```bash
+  pip3 install -r requirements.txt
+  ```
+
+It is recommended to install Python dependencies and run inside a [Python virtual environment](https://virtualenv.pypa.io/en/stable/index.html) to avoid polluting the global Python.
+
+## Inputs <a name="as_inputs"></a>
+
+The assembler framework requires two inputs:
+
+- **Abstract P-ISA kernel**: a kernel of instructions in abstract P-ISA where the memory model is abstracted with a flat, infinite structure.
+
+- **Memory mapping metadata**: metadata information indicating the location where variable identifiers from the kernel are stored in the HERACLES HBM.
+
+Kernels and metadata are structured in comma-separated value (csv) files.
+
+P-ISA kernels, along with corresponding memory metadata required as input to the assembler, are generated by the upper layers in the Encrypted Computing SDK stack, e.g. the [Program Mapper](../../README.md#encrypted-computing-sdk-phase-1-components-and-tasks) component of the [p-ISA tools](../../p-isa_tools).
+
+## Outputs <a name="as_outputs"></a>
+
+On a successful run, given a P-ISA kernel in file `filename.csv` (and corresponding memory metadata file), the assmebler generates three files:
+
+- `filename.minst`: contains the list of instructions for the MINST queue.
+- `filename.cinst`: contains the list of instructions for the CINST queue.
+- `filename.xinst`: contains the list of instructions for the XINST queue.
+
+### Assembler Output Instruction Specs <a name="asm_specs"></a>
+
+The format for the output files and instruction set can be found at [HCGF Instruction Specification](docsrc/specs.md).
+
+## Executing the Assembler <a name="executing_asm"></a>
+### Running for a Pre-Generated Kernel <a name="executing_single"></a>
+
+Given a P-ISA kernel (`filename.csv`) and corresponding memory mapping file (`filename.mem`), there are three steps to assemble them into HERACLES code.
+
+1. Pre-process the P-ISA input kernel using `he_prep.py`.
+
+```bash
+# pre-process kernel: outputs filename.tw.csv
+python3 he_prep.py filename.csv
+```
+
+2. Assemble the pre-processed result using `he_as.py`.
+
+```bash
+# use output from pre-processing as input to asm:
+# outputs filename.tw.minst, filename.tw.cinst, filename.tw.xinst
+python3 he_as.py filename.tw.csv --input_mem_file filename.mem
+```
+
+3. Link the assembler output into a HERACLES program using `he_link.py`.
+
+```bash
+# link assembled output (input prefix: filename.tw)
+# outputs filename.minst, filename.cinst, filename.xinst
+python3 he_link.py filename.tw --input_mem_file filename.mem --output_prefix filename
+```
+
+This will generate the main three output files in the same directory as the input file:
+
+- `filename.minst`: contains the list of instructions for the MINST queue.
+- `filename.cinst`: contains the list of instructions for the CINST queue.
+- `filename.xinst`: contains the list of instructions for the XINST queue.
+
+Intermediate files, if any, are kept as well.
+
+The linker program is able to link several assembled kernels into a single HERACLES program, given a correct memory mapping for the resulting program.
+
+This version of executing is intended for the assembler to be usable as part of a compilation pipeline.
+
+Use commands below for more configuration information, including changing output directories, input and output filenames, etc.
+
+```bash
+python3 he_prep.py -h
+python3 he_as.py -h
+python3 he_link.py -h
+```

assembler_tools/hec-assembler-tools/README.md

@@ -0,0 +1,13 @@
+import os
+
+def makeUniquePath(path: str) -> str:
+    """
+    Returns a unique, normalized, and absolute version of the given file path.
+
+    Args:
+        path (str): The file path to be processed.
+
+    Returns:
+        str: A unique, normalized, and absolute version of the input path.
+    """
+    return os.path.normcase(os.path.realpath(os.path.expanduser(path)))

assembler_tools/hec-assembler-tools/assembler/common/__init__.py

@@ -0,0 +1,30 @@
+
+class GlobalConfig:
+    """
+    A configuration class for controlling various aspects of the assembler's behavior.
+
+    Attributes:
+        suppressComments (bool): 
+            If True, no comments will be emitted in the output generated by the assembler.
+        
+        useHBMPlaceHolders (bool): 
+            Specifies whether to use placeholders (names) for variable locations in HBM 
+            or the actual variable locations.
+        
+        useXInstFetch (bool): 
+            Specifies whether `xinstfetch` instructions should be added into CInstQ or not. 
+            When no `xinstfetch` instructions are added, it is assumed that the HERACLES 
+            automated mechanism for `xinstfetch` will be activated.
+        
+        debugVerbose (int): 
+            If greater than 0, verbose prints will occur. Its value indicates how often to 
+            print within loops (every `debugVerbose` iterations). This is used for internal 
+            debugging purposes.
+        hashHBM (bool): Specifies whether the target architecture has HBM or not.
+    """
+
+    suppressComments = False
+    useHBMPlaceHolders = True
+    useXInstFetch = True
+    debugVerbose: int = 0
+    hasHBM = True