diff --git a/docs/about.rst b/docs/about.rst new file mode 100644 index 00000000..39c02cc8 --- /dev/null +++ b/docs/about.rst @@ -0,0 +1,32 @@ +About Page +=========== + + +Report +------- + +- Report problems - Servicex channel + + +Outreach +--------- + + +Information available in the HEP community. + +Presentations +~~~~~~~~~~~~~~ + +List of talks/posters to the HEP community on ServiceX + +- `IRIS-HEP Poster Session 2020 at Princeton University `_ +- `ServiceX talk at CHEP 2019 `_ +- `ServiceX talk at HSF DAWG – DOMA Access meeting `_ + +Tutorials +~~~~~~~~~ + +A link to our tutorials. We currently have two: + +- `A streaming example for xAOD files using func_adl `_ +- `An example using hep_tables for the frontend `_ \ No newline at end of file diff --git a/docs/conf.py b/docs/conf.py index adab8a5a..36501785 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -17,7 +17,9 @@ extensions = [ 'sphinx.ext.napoleon', 'sphinx.ext.intersphinx', - 'sphinx.ext.viewcode' + 'sphinx.ext.viewcode', + 'sphinx.ext.doctest', + 'code_include.extension' ] templates_path = ['_templates'] diff --git a/docs/configuration.rst b/docs/configuration.rst index aaf88e02..e52db1ae 100644 --- a/docs/configuration.rst +++ b/docs/configuration.rst @@ -1,6 +1,7 @@ Configuration ============= -.. autoclass:: servicex_client.configuration.Configuration + +.. autoclass:: servicex.configuration.Configuration :members: :exclude-members: Config :undoc-members: @@ -8,7 +9,8 @@ Configuration Endpoint -------- -.. autoclass:: servicex_client.configuration.Endpoint + +.. autoclass:: servicex.configuration.Endpoint :members: :undoc-members: :show-inheritance: diff --git a/docs/contribute.rst b/docs/contribute.rst new file mode 100644 index 00000000..0f5c637a --- /dev/null +++ b/docs/contribute.rst @@ -0,0 +1,191 @@ +ServiceX Contributor Guide +=========================== + +Welcome to the ServiceX contributor guide, and thank you for your interest in contributing to the project! + +Overview +-------- + +ServiceX uses a microservice architecture, +and is designed to be hosted on a Kubernetes cluster. +The ServiceX project uses a polyrepo strategy for source code management: +the source code for each microservice is located in a dedicated repo. + +Below is a partial list of these repositories: + +- `ServiceX `_ - Main repository, contains Helm charts for deployment to Kubernetes +- `ServiceX_frontend `_ - The ServiceX Python library, which enables users to send requests to ServiceX. Currently, this is the only ServiceX frontend client. +- `ServiceX_App `_ - The ServiceX API Server, written in Flask. + +Additional repositories related to the project can be found in the `ssl-hep GitHub organization `_. + +Please read our `architecture document `_ for more details. + +Branching Strategy +------------------- + +ServiceX uses a slightly modified GitLab flow. Each repository has a main branch, usually named `develop` (or `master` for the Python frontend). All changes should be made on feature branches and submitted as PRs to the main branch. Releases are frozen on dedicated release branches, e.g. `v1.0.0-RC.2`. + +Development Workflow +--------------------- + +1. Set up a local development environment: + - Decide which microservice (or Helm chart) you'd like to change, and locate the corresponding repository. + - If you are a not a member of the ``ssl-hep`` GitHub organization, fork the repository. + - Clone the (forked) repository to your local machine: + + .. code-block:: bash + + git clone git@github.com:/ServiceX_App.git + + - If you created a fork, add the upstream repository as remote: + + .. code-block:: bash + + git remote add upstream git@github.com:ssl-hep/ServiceX_App.git + + - Set up a new environment via ``conda`` or ``virtualenv``. + - Install dependencies, including test dependencies: + + .. code-block:: bash + + python3 -m pip install -e .[test] + + - If the root directory contains a file named ``.pre-commit-config.yaml``, you can install the `pre-commit `_ hooks with: + + .. code-block:: bash + + pip install pre-commit + pre-commit install + +2. Develop your contribution: + - Pull latest changes from upstream: + + .. code-block:: bash + + git checkout develop + git pull upstream develop + + - Create a branch for the feature you want to work on: + + .. code-block:: bash + + git checkout -b fix-issue-99 + + - Commit locally as you progress with ``git add`` and ``git commit``. + +3. Test your changes: + - Run the full test suite with ``python -m pytest``, or target specific test files with ``python -m pytest tests/path/to/file.py``. + - Please write new unit tests to cover any changes you make. + - You can also manually test microservice changes against a full ServiceX deployment by building the Docker image, pushing it to DockerHub, and setting the `image` and `tag` values as follows: + + .. code-block:: yaml + + app: + image: / + tag: my-feature-branch + + - For more details, please read our full `deployment guide `_. + +4. Submit a pull request to the upstream repository + + +Issues +------ + +Please submit issues for bugs and feature requests to the `main ServiceX repository `_, unless the issue is specific to a single microservice. + +We manage project priorities with a `ZenHub board `_. + +Join us on Slack +----------------- + +We coordinate our efforts on the `IRIS-HEP Slack `_. +Come join this intellectual hub! + +Running the Full ServiceX Chart Locally +---------------------------------------- + +You can run ServiceX on your laptop using `docker` or another similar tool that supports kubernetes. + +Prerequisites +-------------- + +1. ``docker`` is installed and ``kubernetes`` is running (see configuration options). +2. Make sure ``kubectl`` and ``helm`` are both installed in the shell you'll be doing your development work. +3. Follow instructions in the deployment guide to install your x509 certificate if you are going to be using any `rucio` or GRID services for your testing. + +Running the chart +------------------ + + +1. In the ``Servicex/helm`` directory run ``helm dependency update servicex/`` +2. And install the chart with ``helm install -f values.yaml servicex-testing .\servicex\`` +3. As in the deployment guide, you can now port-forward your servicex ``app`` and ``minio``. + +How you write your ``values.yaml`` will depend a lot on what you are testing. Here is an example of a minimal one that will load up the `develop` tag for all the container images, and expects an ATLAS GRID cert: + +.. code-block:: yaml + + postgres: + enabled: true + objectStore: + publicURL: localhost:9000 + + gridAccount: + + x509Secrets: + # For ATLAS + vomsOrg: atlas + + app: + ingress: + host: localhost:5000 + + transformer: + cachePrefix: '""' + + +Making Changes +--------------- + + +The best way to work on ServiceX is using the unit tests. That isn't always possible, of course. When it isn't your development cycle will require you to build any changed containers. A possible workflow is: + +1. Redeploy the ``helm`` chart (or perhaps use ``upgrade`` rather than ``install`` in the ``helm`` command) and add ``pullPolicy: Never`` to the appropriate app section. For example, add it under ``app:`` in the example file above if you are working on ``servicex_app``. +2. Change your code (say, in ``servicex_app``). +3. In the directory for the app should be a ``Dockerfile``. Do the build, and pay attention to the tag. For example, ``docker build -t sslhep/servicex_app:develop .``. +4. Finally restart the pod, which should cause it to pick up the new build. This might kill a port-forward you have in place, so don't forget to restart that! + +Debugging Tips +--------------- + +Microservice architectures can be difficult to test and debug. Here are some +helpful hints to make this easier. + +1. Instead of relying on the DID Finder to locate some particular datafile, you +can mount one of your local directories into the transformer pod and then +instruct the DID Finder to always offer up the path to that file regardless of +the submitted DID. You can use the ``hostMount`` value to have a local directory +mounted into each transformer pod under ``/data``. You can use the +``didFinder.staticFile`` value to instruct DID Finder to offer up a file from that +directory. +2. You can use port-forwarding to expose port 15672 from the RabbitMQ pod to +your laptop and log into the Rabbit admin console using the username: ``user`` and +password ``leftfoot1``. From here you can monitor the queues, purge old messages +and inject your own messages + +Notes for Maintainers +--------------------- + +Hotfixes +-------- + +If a critical bugfix or hotfix must be applied to a previous release, it should be merged to the main branch and then applied to each affected release branch using + +.. code-block:: bash + + git cherry-pick -m 1 + +Merge commits have 2 parents, so the ``-m 1`` flag is used to specify that the first parent (i.e. previous commit on the main branch) should be used. + diff --git a/docs/databinder.rst b/docs/databinder.rst new file mode 100644 index 00000000..39885139 --- /dev/null +++ b/docs/databinder.rst @@ -0,0 +1,2 @@ +DataBinder +---------- \ No newline at end of file diff --git a/docs/dataset.rst b/docs/dataset.rst index c20a0514..dffaeb51 100644 --- a/docs/dataset.rst +++ b/docs/dataset.rst @@ -1,6 +1,6 @@ Dataset Class ============= -.. autoclass:: servicex_client.dataset.Dataset +.. autoclass:: servicex.dataset.Dataset :members: :exclude-members: download_files, generate_selection_string, retrieve_current_transform_status, transform_request, transform_status_listener :undoc-members: diff --git a/docs/dataset_group.rst b/docs/dataset_group.rst index f72167ed..b1247e73 100644 --- a/docs/dataset_group.rst +++ b/docs/dataset_group.rst @@ -1,6 +1,6 @@ DatasetGroup ============ -.. autoclass:: servicex_client.dataset_group.DatasetGroup +.. autoclass:: servicex.dataset_group.DatasetGroup :members: :undoc-members: :show-inheritance: \ No newline at end of file diff --git a/docs/dataset_identifier.rst b/docs/dataset_identifier.rst index c2a74e8b..09c6bf05 100644 --- a/docs/dataset_identifier.rst +++ b/docs/dataset_identifier.rst @@ -1,11 +1,11 @@ DatasetIdentifier Module ======================== -.. autoclass:: servicex_client.dataset_identifier.RucioDatasetIdentifier +.. autoclass:: servicex.dataset_identifier.RucioDatasetIdentifier :members: :undoc-members: :show-inheritance: -.. autoclass:: servicex_client.dataset_identifier.FileListDataset +.. autoclass:: servicex.dataset_identifier.FileListDataset :members: :exclude-members: populate_transform_request :undoc-members: diff --git a/docs/examples.rst b/docs/examples.rst index aa5504a5..11491b98 100644 --- a/docs/examples.rst +++ b/docs/examples.rst @@ -71,3 +71,11 @@ returns an awkward array sx3 = ds.with_uproot_function(run_query).as_pandas() print(sx3) + + +Bigger Uproot +--------------------- +.. code-include :: :func:`examples.bigger_uproot.bigger_uproot` + + + diff --git a/docs/getting_started.rst b/docs/getting_started.rst new file mode 100644 index 00000000..68c5934a --- /dev/null +++ b/docs/getting_started.rst @@ -0,0 +1,148 @@ +Getting Started +=============== + +- Introduce the client with a minimal example, working through line by line (pydantic version) +- Top-level explanation DataBinder configuration yaml + + +First request +-------------- + +Once you've been approved, you're ready to go! + +You can interact with ServiceX by making a transformation request. A transformation request includes the following information: + +- An input dataset +- Filters to be applied +- Computation of new columns (if any) +- Columns to be returned to the user + +Below are some basic examples which you can run to confirm that ServiceX is working for you. + +xAOD +~~~~~ + +.. code-block:: python + + from servicex import ServiceXClient, RucioDatasetIdentifier, ResultFormat + from func_adl_servicex_xaodr22.event_collection import Event + from func_adl_servicex_xaodr22 import calib_tools + + # A Z to ee sample - Release 21 + ds_name = ( + r"mc16_13TeV:mc16_13TeV.361106.PowhegPythia8EvtGen_AZNLOCTEQ6L1_Zee" + r".deriv.DAOD_PHYS.e3601_e5984_s3126_r10201_r10210_p5313") + + sx = ServiceXClient(backend="uc-af") + did = RucioDatasetIdentifier(ds_name, num_files=10) + + ds_raw = sx.func_adl_dataset( + did, codegen="atlasr21", title="Zee", result_format=ResultFormat.parquet, item_type=Event) + + # ds = calib_tools.apply_calibration(ds_raw, "PHYS") + ds = calib_tools.query_update(ds_raw, calib_tools.default_config("PHYSLITE")) + + good_ele = ds.Select( + lambda e: { + "run": e.EventInfo("EventInfo").runNumber(), + "event": e.EventInfo("EventInfo").eventNumber(), + "good_ele": e.Electrons("Electrons") + .Where(lambda e: (e.pt() / 1000 > 25.0) and (abs(e.eta()) < 2.5)), + } + ) + + electron_pt = good_ele.Select(lambda e: { + "run": e.run, + "event": e.event, + "pt": e.good_ele.Select(lambda ele: ele.pt() / 1000.0), + }) + + r = electron_pt.as_signed_urls() + print(f"number of URLs: {len(r.signed_url_list)}") + + +Expected output: + +.. code-block:: python + + JetPt + entry + 0 36.319766 + 1 34.331914 + 2 16.590844 + 3 11.389335 + 4 9.441805 + ... ... + 857133 6.211655 + 857134 47.653145 + 857135 32.738951 + 857136 6.260789 + 857137 5.394783 + + [11355980 rows x 1 columns] + + +uproot +~~~~~~~ + +Instead of a rucio dataset, here we will use a file directly available over `https`, +and a slightly more complex query, and we'll ask for the data to be locally downloaded +so we can access the files directly. + +.. code-block:: python + + + import ast + + import qastle + + from servicex import ServiceXSpec, General, Sample + from servicex.func_adl.func_adl_dataset import FuncADLQuery + from servicex.servicex_client import deliver + + query = FuncADLQuery().Select(lambda e: {'lep_pt': e['lep_pt']}). \ + Where(lambda e: e['lep_pt'] > 1000) + + qstr = """ + FuncADLDataset().Select(lambda e: {'lep_pt': e['lep_pt']}). \ + Where(lambda e: e['lep_pt'] > 1000) + """ + query_ast = ast.parse(qstr) + qastle_query = qastle.python_ast_to_text_ast(qastle.insert_linq_nodes(query_ast)) + print("From str", qastle_query) + q2 = FuncADLQuery() + q2.set_provided_qastle(qastle_query) + print(q2.generate_selection_string()) + print("From python", query.generate_selection_string()) + spec = ServiceXSpec( + General=General( + ServiceX="testing1", + Codegen="uproot", + OutputFormat="parquet", + Delivery="LocalCache" + ), + Sample=[ + Sample( + Name="mc_345060.ggH125_ZZ4lep.4lep", + XRootDFiles="root://eospublic.cern.ch//eos/opendata/atlas/OutreachDatasets/2020-01-22/4lep/MC/mc_345060.ggH125_ZZ4lep.4lep.root", # NOQA E501 + Query=query + ) + ] + ) + + print(deliver(spec)) + + + +Expected output: + +.. code-block:: python + + [{pt: [36.3, 24.7], eta: [2.87, 3.13], phi: [, ... -2.15], mass: [12.3, 6.51, 3.98]}] + 349 + + +Next steps +----------- + +Check out the [requests guide](requests.md) to learn more about specifying transformation requests using func-ADL. \ No newline at end of file diff --git a/docs/img/ServiceX-BlackTextOnly-Transparent.png b/docs/img/ServiceX-BlackTextOnly-Transparent.png new file mode 100644 index 00000000..82e2e076 Binary files /dev/null and b/docs/img/ServiceX-BlackTextOnly-Transparent.png differ diff --git a/docs/img/ServiceX-Color-ImageOnly-Transparent.png b/docs/img/ServiceX-Color-ImageOnly-Transparent.png new file mode 100644 index 00000000..60638024 Binary files /dev/null and b/docs/img/ServiceX-Color-ImageOnly-Transparent.png differ diff --git a/docs/img/ServiceX-Color-Transparent.png b/docs/img/ServiceX-Color-Transparent.png new file mode 100644 index 00000000..5fc7069e Binary files /dev/null and b/docs/img/ServiceX-Color-Transparent.png differ diff --git a/docs/img/ServiceX-WhiteText-Transparent.png b/docs/img/ServiceX-WhiteText-Transparent.png new file mode 100644 index 00000000..e6e4fe3a Binary files /dev/null and b/docs/img/ServiceX-WhiteText-Transparent.png differ diff --git a/docs/img/ServiceX-WhiteTextOnly-Transparent.png b/docs/img/ServiceX-WhiteTextOnly-Transparent.png new file mode 100644 index 00000000..5049aacf Binary files /dev/null and b/docs/img/ServiceX-WhiteTextOnly-Transparent.png differ diff --git a/docs/img/ServiceX-architecture.png b/docs/img/ServiceX-architecture.png new file mode 100644 index 00000000..53dc9af9 Binary files /dev/null and b/docs/img/ServiceX-architecture.png differ diff --git a/docs/img/ServiceX-request-lifecycle.png b/docs/img/ServiceX-request-lifecycle.png new file mode 100644 index 00000000..1af5f966 Binary files /dev/null and b/docs/img/ServiceX-request-lifecycle.png differ diff --git a/docs/img/Z_ee_example.jpg b/docs/img/Z_ee_example.jpg new file mode 100644 index 00000000..521825ce Binary files /dev/null and b/docs/img/Z_ee_example.jpg differ diff --git a/docs/img/develop.png b/docs/img/develop.png new file mode 100644 index 00000000..6dbeadf2 Binary files /dev/null and b/docs/img/develop.png differ diff --git a/docs/img/download-servicex-yaml.jpg b/docs/img/download-servicex-yaml.jpg new file mode 100644 index 00000000..636e9d7a Binary files /dev/null and b/docs/img/download-servicex-yaml.jpg differ diff --git a/docs/img/hep_tables_example.png b/docs/img/hep_tables_example.png new file mode 100644 index 00000000..368ff328 Binary files /dev/null and b/docs/img/hep_tables_example.png differ diff --git a/docs/img/organize.png b/docs/img/organize.png new file mode 100644 index 00000000..8b1dcedf Binary files /dev/null and b/docs/img/organize.png differ diff --git a/docs/img/organize2.png b/docs/img/organize2.png new file mode 100644 index 00000000..9b5fe0ca Binary files /dev/null and b/docs/img/organize2.png differ diff --git a/docs/img/sx-architecture.png b/docs/img/sx-architecture.png new file mode 100644 index 00000000..af3a7641 Binary files /dev/null and b/docs/img/sx-architecture.png differ diff --git a/docs/img/sx-schema.png b/docs/img/sx-schema.png new file mode 100644 index 00000000..f59ac715 Binary files /dev/null and b/docs/img/sx-schema.png differ diff --git a/docs/index.rst b/docs/index.rst index 51714c5c..939cf04a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -8,6 +8,11 @@ ServiceX Client Python SDK and CLI Client for ServiceX +Introduction +------------ + +Why do + Configuration ------------- @@ -177,13 +182,16 @@ also come with synchronous versions to make it easy to do easy things. :maxdepth: 2 :caption: Contents: + introduction + installation + getting_started + transformer_matrix + databinder examples - servicex_client - dataset_identifier - dataset - dataset_group - models - configuration + contribute + troubleshoot + about + modules Indices and tables ================== diff --git a/docs/installation.rst b/docs/installation.rst new file mode 100644 index 00000000..043c06f1 --- /dev/null +++ b/docs/installation.rst @@ -0,0 +1,67 @@ +Installation +============ + +Prerequisites +------------- + +Python 3.6, 3.7, or 3.8 + +Member of the ATLAS or CMS collaborations + +Installation +------------- + + +.. code-block:: bash + + pip install servicex-clients + +This is an umbrella package which includes all of the frontend client libraries used to communicate with a ServiceX backend. There are multiple ways to specify a request, but we will use func-ADL in this tutorial. + +Selecting an endpoint +---------------------- + +To use the ServiceX client, you will need a ServiceX API token issued by a +ServiceX backend instance. Each ServiceX instance is dedicated to a specific +experiment and file format. +You can use one of the centrally-managed instances of ServiceX running on the +University of Chicago's Analysis Facility cluster: + +.. list-table:: + :widths: 40 20 10 20 + :header-rows: 1 + + * - Endpoint + - Collaboration + - Type + - Input + * - ``_ + - ATLAS + - xaod + - xAOD files + * - ``_ + - ATLAS + - uproot + - Flat ntuples + + +You can view the status of these production servers along with our current +development servers by viewing the `Server Status Dashboard `_. + +Visit the instance that meets your needs. Click on the **Sign-in** button in the +upper right hand corner. You will be asked to authenticate via GlobusAuth and +complete a registration form. Once this form is submitted, it will be reviewed +by SSL staff. You will receive an email upon approval. + +At this time you may return to the ServiceX page. Click on your name in the +upper right hand corner and then select **Profile** tab. Click on the download +button to have a servicex.yaml file generated with your access token and +downloaded to your computer. + +.. image:: img/download-servicex-yaml.jpg + :alt: Download button + +You may place this in your home directory or within +the `servicex_frontend search path `_. + +The remainder of this guide will use the xAOD instance. \ No newline at end of file diff --git a/docs/introduction.rst b/docs/introduction.rst new file mode 100644 index 00000000..b7225bc5 --- /dev/null +++ b/docs/introduction.rst @@ -0,0 +1,28 @@ +Introduction +============ + +The High Luminosity Large Hadron Collider (HL-LHC) faces enormous computational challenges in the +2020s. The HL-LHC will produce exabytes of data each year, with increasingly complex event +structure due to high pileup conditions. The ATLAS and CMS experiments will record ~ 10 times as +much data from ~ 100 times as many collisions as were used to discover the Higgs boson. + + +Columnar data delivery +---------------------- + +ServiceX seeks to enable on-demand data delivery of columnar data in a variety of formats for +physics analyses. It provides a uniform backend to data storage services, ensuring the user doesn't +have to know how or where the data is stored, and is capable of on-the-fly data transformations +into a variety of formats (ROOT files, Arrow arrays, Parquet files, ...) The service offers +preprocessing functionality via an analysis description language called +`func-adl `_ that allows users to filter events, request columns, +and even compute new variables. This enables the user to start from any format and extract only the +data needed for an analysis. + +.. image:: img/organize2.png + :alt: Organization + +ServiceX is designed to feed columns to a user running an analysis (e.g. via +`Awkward `_ or +`Coffea `_ tools) based on the results of a query designed by +the user. \ No newline at end of file diff --git a/docs/models.rst b/docs/models.rst index f68cc0c3..e01ffb38 100644 --- a/docs/models.rst +++ b/docs/models.rst @@ -1,6 +1,6 @@ Models ====== -.. automodule:: servicex_client.models +.. automodule:: servicex.models :members: :exclude-members: Config, parse_finish_time :undoc-members: diff --git a/docs/modules.rst b/docs/modules.rst new file mode 100644 index 00000000..094062e2 --- /dev/null +++ b/docs/modules.rst @@ -0,0 +1,7 @@ +ServiceX_frontend +================= + +.. toctree:: + :maxdepth: 4 + + servicex diff --git a/docs/servicex.app.rst b/docs/servicex.app.rst new file mode 100644 index 00000000..88c2c33b --- /dev/null +++ b/docs/servicex.app.rst @@ -0,0 +1,53 @@ +servicex.app package +==================== + +Submodules +---------- + +servicex.app.cache module +------------------------- + +.. automodule:: servicex.app.cache + :members: + :undoc-members: + :show-inheritance: + +servicex.app.cli\_options module +-------------------------------- + +.. automodule:: servicex.app.cli_options + :members: + :undoc-members: + :show-inheritance: + +servicex.app.codegen module +--------------------------- + +.. automodule:: servicex.app.codegen + :members: + :undoc-members: + :show-inheritance: + +servicex.app.main module +------------------------ + +.. automodule:: servicex.app.main + :members: + :undoc-members: + :show-inheritance: + +servicex.app.transforms module +------------------------------ + +.. automodule:: servicex.app.transforms + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: servicex.app + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/servicex.databinder.rst b/docs/servicex.databinder.rst new file mode 100644 index 00000000..cf2886ce --- /dev/null +++ b/docs/servicex.databinder.rst @@ -0,0 +1,53 @@ +servicex.databinder package +=========================== + +Submodules +---------- + +servicex.databinder.databinder module +------------------------------------- + +.. automodule:: servicex.databinder.databinder + :members: + :undoc-members: + :show-inheritance: + +servicex.databinder.databinder\_configuration module +---------------------------------------------------- + +.. automodule:: servicex.databinder.databinder_configuration + :members: + :undoc-members: + :show-inheritance: + +servicex.databinder.databinder\_deliver module +---------------------------------------------- + +.. automodule:: servicex.databinder.databinder_deliver + :members: + :undoc-members: + :show-inheritance: + +servicex.databinder.databinder\_outputs module +---------------------------------------------- + +.. automodule:: servicex.databinder.databinder_outputs + :members: + :undoc-members: + :show-inheritance: + +servicex.databinder.databinder\_requests module +----------------------------------------------- + +.. automodule:: servicex.databinder.databinder_requests + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: servicex.databinder + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/servicex.func_adl.rst b/docs/servicex.func_adl.rst new file mode 100644 index 00000000..c35f4d72 --- /dev/null +++ b/docs/servicex.func_adl.rst @@ -0,0 +1,37 @@ +servicex.func\_adl package +========================== + +Submodules +---------- + +servicex.func\_adl.func\_adl\_dataset module +-------------------------------------------- + +.. automodule:: servicex.func_adl.func_adl_dataset + :members: + :undoc-members: + :show-inheritance: + +servicex.func\_adl.func\_adl\_dataset\_group module +--------------------------------------------------- + +.. automodule:: servicex.func_adl.func_adl_dataset_group + :members: + :undoc-members: + :show-inheritance: + +servicex.func\_adl.util module +------------------------------ + +.. automodule:: servicex.func_adl.util + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: servicex.func_adl + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/servicex.rst b/docs/servicex.rst new file mode 100644 index 00000000..255473e0 --- /dev/null +++ b/docs/servicex.rst @@ -0,0 +1,127 @@ +servicex package +================ + +Subpackages +----------- + +.. toctree:: + :maxdepth: 4 + + servicex.app + servicex.databinder + servicex.func_adl + +Submodules +---------- + +servicex.configuration module +----------------------------- + +.. automodule:: servicex.configuration + :members: + :undoc-members: + :show-inheritance: + +servicex.databinder\_models module +---------------------------------- + +.. automodule:: servicex.databinder_models + :members: + :undoc-members: + :show-inheritance: + +servicex.dataset\_group module +------------------------------ + +.. automodule:: servicex.dataset_group + :members: + :undoc-members: + :show-inheritance: + +servicex.dataset\_identifier module +----------------------------------- + +.. automodule:: servicex.dataset_identifier + :members: + :undoc-members: + :show-inheritance: + +servicex.expandable\_progress module +------------------------------------ + +.. automodule:: servicex.expandable_progress + :members: + :undoc-members: + :show-inheritance: + +servicex.minio\_adapter module +------------------------------ + +.. automodule:: servicex.minio_adapter + :members: + :undoc-members: + :show-inheritance: + +servicex.models module +---------------------- + +.. automodule:: servicex.models + :members: + :undoc-members: + :show-inheritance: + +servicex.python\_dataset module +------------------------------- + +.. automodule:: servicex.python_dataset + :members: + :undoc-members: + :show-inheritance: + +servicex.query module +--------------------- + +.. automodule:: servicex.query + :members: + :undoc-members: + :show-inheritance: + +servicex.query\_cache module +---------------------------- + +.. automodule:: servicex.query_cache + :members: + :undoc-members: + :show-inheritance: + +servicex.servicex\_adapter module +--------------------------------- + +.. automodule:: servicex.servicex_adapter + :members: + :undoc-members: + :show-inheritance: + +servicex.servicex\_client module +-------------------------------- + +.. automodule:: servicex.servicex_client + :members: + :undoc-members: + :show-inheritance: + +servicex.types module +--------------------- + +.. automodule:: servicex.types + :members: + :undoc-members: + :show-inheritance: + +Module contents +--------------- + +.. automodule:: servicex + :members: + :undoc-members: + :show-inheritance: diff --git a/docs/servicex_client.rst b/docs/servicex_client.rst index 1bf83002..f8d0d079 100644 --- a/docs/servicex_client.rst +++ b/docs/servicex_client.rst @@ -1,6 +1,6 @@ ServiceXClient ============== -.. autoclass:: servicex_client.servicex_client.ServiceXClient +.. autoclass:: servicex.servicex_client.ServiceXClient :members: :exclude-members: Config :undoc-members: diff --git a/docs/transformer_matrix.rst b/docs/transformer_matrix.rst new file mode 100644 index 00000000..02f593b9 --- /dev/null +++ b/docs/transformer_matrix.rst @@ -0,0 +1,9 @@ +Tranformer Matrix +----------------- + +- query language vs input data type: show supported combinations +- brief introduction to the languages +- Document FuncADL (UW) +- explain differences between flat ntuple and xAOD languages +- Document uproot-raw (UT) +- Document Python transformer (optional feature not supported by all sites) \ No newline at end of file diff --git a/docs/transformer_matrix/index.rst b/docs/transformer_matrix/index.rst new file mode 100644 index 00000000..e69de29b diff --git a/docs/troubleshoot.rst b/docs/troubleshoot.rst new file mode 100644 index 00000000..e07cb4ae --- /dev/null +++ b/docs/troubleshoot.rst @@ -0,0 +1,4 @@ +Troubleshooting +--------------- + +Common errors encountered in Servicex and their remedies \ No newline at end of file diff --git a/examples/bigger_uproot.py b/examples/bigger_uproot.py index 398d3c6c..9db9300a 100644 --- a/examples/bigger_uproot.py +++ b/examples/bigger_uproot.py @@ -30,22 +30,28 @@ from servicex.func_adl.func_adl_dataset import FuncADLQuery from servicex.servicex_client import deliver -query = FuncADLQuery().Select(lambda e: {'el_pt': e['el_pt']}) -spec = ServiceXSpec( - General=General( - ServiceX="testing1", - Codegen="uproot", - OutputFormat="parquet", - Delivery="LocalCache" - ), - Sample=[ - Sample( - Name="bigger_uproot", - RucioDID="user.kchoi:user.kchoi.fcnc_tHq_ML.ttH.v8", - Tree="nominal", - Query=query - ) - ] -) -print(deliver(spec)) +def bigger_uproot(): + query = FuncADLQuery().Select(lambda e: {'el_pt': e['el_pt']}) + + spec = ServiceXSpec( + General=General( + ServiceX="servicex-uc-af", + Codegen="uproot", + OutputFormat="parquet", + Delivery="LocalCache" + ), + Sample=[ + Sample( + Name="bigger_uproot", + RucioDID="user.kchoi:user.kchoi.fcnc_tHq_ML.ttH.v8", + Tree="nominal", + Query=query + ) + ] + ) + print(deliver(spec)) + + +if __name__ == "__main__": + bigger_uproot() diff --git a/pyproject.toml b/pyproject.toml index 4891618e..6c423813 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -82,6 +82,7 @@ test = [ docs = [ "sphinx>=7.0.1", "furo>=2023.5.20", + "sphinx-code-include>=1.4.0" ] develop = [ "servicex[databinder,pandas,test,docs]",