Merge branch 'master' into pr/SOFTWARE-5164.byelcmaps

Author: matyasselmeci

docs/compute-element/covid-19.md

@@ -21,7 +21,7 @@ To support COVID-19 work, the overall process includes the following:
        If neither solution is viable, or you'd like to discuss the options, please send email to
        <[email protected]> and we'll work with you to arrive at the best solution.
     -  If you already provide resources through an OSG Hosted CE, skip to [this section](#requesting-covid-19-jobs).
-1. [Enable the OSG VO](../security/lcmaps-voms-authentication.md#configuring-the-lcmaps-voms-plugin) on your HTCondor-CE.
+1. [Enable the OSG VO](install-htcondor-ce.md#configuring-authentication) on your HTCondor-CE.
 1. Setup a job route specific to COVID-19 pilot jobs (documented below).
    The job route will allow you to prioritize these jobs using local policy in your site's cluster.
 1. (Optional) To attract more user jobs, install [CVMFS](../worker-node/install-cvmfs.md) and

docs/compute-element/htcondor-ce-overview.md

@@ -9,9 +9,8 @@ Before continuing with the overview, make sure that you are familiar with the fo
 -   An OSG site plan
     -   What is a batch system and which one will you use ([HTCondor](http://htcondor.org/), PBS, LSF, SGE, or
         [SLURM](https://slurm.schedmd.com/))?
-    -   Security in the OSG via [GSI](https://gridcf.org/gct-docs/latest/gsic/index.html) (i.e.,
-        [Certificate authorities](https://en.wikipedia.org/wiki/Certificate_authority), user and host
-        [certificates](https://en.wikipedia.org/wiki/Public_key_certificate), proxies)
+    -   Security in the OSG via [host certificates](../security/host-certs/overview.md) to authenticate servers and
+        [bearer tokens](../security/tokens/overview.md) to authenticate clients
 -   Pilot jobs, frontends, and factories (i.e., [GlideinWMS](http://glideinwms.fnal.gov/doc.prd/index.html),
     AutoPyFactory)
 
@@ -36,7 +35,7 @@ What is HTCondor-CE?
 --------------------
 
 HTCondor-CE is a special configuration of the HTCondor software designed to be a job gateway solution for the OSG.
-It is configured to use the [JobRouter daemon](http://research.cs.wisc.edu/htcondor/manual/v8.6/5_4HTCondor_Job.html) to
+It is configured to use the [JobRouter daemon](https://htcondor.readthedocs.io/en/v9_0/grid-computing/job-router.html) to
 delegate jobs by transforming and submitting them to the site’s batch system.
 
 Benefits of running the HTCondor-CE:
@@ -118,8 +117,8 @@ How the CE is Customized
 Aside from the [basic configuration](install-htcondor-ce.md#configuring-htcondor-ce) required in the CE
 installation, there are two main ways to customize your CE (if you decide any customization is required at all):
 
--   **Deciding which VOs are allowed to run at your site:** The recommended method of authorizing VOs at your site is
-    based on the [LCMAPS framework](../security/lcmaps-voms-authentication.md)
+-   **Deciding which collaborations are allowed to run at your site:** collaborations will submit resource allocation
+    requests to your CE using bearer tokens, and you can configure which collaboration's tokens you are willing to accept.
 -   **How to filter and transform the grid jobs to be run on your batch system:** Filtering and transforming grid jobs
     (i.e., setting site-specific attributes or resource limits), requires configuration of your site’s job routes.
     For examples of common job routes, consult the [JobRouter recipes](job-router-recipes.md) page.
@@ -132,17 +131,14 @@ installation, there are two main ways to customize your CE (if you decide any cu
 How Security Works
 ------------------
 
-In the OSG, security depends on a PKI infrastructure involving Certificate Authorities (CAs) where CAs sign and issue
-certificates.
-When these clients and hosts wish to communicate with each other, the identities of each party is confirmed by
-cross-checking their certificates with the signing CA and establishing trust.
-
-In its default configuration, HTCondor-CE uses GSI-based authentication and authorization to verify the certificate
-chain, which will work with [LCMAPS VOMS authentication](../security/lcmaps-voms-authentication.md).
-Additionally, it can be reconfigured to provide alternate authentication mechanisms such as Kerberos, SSL, shared
-secret, or even IP-based authentication.
-More information about authorization methods can be found
-[here](http://research.cs.wisc.edu/htcondor/manual/v8.6/3_8Security.html#SECTION00483000000000000000).
+In the OSG, communication is secured between various parties using a combination of PKI infrastructure involving
+Certificate Authorities (CAs) and bearer tokens.
+Services such as a Compute Entrypoint, present [host certificates](../security/host-certs/overview.md) to prove their
+identity to clients, much like your browser verifies websites that you may visit.
+
+And to use these services, clients present [bearer tokens](../security/tokens/overview.md) declaring their association
+with a given collaboration and what permissions the collaboration has given the client.
+In turn, the service may be configured to authorize the client based on their collaboration.
 
 Next steps
 ----------

docs/compute-element/install-htcondor-ce.md

@@ -7,7 +7,7 @@ The [HTCondor-CE](http://htcondor-ce.org) software is a *job gateway* for an OSG
 As such, the OSG will submit resource allocation requests (RARs) jobs to your HTCondor-CE and it will handle
 authorization and delegation of RARs to your local batch system.
 In OSG today, RARs are sent to CEs as *pilot jobs* from a factory, which in turn are able to accept and run end-user jobs.
-See the [upstream documentation](https://htcondor.github.io/htcondor-ce/architecture/) for a more detailed introduction.
+See the [upstream documentation](https://htcondor.com/htcondor-ce/architecture/) for a more detailed introduction.
 
 Use this page to learn how to install, configure, run, test, and troubleshoot an OSG HTCondor-CE.
 
@@ -17,18 +17,17 @@ Use this page to learn how to install, configure, run, test, and troubleshoot an
 
 !!! note
     If you are installing an HTCondor-CE for use outside of the OSG, consult
-    [the upstream documentation](https://htcondor.github.io/htcondor-ce/v5/installation/htcondor-ce/) instead.
+    [the upstream documentation](https://htcondor.com/htcondor-ce/) instead.
 
 Before Starting
 ---------------
 
 Before starting the installation process, consider the following points, consulting the upstream references as needed
-([HTCondor-CE 5](https://htcondor.github.io/htcondor-ce/v5/reference/),
-[HTCondor-CE 4](https://htcondor.github.io/htcondor-ce/v4/reference/)):
+([HTCondor-CE 5](https://htcondor.com/htcondor-ce/v5/reference/)):
 
 -   **User IDs:** If they do not exist already, the installation will create the Linux users `condor` (UID 4716) and
-    `gratia` (UID 42401)
-    You will also need to create Unix accounts for each VO that you wish to support.
+    `gratia`
+    You will also need to create Unix accounts for each collaboration that you wish to support.
     See details in the ['Configuring authentication' section below](#configuring-authentication).
 -   **SSL certificate:** The HTCondor-CE service uses a host certificate at `/etc/grid-security/hostcert.pem` and an
     accompanying key at `/etc/grid-security/hostkey.pem`
@@ -37,38 +36,17 @@ Before starting the installation process, consider the following points, consult
 -   **Access point/login node:** HTCondor-CE should be installed on a host that already has the ability to submit jobs
     into your local cluster
 -   **File Systems**: Non-HTCondor batch systems require a
-    [shared file system](https://htcondor.github.io/htcondor-ce/v5/configuration/local-batch-system/#sharing-the-spool-directory)
+    [shared file system](https://htcondor.com/htcondor-ce/v5/configuration/local-batch-system/#sharing-the-spool-directory)
     between the HTCondor-CE host and the batch system worker nodes.
 
 As with all OSG software installations, there are some one-time (per host) steps to prepare in advance:
 
 - Ensure the host has [a supported operating system](../release/supported_platforms.md)
 - Install the appropriate [EPEL](../common/yum.md#install-the-epel-repositories) and
   [OSG](../common/yum.md#install-the-osg-repositories) Yum repositories for your operating system.
-  See [the next section](#choosing-the-osg-yum-repository) for guidance on choosing which OSG Yum repository to install.
 - Obtain root access to the host
 - Install [CA certificates](../common/ca.md)
 
-Choosing the OSG Yum Repository
--------------------------------
-
-!!! danger "Before considering OSG 3.6…"
-    Due to potentially disruptive changes in protocols, contact your virtual organization(s) (VO) to verify that they
-    support token-based authentication and/or HTTP-based data transfer before considering an upgrade to OSG 3.6.
-    If your VO(s) don't support these new protocols or you don't know which protocols your VO(s) support,
-    install or remain on the [OSG 3.5 release series](../release/notes.md).
-
-The OSG distributes different versions of HTCondor-CE and HTCondor in separate [YUM repositories](../common/yum.md).
-Most notably, the repository that you choose will determine the types of credentials that your CE is able to accept.
-Use the following table to decide OSG YUM repository to install HTCondor-CE:
-
-| **YUM Repository**                                              | **Bearer Tokens** | **GSI and VOMS** |
-|-----------------------------------------------------------------|-------------------|------------------|
-| OSG 3.5 upcoming **(recommended)**: HTCondor-CE 5, HTCondor 9.0 | ✅           | ✅          |
-| OSG 3.5 release: HTCondor-CE 4, HTCondor 8.8                    |                   | ✅          |
-| OSG 3.6 release: HTCondor-CE 5, HTCondor 9.0                    | ✅           |                  |
-
-
 Installing HTCondor-CE
 ----------------------
 
@@ -115,15 +93,8 @@ To simplify installation, OSG provides convenience RPMs that install all require
 
 1. Install the CE software where `<PACKAGE>` is the package you selected in the above step.:
 
-    -   If you have decided to install from [3.5 upcoming](#choosing-the-osg-yum-repository), run the following command
-
-            :::console
-            root@host # yum install --enablerepo=osg-upcoming <PACKAGE>
-
-    -   Otherwise, run the following command:
-
-            :::console
-            root@host # yum install <PACKAGE>
+        :::console
+        root@host # yum install <PACKAGE>
 
 
 Configuring HTCondor-CE
@@ -134,27 +105,15 @@ For more advanced configuration, see the section on [optional configurations](#o
 
 ### Configuring the local batch system ###
 
-To configure HTCondor-CE to integrate with your local batch system, please refer to the upstream documentation based on
-your installed version of HTCondor-CE:
-
--   [HTCondor-CE 5](https://htcondor.github.io/htcondor-ce/v5/configuration/local-batch-system/)
--   [HTCondor-CE 4](https://htcondor.github.io/htcondor-ce/v4/installation/htcondor-ce/#configuring-the-batch-system)
+To configure HTCondor-CE to integrate with your local batch system,
+please refer to the [upstream documentation](https://htcondor.com/htcondor-ce/v5/configuration/local-batch-system/).
 
 ### Configuring authentication ###
 
-Depending on the OSG repository from which you have installed HTCondor-CE, you can allow pilot job submission to your CE
-based on X.509 proxies (i.e., GSI and VOMS), bearer tokens, or both.
-
-#### GSI and VOMS (OSG 3.5 only) ####
-
-To configure which VOs and users are authorized to submit pilot jobs to your HTCondor-CE, follow the instructions in
-[the LCMAPS VOMS plugin document](../security/lcmaps-voms-authentication.md#configuring-the-lcmaps-voms-plugin).
-
-
-#### Bearer Tokens (OSG 3.5 upcoming, OSG 3.6)####
-
+HTCondor-CE clients will submit RARs accompanied by [bearer tokens](../security/tokens/overview.md) declaring their
+association with a given collaboration and what permissions the collaboration has given the client
 The `osg-scitokens-mapfile`, pulled in by the `osg-ce` package, provides default token to local user mappings.
-To add support for a particular VO:
+To accept RARs from a particular collaboration:
 
 1.  Create the Unix account(s) corresponding to the last field in the default mapfile:
     `/usr/share/condor-ce/mapfiles.d/osg-scitokens-mapfile.conf`.
@@ -171,7 +130,7 @@ To add support for a particular VO:
         SCITOKENS /^https\:\/\/scitokens\.org\/osg\-connect,/ osgpilot
 
 For more details of the mapfile format, consult the "SciTokens" section of the
-[upstream documentation](https://htcondor.github.io/htcondor-ce/v5/configuration/authentication/#scitokens).
+[upstream documentation](https://htcondor.com/htcondor-ce/v5/configuration/authentication/#scitokens).
 
 
 ### Automatic configuration
@@ -204,14 +163,10 @@ the different pieces of software required for an OSG HTCondor-CE:
 
 In addition to the configurations above, you may need to further configure how pilot jobs are filtered and transformed
 before they are submitted to your local batch system or otherwise change the behavior of your CE.
-For detailed instructions, please refer to the upstream documentation based on your installed version of HTCondor-CE:
+For detailed instructions, please refer to the upstream documentation:
 
--   HTCondor-CE 5
-    -   [Configuring the Job Router](https://htcondor.github.io/htcondor-ce/v5/configuration/job-router-overview/)
-    -   [Optional configuration](https://htcondor.github.io/htcondor-ce/v5/configuration/optional-configuration/)
--   HTCondor-CE 4
-    -   [Configuring the Job Router](https://htcondor.github.io/htcondor-ce/v4/batch-system-integration/)
-    -   [Optional configuration](https://htcondor.github.io/htcondor-ce/v4/installation/htcondor-ce/#optional-configuration)
+-   [Configuring the Job Router](https://htcondor.com/htcondor-ce/v5/configuration/job-router-overview/)
+-   [Optional configuration](https://htcondor.com/htcondor-ce/v5/configuration/optional-configuration/)
 
 #### Accounting with multiple CEs or local user jobs
 
@@ -220,61 +175,26 @@ For detailed instructions, please refer to the upstream documentation based on y
 
 If your site has multiple CEs or you have non-grid users submitting to the same local batch system, the OSG accounting
 software needs to be configured so that it doesn't over report the number of jobs.
-
-1.  Determine which file you need to modify
-
-    -   **For OSG 3.5 installations,** use the following table:
-
-        | If your batch system is… | Then edit the following file on each of your CE(s)… |
-        |:-------------------------|:--------------------------------------------|
-        | LSF                      | `/etc/gratia/pbs-lsf/ProbeConfig`           |
-        | PBS                      | `/etc/gratia/pbs-lsf/ProbeConfig`           |
-        | SGE                      | `/etc/gratia/sge/ProbeConfig`               |
-        | SLURM                    | `/etc/gratia/slurm/ProbeConfig`             |
-
-    -   **For OSG 3.6 installations,** you'll need to modify `/etc/gratia/htcondor-ce/ProbeConfig`
-
-1.  Edit the value of `SuppressNoDNRecords` on each of your CE's so that it reads:
+Modify the value of `SuppressNoDNRecords` in `/etc/gratia/htcondor-ce/ProbeConfig` on each of your CE's so that it
+reads:
 
         :::file
         SuppressNoDNRecords="1"
 
 Starting and Validating HTCondor-CE
 -----------------------------------
 
-For information on how to start and validate the core HTCondor-CE services, please refer to the upstream documentation
-based on your installed version of HTCondor-CE:
-
--   [HTCondor-CE 5](https://htcondor.github.io/htcondor-ce/v5/verification/)
--   [HTCondor-CE 4](https://htcondor.github.io/htcondor-ce/v4/verification/)
-
-### Enabling OSG accounting (OSG 3.5 only)
-
-In addition to the core HTCondor-CE services, an OSG 3.5 HTCondor-CE must also start and enable the accounting service,
-`gratia-probes-cron`:
-
-```console
-root@host # systemctl start gratia-probes-cron
-root@host # systemctl enable gratia-probes-cron
-```
-
-In OSG 3.6, OSG accounting is managed directly by HTCondor-CE
-(see the [update instructions](../release/updating-to-osg-36.md#gratia-probe) for more details).
+For information on how to start and validate the core HTCondor-CE services, please refer to the
+[upstream documentation](https://htcondor.com/htcondor-ce/v5/verification/)
 
 Troubleshooting HTCondor-CE
 ---------------------------
 
-For information on how to troubleshoot your HTCondor-CE, please refer to the upstream documentation based on your
-installed version of HTCondor-CE:
-
--   HTCondor-CE 5:
-    -   [Common issues](https://htcondor.github.io/htcondor-ce/v5/troubleshooting/common-issues/)
-    -   [Debugging tools](https://htcondor.github.io/htcondor-ce/v5/troubleshooting/debugging-tools/)
-    -   [Helpful logs](https://htcondor.github.io/htcondor-ce/v5/troubleshooting/logs/)
--   HTCondor-CE 4
-    -   [Common issues](https://htcondor.github.io/htcondor-ce/v4/troubleshooting/troubleshooting/#htcondor-ce-troubleshooting-items)
-    -   [Debugging tools](https://htcondor.github.io/htcondor-ce/v4/troubleshooting/troubleshooting/#htcondor-ce-troubleshooting-tools)
-    -   [Helpful logs](https://htcondor.github.io/htcondor-ce/v4/troubleshooting/troubleshooting/#htcondor-ce-troubleshooting-data)
+For information on how to troubleshoot your HTCondor-CE, please refer to the upstream documentation:
+
+-   [Common issues](https://htcondor.com/htcondor-ce/v5/troubleshooting/common-issues/)
+-   [Debugging tools](https://htcondor.com/htcondor-ce/v5/troubleshooting/debugging-tools/)
+-   [Helpful logs](https://htcondor.com/htcondor-ce/v5/troubleshooting/logs/)
 
 Registering the CE
 ------------------