NodeSet: add special notation @@source to expand group names

This change allows access to all group names of a specific source in
node set expressions. The special operator `@@` optionally followed
by a source expands group names that can then be manipulated as a set.
The default group source is used when `@@` is used alone.

Note: In that case, group names are not prefixed with @ and as such, not
further expanded into nodes by NodeSet.

Author: thiell

doc/man/man1/cluset.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH CLUSET 1 "2019-12-01" "1.8.3" "ClusterShell User Manual"
+.TH CLUSET 1 "2022-03-19" "1.8.4" "ClusterShell User Manual"
 .SH NAME
 cluset \- compute advanced cluster node set operations
 .
@@ -225,7 +225,7 @@ node[0\-4,11\-13]
 .sp
 This computes a folded nodeset containing nodes found in group @gpu and @slurm:bigmem, but not in both, minus the nodes found in odd chassis groups from 1 to 9.
 .TP
-.B "All nodes" extension (v1.7+)
+.B "All nodes" extension
 The \fB@*\fP and \fB@SOURCE:*\fP special notations may be used in extended patterns to represent all nodes (in SOURCE) according to the \fIall\fP external shell command (see \fBgroups.conf\fP(5)) and are equivalent to:
 .INDENT 7.0
 .INDENT 3.5
@@ -235,6 +235,21 @@ The \fB@*\fP and \fB@SOURCE:*\fP special notations may be used in extended patte
 .UNINDENT
 .UNINDENT
 .UNINDENT
+.TP
+.B Group names in expressions
+The \fB@@SOURCE\fP notation may be used to access all group names from the specified SOURCE (or from the default group source when just \fB@@\fP is used) in node set expressions; this works with either file\-based group sources or with external group sources that have the \fIlist\fP upcall defined (see \fBgroups.conf\fP(5)):
+.INDENT 7.0
+.INDENT 3.5
+.INDENT 0.0
+.TP
+.B $ cluset \-f @@rack
+.UNINDENT
+.nf
+J[1\-3]
+.fi
+.sp
+.UNINDENT
+.UNINDENT
 .UNINDENT
 .SH NODE WILDCARDS
 .sp

doc/man/man1/nodeset.1

@@ -1,6 +1,6 @@
 .\" Man page generated from reStructuredText.
 .
-.TH NODESET 1 "2021-11-03" "1.8.4" "ClusterShell User Manual"
+.TH NODESET 1 "2022-03-19" "1.8.4" "ClusterShell User Manual"
 .SH NAME
 nodeset \- compute advanced nodeset operations
 .
@@ -225,7 +225,7 @@ node[0\-4,11\-13]
 .sp
 This computes a folded nodeset containing nodes found in group @gpu and @slurm:bigmem, but not in both, minus the nodes found in odd chassis groups from 1 to 9.
 .TP
-.B "All nodes" extension (v1.7+)
+.B "All nodes" extension
 The \fB@*\fP and \fB@SOURCE:*\fP special notations may be used in extended patterns to represent all nodes (in SOURCE) according to the \fIall\fP external shell command (see \fBgroups.conf\fP(5)) and are equivalent to:
 .INDENT 7.0
 .INDENT 3.5
@@ -235,6 +235,21 @@ The \fB@*\fP and \fB@SOURCE:*\fP special notations may be used in extended patte
 .UNINDENT
 .UNINDENT
 .UNINDENT
+.TP
+.B Group names in expressions
+The \fB@@SOURCE\fP notation may be used to access all group names from the specified SOURCE (or from the default group source when just \fB@@\fP is used) in node set expressions; this works with either file\-based group sources or with external group sources that have the \fIlist\fP upcall defined (see \fBgroups.conf\fP(5)):
+.INDENT 7.0
+.INDENT 3.5
+.INDENT 0.0
+.TP
+.B $ nodeset \-f @@rack
+.UNINDENT
+.nf
+J[1\-3]
+.fi
+.sp
+.UNINDENT
+.UNINDENT
 .UNINDENT
 .SH NODE WILDCARDS
 .sp

doc/sphinx/tools/nodeset.rst

@@ -634,9 +634,13 @@ configured group sources and also display the default group source (unless
 Listing group names
 """""""""""""""""""
 
-If the **list** external shell command is configured (see
-:ref:`node groups configuration <groups-config>`), it is possible to list
-available groups *from the default source* with the following commands::
+It is always possible to list the groups from a group source if the source is
+:ref:`file-based <group-file-based>`.
+If the source is an :ref:`external group source <group-external-sources>`, the
+**list** upcall must be configured (see also:
+:ref:`node groups configuration <groups-config>`).
+
+To list available groups *from the default source*, use the following command::
 
     $ nodeset -l
     @mgnt
@@ -645,7 +649,7 @@ available groups *from the default source* with the following commands::
     @login
     @compute
 
-Or, to list groups *from a specific group source*, use *-l* in conjunction
+To list groups *from a specific group source*, use *-l* in conjunction
 with *-s* (or *--groupsource*)::
 
     $ nodeset -l -s slurm
@@ -667,6 +671,47 @@ Or, to list groups *from all available group sources*, use *-L* (or
 You can also use ``nodeset -ll`` or ``nodeset -LL`` to see each group's
 associated node sets.
 
+Listing group names in expressions
+""""""""""""""""""""""""""""""""""
+
+ClusterShell 1.8.5 introduces a new operator **@@** optionally followed by
+a source name (e.g. **@@source**) to access the list of *raw group names* of
+the source (without the **@** prefix). If no source is specified (as in *just*
+**@@**), the default group source is used (see :ref:`groups_config_conf`).
+The **@@** operator may be used in any node set expression to manipulate group
+names as a node set.
+
+Example with the default group source::
+
+    $ nodeset -l
+    @mgnt
+    @mds
+    @oss
+    @login
+    @compute
+    
+    $ nodeset -e @@
+    compute login mds mgnt oss
+
+Example with a group source "rack" that defines group names from rack
+locations in a data center::
+
+    $ nodeset -l -s rack
+    @rack:J1
+    @rack:J2
+    @rack:J3
+    
+    $ nodeset -f @@rack
+    J[1-3]
+
+A set of valid, indexed group sources is also accepted by the **@@** operator
+(e.g. **@@dc[1-3]**).
+
+
+.. warning:: An error is generated when using **@@** in an expression if the
+             source is not valid (e.g. invalid name, not configured or upcalls
+             not currently working).
+
 
 Using node groups in basic commands
 """""""""""""""""""""""""""""""""""

doc/txt/cluset.txt

@@ -7,7 +7,7 @@ compute advanced cluster node set operations
 --------------------------------------------
 
 :Author: Stephane Thiell <[email protected]>
-:Date:   2021-11-03
+:Date:   2022-03-19
 :Copyright: GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
 :Version: 1.8.4
 :Manual section: 1
@@ -131,11 +131,19 @@ Example of advanced usage
 
   This computes a folded nodeset containing nodes found in group @gpu and @slurm:bigmem, but not in both, minus the nodes found in odd chassis groups from 1 to 9.
 
-"All nodes" extension (v1.7+)
+"All nodes" extension
   The ``@*`` and ``@SOURCE:*`` special notations may be used in extended patterns to represent all nodes (in SOURCE) according to the *all* external shell command (see ``groups.conf``\(5)) and are equivalent to:
 
     :$ cluset [-s SOURCE] -a -f:
 
+Group names in expressions
+  The ``@@SOURCE`` notation may be used to access all group names from the specified SOURCE (or from the default group source when just ``@@`` is used) in node set expressions; this works with either file-based group sources or with external group sources that have the *list* upcall defined (see ``groups.conf``\(5)):
+
+
+    :$ cluset -f @@rack:
+    
+    | J[1-3]
+
 NODE WILDCARDS
 ==============
 

doc/txt/nodeset.txt

@@ -7,7 +7,7 @@ compute advanced nodeset operations
 -----------------------------------
 
 :Author: Stephane Thiell <[email protected]>
-:Date:   2021-11-03
+:Date:   2022-03-19
 :Copyright: GNU Lesser General Public License version 2.1 or later (LGPLv2.1+)
 :Version: 1.8.4
 :Manual section: 1
@@ -131,11 +131,19 @@ Example of advanced usage
 
   This computes a folded nodeset containing nodes found in group @gpu and @slurm:bigmem, but not in both, minus the nodes found in odd chassis groups from 1 to 9.
 
-"All nodes" extension (v1.7+)
+"All nodes" extension
   The ``@*`` and ``@SOURCE:*`` special notations may be used in extended patterns to represent all nodes (in SOURCE) according to the *all* external shell command (see ``groups.conf``\(5)) and are equivalent to:
 
     :$ nodeset [-s SOURCE] -a -f:
 
+Group names in expressions
+  The ``@@SOURCE`` notation may be used to access all group names from the specified SOURCE (or from the default group source when just ``@@`` is used) in node set expressions; this works with either file-based group sources or with external group sources that have the *list* upcall defined (see ``groups.conf``\(5)):
+
+    :$ nodeset -f @@rack:
+    
+    | J[1-3]
+
+
 NODE WILDCARDS
 ==============
 

lib/ClusterShell/NodeSet.py

@@ -916,6 +916,8 @@ def parse_group_string(self, nodegroup, namespace=None):
             namespace, group = grpstr.split(':', 1)
         if group == '*': # @* or @source:* magic
             reslist = self.all_nodes(namespace)
+        elif group.startswith('@'): # @@source group name list
+            reslist = self.grouplist(grpstr[1:])
         else:
             reslist = self.group_resolver.group_nodes(group, namespace)
         return ','.join(reslist), namespace

tests/NodeSetGroupTest.py

@@ -625,6 +625,11 @@ def testConfigListAllNDWithoutAll(self):
         self.assertEqual(str(NodeSet.fromall(resolver=res)), "rack[1-10]z[1-42]")
         self.assertEqual(res.grouplist(), ['x1y1', 'x1y2', 'x1y[3-4]']) # raw
         self.assertEqual(grouplist(resolver=res), ['x1y1', 'x1y2', 'x1y3', 'x1y4']) # cleaned
+        # test "@@" group name set
+        nodeset = NodeSet("@@", resolver=res)
+        self.assertEqual(str(nodeset), "x1y[1-4]")
+        nodeset = NodeSet("@@local", resolver=res)
+        self.assertEqual(str(nodeset), "x1y[1-4]")
         # test "@*" magic group listing
         nodeset = NodeSet("@*", resolver=res)
         self.assertEqual(str(nodeset), "rack[1-10]z[1-42]")
@@ -1104,6 +1109,11 @@ def test_nodeset_wildcard_grouplist(self):
         self.assertEqual(grouplist(resolver=res), ['a', 'b*', 'c', 'd', 'e'])
         nodeset = NodeSet("*foo*", resolver=res)
         self.assertEqual(str(nodeset), "foo[1-3],foobar,foobar1")
+        # currently @@ triggers group name wildcard expansion
+        nodeset = NodeSet("@@", resolver=res)
+        self.assertEqual(str(nodeset), "a,bar,bar[1-2],c,d,e")
+        nodeset = NodeSet("@@local", resolver=res)
+        self.assertEqual(str(nodeset), "a,bar,bar[1-2],c,d,e")
 
     def test_nodeset_wildcard_support_ranges(self):
         """test NodeSet wildcard support with ranges"""
@@ -1228,6 +1238,16 @@ def testGroupListDefault(self):
         self.assertEqual(len(all_nodes), len(nodes))
         self.assertEqual(all_nodes, nodes)
 
+        # @@ special operator
+        gl_nodes = NodeSet("@@")
+        self.assertEqual(len(gl_nodes), 20)
+        self.assertEqual(str(gl_nodes),
+            "Uppercase,all,chassis[1-12],compute,gpu,gpuchassis,io,mds,oss")
+        gl_nodes = NodeSet("@@default")
+        self.assertEqual(len(gl_nodes), 20)
+        self.assertEqual(str(gl_nodes),
+            "Uppercase,all,chassis[1-12],compute,gpu,gpuchassis,io,mds,oss")
+
     def testGroupListSource2(self):
         """test NodeSet group listing GroupResolver.grouplist(source)"""
         groups = std_group_resolver().grouplist("source2")
@@ -1237,6 +1257,11 @@ def testGroupListSource2(self):
             total += len(NodeSet("@source2:%s" % group))
         self.assertEqual(total, 24)
 
+        # @@ special operator
+        gl_nodes = NodeSet("@@source2")
+        self.assertEqual(len(gl_nodes), 2)
+        self.assertEqual(str(gl_nodes), "gpu,para")
+
     def testGroupNoPrefix(self):
         """test NodeSet group noprefix option"""
         nodeset = NodeSet("montana[32-37,42-55]")
@@ -1641,6 +1666,12 @@ def test_yaml_basic(self):
             nodeset = NodeSet("@unknown", resolver=res)
             self.assertEqual(len(nodeset), 0)
 
+            # @@ grouplist operator
+            nodeset = NodeSet("@@", resolver=res)
+            self.assertEqual(str(nodeset), "bar,foo")
+            nodeset = NodeSet("@@yaml", resolver=res)
+            self.assertEqual(str(nodeset), "bar,foo")
+
             # Regroup
             nodeset = NodeSet("example[1-4,90-100]", resolver=res)
             self.assertEqual(str(nodeset), "example[1-4,90-100]")