Macros are not calling autocomplete functions but aliases do

[Sripadvallabh]

Is there a possible way for macros to call autocomplete functions in case their placeholder arguments are given to the macro??

Basically i have a situation as follows:

macro create x1 words {1} {2} full

alias create xd x1 d

When i invoke the command 'xd new_word <TAB>' is there a way to expect the autocompletion list as in "words d new_word <TAB>" ?????

But if i just have an alias it autocompletes the next word in sequence.

[tleonhardt]

@kmvanbrunt This question is up your alley ...

[kmvanbrunt]

Because macros do not resolve until after hitting Enter, tab completion will only complete paths while typing a macro.

When I wrote this feature 6 years ago, I remember having a lot of trouble solving how to implement tab completing macros in the way you are asking. It gets complicated since arguments can be used in any order in a resolved macro. A later argument could determine what to tab complete in an earlier argument.

Here is an extreme example:
macro create my_macro help {2} {1}

Running my_macro create alias resolves to help alias create

Therefore I couldn't tab complete the first argument without first knowing what the second argument was.

Additionally, an argument number can be repeated in a macro:
macro create ft file_taxes -p {1} -q {2} -r {1}

Lastly macros can resolve into other macros and aliases. So there is a lot of state to keep track of.

[kmvanbrunt]

As you already know, aliases autocomplete as if you typed their actual value. This behavior cannot be changed.

Changing macros to autocomplete something other than paths requires you to override cmd2.Cmd.perform_completion() just to change one line of code.

# Check if a macro was entered
if command in self.macros:
    completer_func = self.path_complete

Note: We are removing macros from cmd2 3.0, so I don't recommend getting used to them.

[Sripadvallabh]

Thanks for your reply.

Maybe i will change that line of code to reflect my app currently.

May i know why are you removing macros? Isn't it not useful & powerful??

[kmvanbrunt]

We are removing them because cmd2 provides better options like aliases and argparse-based commands which offer far more customization for tab completing arguments. Macros offer no advantage over these options.

Are you using argparse-based commands? They will make it simple for you to tab complete the dictionary values.

Otherwise if you are using regular commands, you can implement Cmd.completedefault() to tab complete your dictionary values for commands that have no completer.

https://docs.python.org/3/library/cmd.html#cmd.Cmd.completedefault

Something like this:

def completedefault(self, text, line, begidx, endidx):
    return self.basic_complete(text, line, begidx, endidx, <dictionary values>)

[Sripadvallabh]

Yes i understand your point.

In our case one of the major motivations to use our application is to minimize much of the lengthy underlying commands users need to type all the time. Our app shell in turn calls another application that takes commands whose lengths are > 30 characters.

There are only a few variations between the commands; as an example an argument of (0, 1, 2 or 3) at the start of the command will print different outputs. Hence we try to utilize macros to supply that argument at the start but is not available via aliases. Argparse based options are not applicable since our commands do not use such options because only the underlying app does(a C-based shell).

As an example,

(cmd2-shell) command1 <sub_command1 <XYZ> <NUM_ARG> <XYZ> <OTHER_ARG>>

where sub_command1 is a command in the underlying C shell.

So for this type of command, we create a macro saying macro create mr command1 sub_command1 <XYZ> {1} <XYZ> along with an alias saying alias create mr0 0 for 0 num arg & alias create mr1 1 for 1 num arg & so on upto 4. Btw, this is the predominant use-case via our shell for more than 100 users at least.

All in all, i expect you guys to atleast retain backward compatibility with macros for my sake. Or you can provide me a way ; maybe pip install an older version & keep it running. We can have further conversations via meet also if you would want to see the use-case in live.

Thanks for going through this long message

[kmvanbrunt]

You're the first person I've heard from that uses macros. What an interesting use case.

I originally wrote macros for a particular user and they never ended up using them. I've never used them either, so it's encouraging to know the feature didn't go to waste.

Give me some time to consider your issue. There may be a way to solve it without macros and get tab completion.

If we keep macros, they would have to be improved. The lack of proper tab completion has always bothered me.

[tleonhardt]

@Sripadvallabh Absolutely, cmd2 empowers you to easily have custom auto-completion. Please see our basic_completion.py example which should show you how to accomplish what you want.

[Sripadvallabh]

Yes Todd. However my use-case is slightly complicated as mentioned in the above reply thread.

[tleonhardt]

Macros are a feature we wish we hadn't created because there are so many nasty edge cases. We plan to remove them in the 3.0 release.

[Sripadvallabh]

I understand your concerns.

But can we atleast have a plugin for this to ensure this stays as extended and can ensure backward compatibility?

Trust me, I really need this one, a lot of users benefit from this. I can share more details of the use cases and benefits in mail as well.

[tleonhardt]

@Sripadvallabh We can investigate if macros are something that can be supported via a plugin. That is a potentially good idea.

I'd appreciate it if you could share details of your use case. That could help us understand the benefits. My email is <first_name>.<last_name>@gmail.com and my full name is in my GitHub profile.

[Sripadvallabh]

Sure thanks todd. Will send a mail across consisting of the detailed use case and implementation. Just give me a day or two.

I can help contribute towards the plugin as well if you guys are busy as it is something that I would benefit from a lot.

[tleonhardt]

We'd very much appreciate your help contributing towards a plugin for this and we'd be happy to collaborate with you on that. I look forward to your email - no rush.

[tleonhardt]

@Sripadvallabh
We are now leaning towards leaving Macros in. However they are a pain to work with and maintain. We probably wont' be improving them any.

If you want to submit a PR to improve them somehow to fix any perceived shortcomings, we'd be happy to entertain that.

[Sripadvallabh]

Thanks a lot for this. I understand there are some nasty side effects from using macros, hence i appreciate your courage leaving them in 3.0 which you guys have tried best to improve.

Let me check the macro implementation and try to see if we can improve the code. Although I am still a beginner, still I will try my best alongside chatgpt :) to check them out.

Really appreciate your commitment to improving cmd2. Thanks a lot! It's a wonderful module to work with

[pdalloz]

Not sure if it's too late to chime in on macros. I find them very useful despite lack of autocomplete.
In pobshell I provide bash-like commands to explore live Python objects. https://github.com/pdalloz/pobshell

Some command can call sub-commands and it can get verbose, e.g. the find command can run a command string given by --cmd CMD on each matched object. I will work on improving syntax of the find command, but right now I'd be very sad to lose macros, they cut typing effort significantly

(I was planning to include macros in upcoming demo videos. Now I'm not sure what to do.)

Here is a list of my favourite macros. And I'm discovering new tricks all the time

gn name

• find name somewhere in current object and give long listing of it

/ ▶ macro create gn find . --name {1}  -l  
/ ▶ gn get_names 
/cmd2/Cmd/get_names  function                  <function Cmd.get_names at 0x13b4104a0>

grep code pattern

• find objects with code that matches pattern; list object details then relevant lines

/cmd2 ▶ macro create grep find --cat '*{1}*'  --cmd "ls -vl .;cat . | grep '{1}'"
/cmd2 ▶ grep get_names -L 3 .
/cmd2/Cmd  type                      <class 'cmd2.cmd2.Cmd'>
    def get_names(self) -> List[str]:
            for name in self.get_names()
            for name in self.get_names()
/cmd2/cmd2  module                    <module 'cmd2.cmd2' from '/opt/anaconda3/envs/PobEnv312all/lib/python3.12/site-pa…
    def get_names(self) -> List[str]:
            for name in self.get_names()
            for name in self.get_names()
/cmd2/Cmd/completenames  function                  <function Cmd.completenames at 0x13a5d8040>
        return [a[3:] for a in self.get_names() if a.startswith(dotext)]

for name-pattern pob-command

• run pob-command on each object with matching name

/ ▶ macro create for find . --name {1} --cmd '{2}'
Macro 'for' overwritten
/ ▶ for get_names "cat -v1 ." 
/cmd2/Cmd/get_names      def get_names(self) -> List[str]:
/cmd2/cmd2/Cmd/get_names      def get_names(self) -> List[str]:

refs name

• find name in source code, report objects referening it and show matching lines
  • differs from grep because it looks for name as a token; delimits search pattern with \b

  /PVi ▶ macro create refs find -r --cat '\b{1}\b' --cmd "ls -l .; cat -q . | grep -E '\b{1}\b'"
  Macro 'refs' overwritten
  /PVi ▶ refs TestCollector
  TestCollector  type                      <class 'pobshell.pobmain.Pobiverse.TestCollector'>
              self._pob = pob  # keep a reference to Pobiverse instance, so TestCollector can run scripts etc
  do_test  method                    <bound method Pobiverse.do_test of <pobshell.pobmain.Pobiverse object at 0x13e6cb320…
              self.test_collector = self.TestCollector(self, self.root, self.history)
              self.test_collector = self.TestCollector(self, self.root, self.history)

where name

• find objects with name inside current object and enumerate the objects so they're easy to visit 'cd $N'
  NB stops searching after three matches
  NB Add additional options as required: -a, -d 6, --CODE, --map everything

/ ▶ macro create where find --name {1} -le -L 3
/ ▶ where cmd2
0: /pobmain/cmd2  module                    <module 'cmd2' from '/opt/anaconda3…
1: /pobshell/infocmds/cmd2  module                    <module 'cmd2' from '/opt…
2: /pobshell/pobmain/cmd2  module                    <module 'cmd2' from '/opt/…
/ ▶ cd $0
/pobmain/cmd2 ▶ 

+ pob-command (Macro called '+')

• Run multiple ';'-separated pob-commands on each member of current object

  / ▶ macro create + find --cmd "{1}" -d 1
  Macro '+' created

  / ▶ + "ls -l . ; doc -1 .; cat -1 .; newline"
  /  PobNS                     /
  /  # Exploring use of Pobiverse with JSON
  
  feed  dict                      {'Schedule': {'conferences': [{'serial': 115}], 'events': [{'categories': ['Emerging La…
  
  fp  TextIOWrapper             <_io.TextIOWrapper name='/Users/peterdalloz/Dropbox/PYTHONDROPBOX/pobshell/Use_Cases/osco…
  fp  Character and line based layer over a BufferedIOBase object, buffer.
  
  json  module                    <module 'json' from '/opt/anaconda3/envs/PobEnv312all/lib/python3.12/json/__init__.py'>
  json  JSON (JavaScript Object Notation) <https://json.org> is a subset of
  json  r"""JSON (JavaScript Object Notation) <https://json.org> is a subset of
  
  sys  module                    <module 'sys' (built-in)>
  sys  This module provides access to some objects used or maintained by the

[tleonhardt]

We listened to user feedback and have restored macros in PR #1464 .

At the same time we added Cmd.macro_arg_complete() which tab completes arguments to a macro. Its default behavior is to perform path completion, but it can be overridden as needed.

[pdalloz]

Thank you very much!

[Sripadvallabh]

Thank you so much for bringing this feature back, Todd — I really appreciate the change. It will be a big help to both me and the users of my tool.

I understand the reasoning behind separating macro resolution (the outer loop) from alias or command resolution. It definitely helps keep concerns isolated and makes the logic easier to test and reason about, rather than embedding it deep in the parser. That said, my users have found the macro functionality very valuable, so I’m glad to see it return. I’ll continue brainstorming with colleagues to see if there’s a more elegant or sustainable way to do this long-term.

@kmvanbrunt, it’s often the case that we build something forward-looking, but its usefulness may not be fully recognized until later. I think macros are one of those ideas — a powerful feature for future cmd2 applications. Thank you for thinking ahead and being open to community feedback. It’s great to see thoughtful design like this coming full circle.