feat: add some capabilities to the config parser and a stream parser (#630)

phlptp · pre-commit-ci[bot] · web-flow · commit 8b785a6c7d5b · 2021-08-22T23:53:06.000-04:00
* add some capabilities to the config parser and a stream parser

* style: pre-commit.ci fixes

* add additional tests for the config parser

* additional tests of config sections and indexing

* style: pre-commit.ci fixes

* add initialization for member variables

* warning and error fixes

* add test for `parse_from_stream`

Co-authored-by: pre-commit-ci[bot] &lt;66853113+pre-commit-ci[bot]@users.noreply.github.com&gt;
diff --git a/README.md b/README.md
@@ -766,7 +766,7 @@ sub.subcommand = true
 Spaces before and after the name and argument are ignored. Multiple arguments are separated by spaces. One set of quotes will be removed, preserving spaces (the same way the command line works). Boolean options can be `true`, `on`, `1`, `yes`, `enable`; or `false`, `off`, `0`, `no`, `disable` (case insensitive). Sections (and `.` separated names) are treated as subcommands (note: this does not necessarily mean that subcommand was passed, it just sets the "defaults"). You cannot set positional-only arguments.  Subcommands can be triggered from configuration files if the `configurable` flag was set on the subcommand.  Then the use of `[subcommand]` notation will trigger a subcommand and cause it to act as if it were on the command line.
 
 To print a configuration file from the passed
-arguments, use `.config_to_str(default_also=false, write_description=false)`, where `default_also` will also show any defaulted arguments, and `write_description` will include the app and option descriptions.  See [Config files](https://cliutils.github.io/CLI11/book/chapters/config.html) for some additional details.
+arguments, use `.config_to_str(default_also=false, write_description=false)`, where `default_also` will also show any defaulted arguments, and `write_description` will include the app and option descriptions.  See [Config files](https://cliutils.github.io/CLI11/book/chapters/config.html) for some additional details and customization points.
 
 If it is desired that multiple configuration be allowed.  Use
 
diff --git a/book/chapters/config.md b/book/chapters/config.md
@@ -39,6 +39,12 @@ app.allow_config_extras(CLI::config_extras_mode::error);
 
 is equivalent to `app.allow_config_extras(false);`
 
+```cpp
+app.allow_config_extras(CLI::config_extras_mode::ignore_all);
+```
+
+will completely ignore any mismatches, extras, or other issues with the config file
+
 ### Getting the used configuration file name
 
 If it is needed to get the configuration file name used this can be obtained via
@@ -118,7 +124,7 @@ if a prefix is needed to print before the options, for example to print a config
 
 ### Customization of configure file output
 
-The default config parser/generator has some customization points that allow variations on the TOML format.  The default formatter has a base configuration that matches the TOML format.  It defines 5 characters that define how different aspects of the configuration are handled
+The default config parser/generator has some customization points that allow variations on the TOML format.  The default formatter has a base configuration that matches the TOML format.  It defines 5 characters that define how different aspects of the configuration are handled.  You must use `get_config_formatter_base()` to have access to these fields
 
 ```cpp
 /// the character used for comments
@@ -131,6 +137,18 @@ char arrayEnd = ']';
 char arraySeparator = ',';
 /// the character used separate the name from the value
 char valueDelimiter = '=';
+/// the character to use around strings
+char stringQuote = '"';
+/// the character to use around single characters
+char characterQuote = '\'';
+/// the maximum number of layers to allow
+uint8_t maximumLayers{255};
+/// the separator used to separator parent layers
+char parentSeparatorChar{'.'};
+/// Specify the configuration index to use for arrayed sections
+uint16_t configIndex{0};
+/// Specify the configuration section that should be used
+std::string configSection;
 ```
 
 These can be modified via setter functions
@@ -139,6 +157,11 @@ These can be modified via setter functions
 * `ConfigBase *arrayBounds(char aStart, char aEnd)`: Specify the start and end characters for an array
 * `ConfigBase *arrayDelimiter(char aSep)`: Specify the delimiter character for an array
 * `ConfigBase *valueSeparator(char vSep)`: Specify the delimiter between a name and value
+* `ConfigBase *quoteCharacter(char qString, char qChar)` :specify the characters to use around strings and single characters
+* `ConfigBase *maxLayers(uint8_t layers)` : specify the maximum number of parent layers to process. This is useful to limit processing for larger config files
+* `ConfigBase *parentSeparator(char sep)` : specify the character to separate parent layers from options
+* `ConfigBase *section(const std::string &sectionName)` : specify the section name to use to get the option values, only this section will be processed
+* `ConfigBase *index(uint16_t sectionIndex)` : specify an index section to use for processing if multiple TOML sections of the same name are present `[[section]]`
 
 For example, to specify reading a configure file that used `:` to separate name and values:
 
@@ -174,15 +197,37 @@ app.config_formatter(std::make_shared<NewConfig>());
 
 See [`examples/json.cpp`](https://github.com/CLIUtils/CLI11/blob/master/examples/json.cpp) for a complete JSON config example.
 
+### Trivial JSON configuration example
+
+```JSON
+{
+   "test": 56,
+   "testb": "test",
+   "flag": true
+}
+```
+
+The parser can handle these structures with only a minor tweak
+
+```cpp
+app.get_config_formatter_base()->valueSeparator(':');
+```
+
+The open and close brackets must be on a separate line and the comma gets interpreted as an array separator but since no values are after the comma they get ignored as well.  This will not support multiple layers or sections or any other moderately complex JSON, but can work if the input file is simple.
+
 ## Triggering Subcommands
 
 Configuration files can be used to trigger subcommands if a subcommand is set to configure.  By default configuration file just set the default values of a subcommand.  But if the `configure()` option is set on a subcommand then the if the subcommand is utilized via a `[subname]` block in the configuration file it will act as if it were called from the command line.  Subsubcommands can be triggered via `[subname.subsubname]`.  Using the `[[subname]]` will be as if the subcommand were triggered multiple times from the command line.  This functionality can allow the configuration file to act as a scripting file.
 
 For custom configuration files this behavior can be triggered by specifying the parent subcommands in the structure and `++` as the name to open a new subcommand scope and `--` to close it.  These names trigger the different callbacks of configurable subcommands.
 
+## Stream parsing
+
+In addition to the regular parse functions a `parse_from_stream(std::istream &input)` is available to directly parse a stream operator.  For example to process some arguments in an already open file stream.  The stream is fed directly in the config parser so bypasses the normal command line parsing.
+
 ## Implementation Notes
 
-The config file input works with any form of the option given:  Long, short, positional, or the environment variable name.  When generating a config file it will create a name in following priority.
+The config file input works with any form of the option given:  Long, short, positional, or the environment variable name.  When generating a config file it will create an option name in following priority.
 
 1. First long name
 2. Positional name
diff --git a/include/CLI/App.hpp b/include/CLI/App.hpp
@@ -55,7 +55,7 @@ std::string help(const App *app, const Error &e);
 
 /// enumeration of modes of how to deal with extras in config files
 
-enum class config_extras_mode : char { error = 0, ignore, capture };
+enum class config_extras_mode : char { error = 0, ignore, ignore_all, capture };
 
 class App;
 
@@ -1290,6 +1290,16 @@ class App {
         run_callback();
     }
 
+    void parse_from_stream(std::istream &input) {
+        if(parsed_ == 0) {
+            _validate();
+            _configure();
+            // set the parent as nullptr as this object should be the top now
+        }
+
+        _parse_stream(input);
+        run_callback();
+    }
     /// Provide a function to print a help message. The function gets access to the App pointer and error.
     void failure_message(std::function<std::string(const App *, const Error &e)> function) {
         failure_message_ = function;
@@ -2349,6 +2359,18 @@ class App {
         _process_extras();
     }
 
+    /// Internal function to parse a stream
+    void _parse_stream(std::istream &input) {
+        auto values = config_formatter_->from_config(input);
+        _parse_config(values);
+        increment_parsed();
+        _trigger_pre_parse(values.size());
+        _process();
+
+        // Throw error if any items are left over (depending on settings)
+        _process_extras();
+    }
+
     /// Parse one config param, return false if not found in any subcommand, remove if it is
     ///
     /// If this has more than one dot.separated.name, go into the subcommand matching it
@@ -2409,8 +2431,12 @@ class App {
             return false;
         }
 
-        if(!op->get_configurable())
+        if(!op->get_configurable()) {
+            if(get_allow_config_extras() == config_extras_mode::ignore_all) {
+                return false;
+            }
             throw ConfigError::NotConfigurable(item.fullname());
+        }
 
         if(op->empty()) {
             // Flag parsing
diff --git a/include/CLI/Config.hpp b/include/CLI/Config.hpp
@@ -94,17 +94,17 @@ inline std::string ini_join(const std::vector<std::string> &args,
     return joined;
 }
 
-inline std::vector<std::string> generate_parents(const std::string &section, std::string &name) {
+inline std::vector<std::string> generate_parents(const std::string &section, std::string &name, char parentSeparator) {
     std::vector<std::string> parents;
     if(detail::to_lower(section) != "default") {
-        if(section.find('.') != std::string::npos) {
-            parents = detail::split(section, '.');
+        if(section.find(parentSeparator) != std::string::npos) {
+            parents = detail::split(section, parentSeparator);
         } else {
             parents = {section};
         }
     }
-    if(name.find('.') != std::string::npos) {
-        std::vector<std::string> plist = detail::split(name, '.');
+    if(name.find(parentSeparator) != std::string::npos) {
+        std::vector<std::string> plist = detail::split(name, parentSeparator);
         name = plist.back();
         detail::remove_quotes(name);
         plist.pop_back();
@@ -119,10 +119,11 @@ inline std::vector<std::string> generate_parents(const std::string &section, std
 }
 
 /// assuming non default segments do a check on the close and open of the segments in a configItem structure
-inline void checkParentSegments(std::vector<ConfigItem> &output, const std::string &currentSection) {
+inline void
+checkParentSegments(std::vector<ConfigItem> &output, const std::string &currentSection, char parentSeparator) {
 
     std::string estring;
-    auto parents = detail::generate_parents(currentSection, estring);
+    auto parents = detail::generate_parents(currentSection, estring, parentSeparator);
     if(!output.empty() && output.back().name == "--") {
         std::size_t msize = (parents.size() > 1U) ? parents.size() : 2;
         while(output.back().parents.size() >= msize) {
@@ -170,43 +171,53 @@ inline void checkParentSegments(std::vector<ConfigItem> &output, const std::stri
 
 inline std::vector<ConfigItem> ConfigBase::from_config(std::istream &input) const {
     std::string line;
-    std::string section = "default";
-
+    std::string currentSection = "default";
+    std::string previousSection = "default";
     std::vector<ConfigItem> output;
     bool isDefaultArray = (arrayStart == '[' && arrayEnd == ']' && arraySeparator == ',');
     bool isINIArray = (arrayStart == '\0' || arrayStart == ' ') && arrayStart == arrayEnd;
+    bool inSection{false};
     char aStart = (isINIArray) ? '[' : arrayStart;
     char aEnd = (isINIArray) ? ']' : arrayEnd;
     char aSep = (isINIArray && arraySeparator == ' ') ? ',' : arraySeparator;
-
+    int currentSectionIndex{0};
     while(getline(input, line)) {
         std::vector<std::string> items_buffer;
         std::string name;
 
         detail::trim(line);
         std::size_t len = line.length();
-        if(len > 1 && line.front() == '[' && line.back() == ']') {
-            if(section != "default") {
+        // lines have to be at least 3 characters to have any meaning to CLI just skip the rest
+        if(len < 3) {
+            continue;
+        }
+        if(line.front() == '[' && line.back() == ']') {
+            if(currentSection != "default") {
                 // insert a section end which is just an empty items_buffer
                 output.emplace_back();
-                output.back().parents = detail::generate_parents(section, name);
+                output.back().parents = detail::generate_parents(currentSection, name, parentSeparatorChar);
                 output.back().name = "--";
             }
-            section = line.substr(1, len - 2);
+            currentSection = line.substr(1, len - 2);
             // deal with double brackets for TOML
-            if(section.size() > 1 && section.front() == '[' && section.back() == ']') {
-                section = section.substr(1, section.size() - 2);
+            if(currentSection.size() > 1 && currentSection.front() == '[' && currentSection.back() == ']') {
+                currentSection = currentSection.substr(1, currentSection.size() - 2);
             }
-            if(detail::to_lower(section) == "default") {
-                section = "default";
+            if(detail::to_lower(currentSection) == "default") {
+                currentSection = "default";
             } else {
-                detail::checkParentSegments(output, section);
+                detail::checkParentSegments(output, currentSection, parentSeparatorChar);
+            }
+            inSection = false;
+            if(currentSection == previousSection) {
+                ++currentSectionIndex;
+            } else {
+                currentSectionIndex = 0;
+                previousSection = currentSection;
             }
             continue;
         }
-        if(len == 0) {
-            continue;
-        }
+
         // comment lines
         if(line.front() == ';' || line.front() == '#' || line.front() == commentChar) {
             continue;
@@ -217,6 +228,11 @@ inline std::vector<ConfigItem> ConfigBase::from_config(std::istream &input) cons
         if(pos != std::string::npos) {
             name = detail::trim_copy(line.substr(0, pos));
             std::string item = detail::trim_copy(line.substr(pos + 1));
+            auto cloc = item.find(commentChar);
+            if(cloc != std::string::npos) {
+                item.erase(cloc, std::string::npos);
+                detail::trim(item);
+            }
             if(item.size() > 1 && item.front() == aStart) {
                 for(std::string multiline; item.back() != aEnd && std::getline(input, multiline);) {
                     detail::trim(multiline);
@@ -232,18 +248,36 @@ inline std::vector<ConfigItem> ConfigBase::from_config(std::istream &input) cons
             }
         } else {
             name = detail::trim_copy(line);
+            auto cloc = name.find(commentChar);
+            if(cloc != std::string::npos) {
+                name.erase(cloc, std::string::npos);
+                detail::trim(name);
+            }
+
             items_buffer = {"true"};
         }
-        if(name.find('.') == std::string::npos) {
+        if(name.find(parentSeparatorChar) == std::string::npos) {
             detail::remove_quotes(name);
         }
         // clean up quotes on the items
         for(auto &it : items_buffer) {
             detail::remove_quotes(it);
         }
 
-        std::vector<std::string> parents = detail::generate_parents(section, name);
-
+        std::vector<std::string> parents = detail::generate_parents(currentSection, name, parentSeparatorChar);
+        if(parents.size() > maximumLayers) {
+            continue;
+        }
+        if(!configSection.empty() && !inSection) {
+            if(parents.empty() || parents.front() != configSection) {
+                continue;
+            }
+            if(configIndex >= 0 && currentSectionIndex != configIndex) {
+                continue;
+            }
+            parents.erase(parents.begin());
+            inSection = true;
+        }
         if(!output.empty() && name == output.back().name && parents == output.back().parents) {
             output.back().inputs.insert(output.back().inputs.end(), items_buffer.begin(), items_buffer.end());
         } else {
@@ -253,11 +287,11 @@ inline std::vector<ConfigItem> ConfigBase::from_config(std::istream &input) cons
             output.back().inputs = std::move(items_buffer);
         }
     }
-    if(section != "default") {
+    if(currentSection != "default") {
         // insert a section end which is just an empty items_buffer
         std::string ename;
         output.emplace_back();
-        output.back().parents = detail::generate_parents(section, ename);
+        output.back().parents = detail::generate_parents(currentSection, ename, parentSeparatorChar);
         output.back().name = "--";
         while(output.back().parents.size() > 1) {
             output.push_back(output.back());
@@ -339,17 +373,18 @@ ConfigBase::to_config(const App *app, bool default_also, bool write_description,
                 if(!prefix.empty() || app->get_parent() == nullptr) {
                     out << '[' << prefix << subcom->get_name() << "]\n";
                 } else {
-                    std::string subname = app->get_name() + "." + subcom->get_name();
+                    std::string subname = app->get_name() + parentSeparatorChar + subcom->get_name();
                     auto p = app->get_parent();
                     while(p->get_parent() != nullptr) {
-                        subname = p->get_name() + "." + subname;
+                        subname = p->get_name() + parentSeparatorChar + subname;
                         p = p->get_parent();
                     }
                     out << '[' << subname << "]\n";
                 }
                 out << to_config(subcom, default_also, write_description, "");
             } else {
-                out << to_config(subcom, default_also, write_description, prefix + subcom->get_name() + ".");
+                out << to_config(
+                    subcom, default_also, write_description, prefix + subcom->get_name() + parentSeparatorChar);
             }
         }
     }
diff --git a/include/CLI/ConfigFwd.hpp b/include/CLI/ConfigFwd.hpp
diff --git a/tests/ConfigFileTest.cpp b/tests/ConfigFileTest.cpp
