A better Help formatter (V2) (#866)

_This is the new PR I've mentioned to work on in PR #858_

## A better Help Formatter
_See below for images of the new help page_

Finally, after a lot of planning, understanding CLI11's codebase,
testing and coding, the new default Help Formatter is done. There are a
lot of changes to make the help page more readable and closer to UNIX
standards, see Changelog below for details. One of the highlights is
automatic paragraph formatting with correct line wrapping for App and
options/flag descriptions as well as the footer.
A goal was to provide more flexibility and better readability for the
help page while providing full compatibility with Apps using CLI11 (no
breaking changes and no changes to Apps required). Also better support
for different terminal sizes. Users can now specify three new optional
attributes: `right_column_width_`, `description_paragraph_width_` and
`footer_paragraph_width_`. See code documentation for more details. The
different columns for options/flags now scale with the set
`column_width_` value: Single dash flags occupy 33% of the set
`column_width_`, double dash flags and options (like REQUIRED) 66%.
These new attributes allow for indirectly respecting terminal geometry,
footer paragraph formatting has also been added (#355). This PR also
implements the issues #353 and #856.
The new help page formatting can also be used as an input for man page
generation, since it's oriented on the man page style (#413).
[help2man](https://www.gnu.org/software/help2man/) can be used to
generate man pages from help output (see comment down below for
example).

I thoroughly tested this code with all possible combinations of flags,
options, positionals, subcommands, validators, ...
So far everything works great.
I hope this PR looks good and meets all requirements. I'm looking
forward to the implementation of this PR into CLI11. If you have any
questions or suggestions feel free to comment.

### Fixed/implemented issues by this PR
- #353 Better options formatting
- #856 Space between options
- #355 Footer formatting
- #413 Man page generation can be achieved using help2man with the new
help formatting
- #384 (comment)
Better help formatting can be marked as complete

### What about the failing tests?
Of course the tests expect the old help text format. This is why 6 of
the tests are failing. Since it is a bit of work to migrate the tests to
the new help format, I first wanted to push out this PR and get
confirmation before I'll update all the tests.
So please let me know if this PR gets implemented, what changes should
be made and then I'll migrate the tests to the new help format, either
in this PR or I'll make a new one.

## Changelog:
#### There are _no breaking changes_. Every App using CLI11 will work
with this new formatter with no changes required.
- Added empty lines at beginning and end of help text
- Removed double new-line between option groups for consistency. Now all
sections have the same number of new-lines
- Switched usage and description order
- Only show "Usage"-string if no App name is present. This provides
better readability
- Made categories (Options, Positionals, ...) capital
- Changed `ConfigBase::to_config` to correctly process capital
"OPTIONS"-group (only affects descriptions of the config file, not a
breaking change)
- Added a paragraph formatter function `streamOutAsParagraph` to
StringTools.hpp
- Made "description" a paragraph block with correct, word respecting
line wrapping and indentation (using the new paragraph formatter
function)
- Made the footer a paragraph block with correct, word respecting line
wrapping and indentation
- Updated documentation for `column_width_` to make it more clear
- Added new member: `right_column_width_`, added getter and setter for
`right_column_width_`
- Added new member: `description_paragraph_width_`, added getter and
setter for `description_paragraph_width_`
- Added new member: `footer_paragraph_width_`, added getter and setter
for `footer_paragraph_width_ `
- Positionals description are now formatted as paragraph with correct,
word respecting line wrapping
- Options description are now formatted as paragraph with correct, word
respecting line wrapping
- Short and long options/flags/names are now correctly formatted to
always be at the right position (also for subcommand options/flags)
- Short and long options/flags/names column widths scale linearly with
the `column_width_` attribute to better adapt to different
`column_width_` sizes
- Merged PR #860

## What's planned for the future?
- I'm thinking of better formatting the options of flags (like REQUIRED,
TEXT, INT, ...) and make them also in a seperate column. This way they
would also always be at the same position. However I decided against it
for this PR, since I wanted them to be as close as possible to the
actual flag. With my implementation it is quite easy to add this change
in the future.
- Subcommands: I'm planning on better formatting the Subcommands. With
this PR only the short and long flags/options of subcommands are better
formatted (like it is with the main flags, see images down below).
- Maybe implement a different way to display expected data type options
(TEXT, INT, ...). For example: `--file-name=<TEXT>` for long flags only
and if `disable_flag_override_` is false.
- Maybe add something like this:
#554

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Philip Top <[email protected]>

.codecov.yml

@@ -5,3 +5,11 @@ ignore:
   - "docs"
   - "test_package"
   - "fuzz"
+
+parsers:
+  gcov:
+    branch_detection:
+      conditional: yes
+      loop: yes
+      method: no
+      macro: no

examples/CMakeLists.txt

@@ -75,8 +75,8 @@ set_property(
            "Working on count: 2, direct count: 2, opt count: 2" "Some value: 1.2")
 # test shows that the help prints out for unnamed subcommands
 add_test(NAME subcom_partitioned_help COMMAND subcom_partitioned --help)
-set_property(TEST subcom_partitioned_help PROPERTY PASS_REGULAR_EXPRESSION
-                                                   "-f,--file TEXT REQUIRED" "-d,--double FLOAT")
+set_property(TEST subcom_partitioned_help
+             PROPERTY PASS_REGULAR_EXPRESSION "-f,[ \\t]*--file TEXT REQUIRED" "-d,--double FLOAT")
 
 ####################################################
 add_cli_exe(config_app config_app.cpp)
@@ -145,8 +145,8 @@ add_cli_exe(validators validators.cpp)
 add_test(NAME validators_help COMMAND validators --help)
 set_property(
   TEST validators_help
-  PROPERTY PASS_REGULAR_EXPRESSION "  -f,--file TEXT:FILE[\\r\\n\\t ]+File name"
-           "  -v,--value INT:INT in [3 - 6][\\r\\n\\t ]+Value in range")
+  PROPERTY PASS_REGULAR_EXPRESSION "  -f,[ \\t]*--file TEXT:FILE[\\r\\n\\t ]+File name"
+           "  -v,[ \\t]*--value INT:INT in [3 - 6][\\r\\n\\t ]+Value in range")
 add_test(NAME validators_file COMMAND validators --file nonex.xxx)
 set_property(
   TEST validators_file PROPERTY PASS_REGULAR_EXPRESSION "--file: File does not exist: nonex.xxx"

fuzz/fuzzApp.cpp

@@ -44,6 +44,7 @@ std::optional<std::string>> tcomplex; std::string_view vstrv;
 std::shared_ptr<CLI::App> FuzzApp::generateApp() {
     auto fApp = std::make_shared<CLI::App>("fuzzing App", "fuzzer");
     fApp->set_config("--config");
+    fApp->set_help_all_flag("--help-all");
     fApp->add_flag("-a,--flag");
     fApp->add_flag("-b,--flag2,!--nflag2", flag1);
     fApp->add_flag("-c{34},--flag3{1}", flagCnt)->disable_flag_override();

include/CLI/App.hpp

@@ -275,7 +275,7 @@ class App {
     App *parent_{nullptr};
 
     /// The group membership INHERITABLE
-    std::string group_{"Subcommands"};
+    std::string group_{"SUBCOMMANDS"};
 
     /// Alias names for the subcommand
     std::vector<std::string> aliases_{};

include/CLI/FormatterFwd.hpp

@@ -44,9 +44,18 @@ class FormatterBase {
     /// @name Options
     ///@{
 
-    /// The width of the first column
+    /// The width of the left column (options/flags/subcommands)
     std::size_t column_width_{30};
 
+    /// The width of the right column (description of options/flags/subcommands)
+    std::size_t right_column_width_{65};
+
+    /// The width of the description paragraph at the top of help
+    std::size_t description_paragraph_width_{80};
+
+    /// The width of the footer paragraph
+    std::size_t footer_paragraph_width_{80};
+
     /// @brief The required help printout labels (user changeable)
     /// Values are Needs, Excludes, etc.
     std::map<std::string, std::string> labels_{};
@@ -75,9 +84,18 @@ class FormatterBase {
     /// Set the "REQUIRED" label
     void label(std::string key, std::string val) { labels_[key] = val; }
 
-    /// Set the column width
+    /// Set the left column width (options/flags/subcommands)
     void column_width(std::size_t val) { column_width_ = val; }
 
+    /// Set the right column width (description of options/flags/subcommands)
+    void right_column_width(std::size_t val) { right_column_width_ = val; }
+
+    /// Set the description paragraph width at the top of help
+    void description_paragraph_width(std::size_t val) { description_paragraph_width_ = val; }
+
+    /// Set the footer paragraph width
+    void footer_paragraph_width(std::size_t val) { footer_paragraph_width_ = val; }
+
     ///@}
     /// @name Getters
     ///@{
@@ -89,9 +107,18 @@ class FormatterBase {
         return labels_.at(key);
     }
 
-    /// Get the current column width
+    /// Get the current left column width (options/flags/subcommands)
     CLI11_NODISCARD std::size_t get_column_width() const { return column_width_; }
 
+    /// Get the current right column width (description of options/flags/subcommands)
+    CLI11_NODISCARD std::size_t get_right_column_width() const { return right_column_width_; }
+
+    /// Get the current description paragraph width at the top of help
+    CLI11_NODISCARD std::size_t get_description_paragraph_width() const { return description_paragraph_width_; }
+
+    /// Get the current footer paragraph width
+    CLI11_NODISCARD std::size_t get_footer_paragraph_width() const { return footer_paragraph_width_; }
+
     ///@}
 };
 
@@ -146,7 +173,7 @@ class Formatter : public FormatterBase {
     virtual std::string make_subcommand(const App *sub) const;
 
     /// This prints out a subcommand in help-all
-    virtual std::string make_expanded(const App *sub) const;
+    virtual std::string make_expanded(const App *sub, AppFormatMode mode) const;
 
     /// This prints out all the groups of options
     virtual std::string make_footer(const App *app) const;
@@ -158,19 +185,14 @@ class Formatter : public FormatterBase {
     virtual std::string make_usage(const App *app, std::string name) const;
 
     /// This puts everything together
-    std::string make_help(const App * /*app*/, std::string, AppFormatMode) const override;
+    std::string make_help(const App *app, std::string, AppFormatMode mode) const override;
 
     ///@}
     /// @name Options
     ///@{
 
     /// This prints out an option help line, either positional or optional form
-    virtual std::string make_option(const Option *opt, bool is_positional) const {
-        std::stringstream out;
-        detail::format_help(
-            out, make_option_name(opt, is_positional) + make_option_opts(opt), make_option_desc(opt), column_width_);
-        return out.str();
-    }
+    virtual std::string make_option(const Option *, bool) const;
 
     /// @brief This is the name part of an option, Default: left column
     virtual std::string make_option_name(const Option *, bool) const;

include/CLI/Option.hpp

@@ -54,7 +54,7 @@ template <typename CRTP> class OptionBase {
 
   protected:
     /// The group membership
-    std::string group_ = std::string("Options");
+    std::string group_ = std::string("OPTIONS");
 
     /// True if this is a required option
     bool required_{false};

include/CLI/StringTools.hpp

@@ -141,9 +141,6 @@ inline std::string trim_copy(const std::string &str, const std::string &filter)
     std::string s = str;
     return trim(s, filter);
 }
-/// Print a two part "help" string
-CLI11_INLINE std::ostream &
-format_help(std::ostream &out, std::string name, const std::string &description, std::size_t wid);
 
 /// Print subcommand aliases
 CLI11_INLINE std::ostream &format_aliases(std::ostream &out, const std::vector<std::string> &aliases, std::size_t wid);
@@ -263,6 +260,14 @@ CLI11_INLINE std::string extract_binary_string(const std::string &escaped_string
 /// process a quoted string, remove the quotes and if appropriate handle escaped characters
 CLI11_INLINE bool process_quoted_string(std::string &str, char string_char = '\"', char literal_char = '\'');
 
+/// This function formats the given text as a paragraph with fixed width and applies correct line wrapping
+/// with a custom line prefix. The paragraph will get streamed to the given ostrean.
+CLI11_INLINE std::ostream &streamOutAsParagraph(std::ostream &out,
+                                                const std::string &text,
+                                                std::size_t paragraphWidth,
+                                                const std::string &linePrefix = "",
+                                                bool skipPrefixOnFirstLine = false);
+
 }  // namespace detail
 
 // [CLI11:string_tools_hpp:end]

include/CLI/TypeTools.hpp

@@ -904,7 +904,7 @@ bool integral_conversion(const std::string &input, T &output) noexcept {
         nstring.erase(std::remove(nstring.begin(), nstring.end(), '\''), nstring.end());
         return integral_conversion(nstring, output);
     }
-    if(input.compare(0, 2, "0o") == 0) {
+    if(input.compare(0, 2, "0o") == 0 || input.compare(0, 2, "0O") == 0) {
         val = nullptr;
         errno = 0;
         output_ll = std::strtoull(input.c_str() + 2, &val, 8);
@@ -914,7 +914,10 @@ bool integral_conversion(const std::string &input, T &output) noexcept {
         output = static_cast<T>(output_ll);
         return (val == (input.c_str() + input.size()) && static_cast<std::uint64_t>(output) == output_ll);
     }
-    if(input.compare(0, 2, "0b") == 0) {
+    if(input.compare(0, 2, "0b") == 0 || input.compare(0, 2, "0B") == 0) {
+        // LCOV_EXCL_START
+        // In some new compilers including the coverage testing one binary strings are handled properly in strtoull
+        // automatically so this coverage is missing but is well tested in other compilers
         val = nullptr;
         errno = 0;
         output_ll = std::strtoull(input.c_str() + 2, &val, 2);
@@ -923,6 +926,7 @@ bool integral_conversion(const std::string &input, T &output) noexcept {
         }
         output = static_cast<T>(output_ll);
         return (val == (input.c_str() + input.size()) && static_cast<std::uint64_t>(output) == output_ll);
+        // LCOV_EXCL_STOP
     }
     return false;
 }
@@ -955,7 +959,7 @@ bool integral_conversion(const std::string &input, T &output) noexcept {
         nstring.erase(std::remove(nstring.begin(), nstring.end(), '\''), nstring.end());
         return integral_conversion(nstring, output);
     }
-    if(input.compare(0, 2, "0o") == 0) {
+    if(input.compare(0, 2, "0o") == 0 || input.compare(0, 2, "0O") == 0) {
         val = nullptr;
         errno = 0;
         output_ll = std::strtoll(input.c_str() + 2, &val, 8);
@@ -965,7 +969,10 @@ bool integral_conversion(const std::string &input, T &output) noexcept {
         output = static_cast<T>(output_ll);
         return (val == (input.c_str() + input.size()) && static_cast<std::int64_t>(output) == output_ll);
     }
-    if(input.compare(0, 2, "0b") == 0) {
+    if(input.compare(0, 2, "0b") == 0 || input.compare(0, 2, "0B") == 0) {
+        // LCOV_EXCL_START
+        // In some new compilers including the coverage testing one binary strings are handled properly in strtoll
+        // automatically so this coverage is missing but is well tested in other compilers
         val = nullptr;
         errno = 0;
         output_ll = std::strtoll(input.c_str() + 2, &val, 2);
@@ -974,6 +981,7 @@ bool integral_conversion(const std::string &input, T &output) noexcept {
         }
         output = static_cast<T>(output_ll);
         return (val == (input.c_str() + input.size()) && static_cast<std::int64_t>(output) == output_ll);
+        // LCOV_EXCL_STOP
     }
     return false;
 }

include/CLI/impl/App_inl.hpp

@@ -2265,11 +2265,14 @@ CLI11_INLINE void retire_option(App *app, Option *opt) {
         ->expected(option_copy->get_expected_min(), option_copy->get_expected_max())
         ->allow_extra_args(option_copy->get_allow_extra_args());
 
+    // LCOV_EXCL_START
+    // something odd with coverage on new compilers
     Validator retired_warning{[opt2](std::string &) {
                                   std::cout << "WARNING " << opt2->get_name() << " is retired and has no effect\n";
                                   return std::string();
                               },
                               ""};
+    // LCOV_EXCL_STOP
     retired_warning.application_index(0);
     opt2->check(retired_warning);
 }
@@ -2287,11 +2290,14 @@ CLI11_INLINE void retire_option(App *app, const std::string &option_name) {
                      ->type_name("RETIRED")
                      ->expected(0, 1)
                      ->default_str("RETIRED");
+    // LCOV_EXCL_START
+    // something odd with coverage on new compilers
     Validator retired_warning{[opt2](std::string &) {
                                   std::cout << "WARNING " << opt2->get_name() << " is retired and has no effect\n";
                                   return std::string();
                               },
                               ""};
+    // LCOV_EXCL_STOP
     retired_warning.application_index(0);
     opt2->check(retired_warning);
 }

include/CLI/impl/Config_inl.hpp

@@ -517,26 +517,26 @@ ConfigBase::to_config(const App *app, bool default_also, bool write_description,
 
     std::vector<std::string> groups = app->get_groups();
     bool defaultUsed = false;
-    groups.insert(groups.begin(), std::string("Options"));
+    groups.insert(groups.begin(), std::string("OPTIONS"));
     if(write_description && (app->get_configurable() || app->get_parent() == nullptr || app->get_name().empty())) {
         out << commentLead << detail::fix_newlines(commentLead, app->get_description()) << '\n';
     }
     for(auto &group : groups) {
-        if(group == "Options" || group.empty()) {
+        if(group == "OPTIONS" || group.empty()) {
             if(defaultUsed) {
                 continue;
             }
             defaultUsed = true;
         }
-        if(write_description && group != "Options" && !group.empty()) {
+        if(write_description && group != "OPTIONS" && !group.empty()) {
             out << '\n' << commentLead << group << " Options\n";
         }
         for(const Option *opt : app->get_options({})) {
 
             // Only process options that are configurable
             if(opt->get_configurable()) {
                 if(opt->get_group() != group) {
-                    if(!(group == "Options" && opt->get_group().empty())) {
+                    if(!(group == "OPTIONS" && opt->get_group().empty())) {
                         continue;
                     }
                 }