Provide a parser termination character

[NWilson]

I'm a little surprised we don't seem to have this, and no-one has asked for it before (apparently). The PCRE2 rules for processing backslashes, including \Q...\E, are rather complicated, making it hard to do a scan of a regex to find where it ends.

For example, suppose an application has a text format or syntax that encloses a regex in /.../im slashes with trailing options. The application calls PCRE2 to parse when it encounters the opening slash. But how does the application know where it ends?

Currently, I believe that the application has to duplicate a fair amount of backslash escaping logic, and work out where the first (unescaped) slash is, and pass in the contents of the regex to PCRE2.

It would be preferable, to me, to offer a method on the compile context to set a terminator character. When PCRE2 encounters this, it stops, as if it had reached the end of the pattern, and the compile context can also return the terminator position (or NULL if end-of-string was encountered without seeing a terminator).

[carenas]

The application calls PCRE2 to parse when it encounters the opening slash. But how does the application know where it ends?

The application SAYS where it ends, by either telling pcre2_compile() how many PCRE2_UCHAR sized elements it has on its length parameter, or by using PCRE2_ZERO_TERMINATED, which implies that there is none and we are operating in a PCRE2_SPTR that is NUL terminated (C string or equivalent for the non 8 bit libraries).

How the application represents the pattern internally, is of no concern to PCRE2, and therefore it is expected that it might be converted from the original before being sent (IMHO another reason why we use const for PCRE2_SPTR, as it is also immutable), but I can see how adding a "special" character to help the parser might work.

Indeed, because reading the pattern was done with *gets(), at least in Windows an encoded ctrl+Z does that job already, and that has been documented.

[NWilson]

There is another option here. Languages like JavaScript integrate the syntax of their regex literals into the language itself (in the form of var foo = /abc/i literals). This is done in the language grammar by using the actual regex syntax to define when a "significant" slash is encountered to terminate the regex. The full regex parser must be run to find the end of the literal.

However Perl uses a different model. It changes the way it parses based on the options at the end of the regex, so it does a pre-scan to find the terminator; extracts the options bits; and then run the regex parser on the known full string.

As a consequence:

var foo = /a[/b]c/i; // Valid JavaScript; knows when the terminator is at top level

# NOT valid Perl
# Pre-scan for terminating '/' is completely blind to internal regex syntax
# It does however have some ..."funky" handling of backslash. '\\' and '\<TERM>'
# are special-cased.
my $regex = qr/a[/b]c/;
# => Unknown regexp modifier "/b"

So, I'm not actually sure whether applications are likely to want to use PCRE2's parser to determine the end of a regex (depending on whether they want JS-style or Perl-style syntax paradigm).

We could of course provide both:

1. a parser mode for handling the terminator
2. a utility function to implement a Perl-compatible pre-scan for a terminator

[carenas]

the way Perl suggests handling ambiguous terminators on their RE is by using alternate characters for them:

use strict;
use warnings;

my $regex = qr|a[/b]|;
my @strings = ("a", "a/", "ab");

foreach my $s (@strings) {
	print "$s MATCH\n" if ($s =~ m/$regex/);
}

this is also the reason why pcre2test supports alternate characters as well.

[PhilipHazel]

IIRC pcre2test just looks for an unescaped delimiter. I suspect it would not skip a delimiter between \Q...\E, which, IMHO, is not a problem.