1 # Pattern Matching for Tomo
   2 
   3 This is a lightweight pattern matching library for
   4 [Tomo](https://tomo.bruce-hill.com) as an alternative to regular expressions.
   5 It is intended to solve 80% of use cases in under 1% of the code size (PCRE's
   6 codebase is roughly 150k lines of code, and Tomo's pattern matching code is a
   7 bit under 1k lines of code). Tomo's pattern matching syntax is highly readable
   8 and works well for matching literal text without getting [leaning toothpick
   9 syndrome](https://en.wikipedia.org/wiki/Leaning_toothpick_syndrome).
  10 
  11 For more advanced use cases, consider linking against a C library for regular
  12 expressions or pattern matching.
  13 
  14 `Pat` is a [domain-specific language](docs/langs.md), in other words, it's
  15 like a `Text`, but it has a distinct type.
  16 
  17 Patterns are used in a small, but very powerful API that handles many text
  18 functions that would normally be handled by a more extensive API:
  19 
  20 - [`by_pattern(text:Text, pattern:Pat -> func(->PatternMatch?))`](#by_pattern)
  21 - [`by_pattern_split(text:Text, pattern:Pat -> func(->Text?))`](#by_pattern_split)
  22 - [`each_pattern(text:Text, pattern:Pat, fn:func(m:PatternMatch), recursive=yes)`](#each_pattern)
  23 - [`find_patterns(text:Text, pattern:Pat -> [PatternMatch])`](#find_patterns)
  24 - [`has_pattern(text:Text, pattern:Pat -> Bool)`](#has_pattern)
  25 - [`map_pattern(text:Text, pattern:Pat, fn:func(m:PatternMatch -> Text), recursive=yes -> Text)`](#map_pattern)
  26 - [`matches_pattern(text:Text, pattern:Pat -> Bool)`](#matches_pattern)
  27 - [`pattern_captures(text:Text, pattern:Pat -> [Text]?)`](#pattern_captures)
  28 - [`replace_pattern(text:Text, pattern:Pat, replacement:Text, backref="@", recursive=yes -> Text)`](#replace_pattern)
  29 - [`split_pattern(text:Text, pattern:Pat -> [Text])`](#split_pattern)
  30 - [`translate_patterns(text:Text, replacements:{Pat,Text}, backref="@", recursive=yes -> Text)`](#translate_patterns)
  31 - [`trim_pattern(text:Text, pattern=$Pat"{space}", left=yes, right=yes -> Text)`](#trim_pattern)
  32 
  33 ## Matches
  34 
  35 Pattern matching functions work with a type called `PatternMatch` that has three fields:
  36 
  37 - `text`: The full text of the match.
  38 - `index`: The index in the text where the match was found.
  39 - `captures`: A list containing the matching text of each non-literal pattern group.
  40 
  41 See [Text Functions](text.md#Text-Functions) for the full API documentation.
  42 
  43 ## Syntax
  44 
  45 Patterns have three types of syntax:
  46 
  47 - `{` followed by an optional count (`n`, `n-m`, or `n+`), followed by an
  48   optional `!` to negate the pattern, followed by an optional pattern name or
  49   Unicode character name, followed by a required `}`.
  50 
  51 - Any matching pair of quotes or parentheses or braces with a `?` in the middle
  52   (e.g. `"?"` or `(?)`).
  53 
  54 - Any other character is treated as a literal to be matched exactly.
  55 
  56 ## Named Patterns
  57 
  58 Named patterns match certain pre-defined patterns that are commonly useful. To
  59 use a named pattern, use the syntax `{name}`. Names are case-insensitive and
  60 mostly ignore spaces, underscores, and dashes.
  61 
  62 - `..` - Any character (note that a single `.` would mean the literal period
  63   character).
  64 - `digit` - A unicode digit
  65 - `email` - an email address
  66 - `emoji` - an emoji
  67 - `end` - the very end of the text
  68 - `id` - A unicode identifier
  69 - `int` - One or more digits with an optional `-` (minus sign) in front
  70 - `ip` - an IP address (IPv4 or IPv6)
  71 - `ipv4` - an IPv4 address
  72 - `ipv6` - an IPv6 address
  73 - `nl`/`newline`/`crlf` - A line break (either `\r\n` or `\n`)
  74 - `num` - One or more digits with an optional `-` (minus sign) in front and an optional `.` and more digits after
  75 - `start` - the very start of the text
  76 - `uri` - a URI
  77 - `url` - a URL (URI that specifically starts with `http://`, `https://`, `ws://`, `wss://`, or `ftp://`)
  78 - `word` - A unicode identifier (same as `id`)
  79 
  80 For non-alphabetic characters, any single character is treated as matching
  81 exactly that character. For example, `{1{}` matches exactly one `{`
  82 character. Or, `{1.}` matches exactly one `.` character.
  83 
  84 Patterns can also use any Unicode property name. Some helpful ones are:
  85 
  86 - `hex` - Hexidecimal digits
  87 - `lower` - Lowercase letters
  88 - `space` - The space character
  89 - `upper` - Uppercase letters
  90 - `whitespace` - Whitespace characters
  91 
  92 Patterns may also use exact Unicode codepoint names. For example: `{1 latin
  93 small letter A}` matches `a`.
  94 
  95 ## Negating Patterns
  96 
  97 If an exclamation mark (`!`) is placed before a pattern's name, then characters
  98 are matched only when they _don't_ match the pattern. For example, `{!alpha}`
  99 will match all characters _except_ alphabetic ones.
 100 
 101 ## Interpolating Text and Escaping
 102 
 103 To escape a character in a pattern (e.g. if you want to match the literal
 104 character `?`), you can use the syntax `{1 ?}`. This is almost never necessary
 105 unless you have text that looks like a Tomo text pattern and has something like
 106 `{` or `(?)` inside it.
 107 
 108 However, if you're trying to do an exact match of arbitrary text values, you'll
 109 want to have the text automatically escaped. Fortunately, Tomo's injection-safe
 110 DSL text interpolation supports automatic text escaping. This means that if you
 111 use text interpolation with the `$` sign to insert a text value, the value will
 112 be automatically escaped using the `{1 ?}` rule described above:
 113 
 114 ```tomo
 115 # Risk of code injection (would cause an error because 'xxx' is not a valid
 116 # pattern name:
 117 >> user_input := get_user_input()
 118 = "{xxx}"
 119 
 120 # Interpolation automatically escapes:
 121 >> $/$user_input/
 122 = $/{1{}..xxx}/
 123 
 124 # This is: `{ 1{ }` (one open brace) followed by the literal text "..xxx}"
 125 
 126 # No error:
 127 >> some_text.find($/$user_input/)
 128 = 0
 129 ```
 130 
 131 If you prefer, you can also use this to insert literal characters:
 132 
 133 ```tomo
 134 >> $/literal $"{..}"/
 135 = $/literal {1{}..}/
 136 ```
 137 
 138 ## Repetitions
 139 
 140 By default, named patterns match 1 or more repetitions, but you can specify how
 141 many repetitions you want by putting a number or range of numbers first using
 142 `n` (exactly `n` repetitions), `n-m` (between `n` and `m` repetitions), or `n+`
 143 (`n` or more repetitions):
 144 
 145 ```
 146 {4-5 alpha}
 147 0x{hex}
 148 {4 digit}-{2 digit}-{2 digit}
 149 {2+ space}
 150 {0-1 question mark}
 151 ```
 152 
 153 
 154 # Methods
 155 
 156 ### `by_pattern`
 157 Returns an iterator function that yields `PatternMatch` objects for each occurrence.
 158 
 159 ```tomo
 160 func by_pattern(text:Text, pattern:Pat -> func(->PatternMatch?))
 161 ```
 162 
 163 - `text`: The text to search.
 164 - `pattern`: The pattern to match.
 165 
 166 **Returns:**
 167 An iterator function that yields `PatternMatch` objects one at a time.
 168 
 169 **Example:**
 170 ```tomo
 171 text := "one, two, three"
 172 for word in text.by_pattern($Pat"{id}"):
 173     say(word.text)
 174 ```
 175 
 176 ---
 177 
 178 ### `by_pattern_split`
 179 Returns an iterator function that yields text segments split by a pattern.
 180 
 181 ```tomo
 182 func by_pattern_split(text:Text, pattern:Pat -> func(->Text?))
 183 ```
 184 
 185 - `text`: The text to split.
 186 - `pattern`: The pattern to use as a separator.
 187 
 188 **Returns:**
 189 An iterator function that yields text segments.
 190 
 191 **Example:**
 192 ```tomo
 193 text := "one two three"
 194 for word in text.by_pattern_split($Pat"{whitespace}"):
 195     say(word.text)
 196 ```
 197 
 198 ---
 199 
 200 ### `each_pattern`
 201 Applies a function to each occurrence of a pattern in the text.
 202 
 203 ```tomo
 204 func each_pattern(text:Text, pattern:Pat, fn:func(m:PatternMatch), recursive=yes)
 205 ```
 206 
 207 - `text`: The text to search.
 208 - `pattern`: The pattern to match.
 209 - `fn`: The function to apply to each match.
 210 - `recursive`: If `yes`, applies the function recursively on modified text.
 211 
 212 **Example:**
 213 ```tomo
 214 text := "one two three"
 215 text.each_pattern($Pat"{id}", func(m:PatternMatch):
 216     say(m.txt)
 217 )
 218 ```
 219 
 220 ---
 221 
 222 ### `find_patterns`
 223 Finds all occurrences of a pattern in a text and returns them as `PatternMatch` objects.
 224 
 225 ```tomo
 226 func find_patterns(text:Text, pattern:Pat -> [PatternMatch])
 227 ```
 228 
 229 - `text`: The text to search.
 230 - `pattern`: The pattern to match.
 231 
 232 **Returns:**
 233 A list of `PatternMatch` objects.
 234 
 235 **Example:**
 236 ```tomo
 237 text := "one! two three!"
 238 >> text.find_patterns($Pat"{id}!")
 239 = [PatternMatch(text="one!", index=1, captures=["one"]), PatternMatch(text="three!", index=10, captures=["three"])]
 240 ```
 241 
 242 ---
 243 
 244 ### `has_pattern`
 245 Checks whether a given pattern appears in the text.
 246 
 247 ```tomo
 248 func has_pattern(text:Text, pattern:Pat -> Bool)
 249 ```
 250 
 251 - `text`: The text to search.
 252 - `pattern`: The pattern to check for.
 253 
 254 **Returns:**
 255 `yes` if a match is found, otherwise `no`.
 256 
 257 **Example:**
 258 ```tomo
 259 text := "...okay..."
 260 >> text.has_pattern($Pat"{id}")
 261 = yes
 262 ```
 263 
 264 ---
 265 
 266 ### `map_pattern`
 267 Transforms matches of a pattern using a mapping function.
 268 
 269 ```tomo
 270 func map_pattern(text:Text, pattern:Pat, fn:func(m:PatternMatch -> Text), recursive=yes -> Text)
 271 ```
 272 
 273 - `text`: The text to modify.
 274 - `pattern`: The pattern to match.
 275 - `fn`: A function that transforms matches.
 276 - `recursive`: If `yes`, applies transformations recursively.
 277 
 278 **Returns:**
 279 A new text with the transformed matches.
 280 
 281 **Example:**
 282 ```tomo
 283 text := "I have #apples and #oranges and #plums"
 284 fruits := {"apples"=4, "oranges"=5}
 285 >> text.map_pattern($Pat'#{id}', func(match:PatternMatch):
 286     fruit := match.captures[1]
 287     "$(fruits[fruit] or 0) $fruit"
 288 )
 289 = "I have 4 apples and 5 oranges and 0 plums"
 290 ```
 291 
 292 ---
 293 
 294 ### `matches_pattern`
 295 Returns whether or not text matches a pattern completely.
 296 
 297 ```tomo
 298 func matches_pattern(text:Text, pattern:Pat -> Bool)
 299 ```
 300 
 301 - `text`: The text to match against.
 302 - `pattern`: The pattern to match.
 303 
 304 **Returns:**
 305 `yes` if the whole text matches the pattern, otherwise `no`.
 306 
 307 **Example:**
 308 ```tomo
 309 >> "Hello!!!".matches_pattern($Pat"{id}")
 310 = no
 311 >> "Hello".matches_pattern($Pat"{id}")
 312 = yes
 313 ```
 314 
 315 ---
 316 
 317 ### `pattern_captures`
 318 Returns a list of pattern captures for the given pattern.
 319 
 320 ```tomo
 321 func pattern_captures(text:Text, pattern:Pat -> [Text]?)
 322 ```
 323 
 324 - `text`: The text to match against.
 325 - `pattern`: The pattern to match.
 326 
 327 **Returns:**
 328 An optional list of matched pattern captures. Returns `none` if the text does
 329 not match the pattern.
 330 
 331 **Example:**
 332 ```tomo
 333 >> "123 boxes".pattern_captures($Pat"{int} {id}")
 334 = ["123", "boxes"]?
 335 >> "xxx".pattern_captures($Pat"{int} {id}")
 336 = none
 337 ```
 338 
 339 ---
 340 
 341 ### `replace_pattern`
 342 Replaces occurrences of a pattern with a replacement text, supporting backreferences.
 343 
 344 ```tomo
 345 func replace_pattern(text:Text, pattern:Pat, replacement:Text, backref="@", recursive=yes -> Text)
 346 ```
 347 
 348 - `text`: The text to modify.
 349 - `pattern`: The pattern to match.
 350 - `replacement`: The text to replace matches with.
 351 - `backref`: The symbol for backreferences in the replacement.
 352 - `recursive`: If `yes`, applies replacements recursively.
 353 
 354 **Returns:**
 355 A new text with replacements applied.
 356 
 357 **Example:**
 358 ```tomo
 359 >> "I have 123 apples and 456 oranges".replace_pattern($Pat"{int}", "some")
 360 = "I have some apples and some oranges"
 361 
 362 >> "I have 123 apples and 456 oranges".replace_pattern($Pat"{int}", "(@1)")
 363 = "I have (123) apples and (456) oranges"
 364 
 365 >> "I have 123 apples and 456 oranges".replace_pattern($Pat"{int}", "(?1)", backref="?")
 366 = "I have (123) apples and (456) oranges"
 367 
 368 >> "bad(fn(), bad(notbad))".replace_pattern($Pat"bad(?)", "good(@1)")
 369 = "good(fn(), good(notbad))"
 370 
 371 >> "bad(fn(), bad(notbad))".replace_pattern($Pat"bad(?)", "good(@1)", recursive=no)
 372 = "good(fn(), bad(notbad))"
 373 ```
 374 
 375 ---
 376 
 377 ### `split_pattern`
 378 Splits a text into segments using a pattern as the delimiter.
 379 
 380 ```tomo
 381 func split_pattern(text:Text, pattern:Pat -> [Text])
 382 ```
 383 
 384 - `text`: The text to split.
 385 - `pattern`: The pattern to use as a separator.
 386 
 387 **Returns:**
 388 A list of text segments.
 389 
 390 **Example:**
 391 ```tomo
 392 >> "one two three".split_pattern($Pat"{whitespace}")
 393 = ["one", "two", "three"]
 394 ```
 395 
 396 ---
 397 
 398 ### `translate_patterns`
 399 Replaces multiple patterns using a mapping of patterns to replacement texts.
 400 
 401 ```tomo
 402 func translate_patterns(text:Text, replacements:{Pat,Text}, backref="@", recursive=yes -> Text)
 403 ```
 404 
 405 - `text`: The text to modify.
 406 - `replacements`: A table mapping patterns to their replacements.
 407 - `backref`: The symbol for backreferences in replacements.
 408 - `recursive`: If `yes`, applies replacements recursively.
 409 
 410 **Returns:**
 411 A new text with all specified replacements applied.
 412 
 413 **Example:**
 414 ```tomo
 415 >> text := "foo(x, baz(1))"
 416 >> text.translate_patterns({
 417     $Pat"{id}(?)"="call(fn('@1'), @2)",
 418     $Pat"{id}"="var('@1')",
 419     $Pat"{int}"="int(@1)",
 420 })
 421 = "call(fn('foo'), var('x'), call(fn('baz'), int(1)))"
 422 ```
 423 
 424 ---
 425 
 426 ### `trim_pattern`
 427 Removes matching patterns from the beginning and/or end of a text.
 428 
 429 ```tomo
 430 func trim_pattern(text:Text, pattern=$Pat"{space}", left=yes, right=yes -> Text)
 431 ```
 432 
 433 - `text`: The text to trim.
 434 - `pattern`: The pattern to trim (defaults to whitespace).
 435 - `left`: If `yes`, trims from the beginning.
 436 - `right`: If `yes`, trims from the end.
 437 
 438 **Returns:**
 439 The trimmed text.
 440 
 441 **Example:**
 442 ```tomo
 443 >> "123abc456".trim_pattern($Pat"{digit}")
 444 = "abc"
 445 ```