tomo/docs/strings.md

# Strings

Strings are implemented as immutable UTF-8-encoded values using:

- The Boehm Cord library for efficient storage and concatenation.
- GNU libunistring for unicode functionality (grapheme cluster counts,
  capitalization, etc.)
- My own BP library for simple pattern matching operations (similar to regex)

## Syntax

Strings have a flexible syntax designed to make it easy to hold values from
different languages without the need to have lots of escape sequences and
without using printf-style string formatting.

```
// Basic string:
str := "Hello world"
str2 := 'Also a string'
```

## Line Splits

Long strings can be split across multiple lines by having two or more dots at
the start of a new line on the same indentation level that started the string:

```
str := "This is a long
.... line that is split in code"
```

## Multi-line Strings

Multi-line strings have indented (i.e. at least one tab more than the start of
the string) text inside quotation marks. The leading and trailing newline are
ignored:

```
multi_line := "
    This string has multiple lines.
    Line two.

    You can split a line
.... using two or more dots to make an elipsis.

    Remember to include whitespace after the elipsis if desired.

    Or don't if you're splitting a long word like supercalifragilisticexpia
....lidocious

        This text is indented by one level in the string

    "quotes" are ignored unless they're at the same indentation level as the
.... start of the string.

    The end (no newline after this).
"
```

## String Interpolations

Inside a double quoted string, you can use curly braces (`{...}`) to insert an
expression that you want converted to a string. This is called string
interpolation:

```
// Interpolation:
str := "Sum: {1 + 2}"
// equivalent to "Sum: 3"
```

Single-quoted strings do not have interpolations:

```
// No interpolation here:
str := 'Sum: {1 + 2}'
```

## String Escapes

Unlike other languages, backslash is *not* a special character inside of a
string. For example, `"x\ny"` has the characters `x`, `\`, `n`, `y`, not a
newline. Instead, a series of character escapes act as complete string literals
without quotation marks:

```
newline := \n
crlf := \r\n
quote := \"
```

These string literals can be used as interpolation values:

```
two_lines := "one{\n}two"
has_quotes := "some {\"}quotes{\"} here"
```

However, in general it is best practice to use multi-line strings to avoid these problems:

```
str := "
    This has
    multiple lines and "quotes" too!
"
```

### Multi-line Strings

There are two reasons for strings to span multiple lines in code: either you
have a string that contains newlines and you want to represent it without `\n`
escapes, or you have a long single-line string that you want to split across
multiple lines for readability. To support this, you can use newlines inside of
strings with indentation-sensitivity. For splitting long lines, use two or more
"."s at the same indentation level as the start of the string literal:

```
single_line := "This is a long string that
.... spans multiple lines"
```
For strings that contain newlines, you may put multiple indented lines inside
the quotes:

```
multi_line := "
    line one
    line two
        this line is indented
    last line
"
```

Strings may only end on lines with the same indentation as the starting quote
and nested quotes are ignored:

```
multi_line := "
    Quotes in indented regions like this: " don't count
"
```

If there is a leading or trailing newline, it is ignored and not included in
the string.

```
str := "
    one line
"

>>> str == "one line"
=== yes
```

Additional newlines *are* counted though:

```
str := "
    
    blank lines

"

>>> str == "{\n}blank lines{\n}"
```

### Advanced $-Strings

Sometimes you need to use many `{`s or `"`s inside a string, but you don't want
to type `{\{}` or `{\"}` each time. In such cases, you can use the more
advanced form of strings. The advanced form lets you explicitly specify which
characters are used for interpolation and which characters are used for
opening/closing the string. Advanced strings begin with a dollar sign (`$`),
followed by what interpolation style to use, followed by the character to use
to delimit the string, followed by the string contents and a closing string
delimiter. The interpolation style can be a matching pair (`()`, `[]`, `{}`, or
`<>`) or any other single character. When the interpolation style is a matching
pair, the interpolation is any expression enclosed in that pair (e.g.
`${}"interpolate {1 + 2}"`). When the interpolation style is a single
character, the interpolation must be either a parenthesized expression or a
single term with no infix operators (e.g. a variable), for example:
`$@"Interpolate @var or @(1 + 2)"`.

Here are some examples:

```
$[]"In here, quotes delimit the string and square brackets interpolate: [1 + 2]"
$@"For single-letter interpolations, the interpolation is a single term like @my_var without a closing symbol"
$@"But you can parenthesize expressions like: @(x + y) if you need to"
$$"Double dollars means dollar signs interpolate: $my_var $(1 + 2)"
$${If you have a string with "quotes" and 'single quotes', you can choose something else like curly braces to delimit the string}
$?#Here hashes delimit the string and question marks interpolate: ?(1 + 2)#
```

When strings are delimited by matching pairs (`()`, `[]`, `{}`, or `<>`), they
can only be closed by a matched closing character at the same indentation
level, ignoring nested pairs:

```
$$(Inside parens, you can have (nested ()) parens no problem)
$$"But only (), [], {}, and <> are matching pairs, you can't have nested quotes"
$$(
    When indented, an unmatched ) won't close the string
    An unmatched ( won't mess things up either
    Only matching pairs on the same indentation level are counted:
)
$$(Multi-line string with nested (parens) and
.. line continuation)
```

As a special case, when you use the same character for interpolation and string
delimiting, no interpolations are allowed:

```
plain := $""This string has {no interpolations}!"
```

**Note:** Normal doubly quoted strings with no dollar sign (e.g. `"foo"`) are a
shorthand for `${}"foo"`. Singly quoted strings with no dollar sign (e.g.
`'foo'`) are shorthand for `$''foo'`.

## Operations

### Concatenation

Concatenation in the typical case is an O(1) operation: `"{x}{y}"` or `x ++ y`.

Because string concatenation is typically an O(1) operation, there is no need
for a separate string builder class in the language and no need to use an array
of string fragments.

### String Length

String length is an ambiguous term in the context of UTF-8 strings. There are
several possible meanings, so each of these meanings is split into a separate
method:

- Number of grapheme clusters: `string:num_graphemes()`
- Size in bytes: `string:num_bytes()`
- Number of unicode codepoints: `string:num_codepoints()` (you probably want to
  use graphemes, not codepoints in most applications)

Since the typical user expectation is that string length refers to "letters,"
the `#` length operator returns the number of grapheme clusters, which is the
closest unicode equivalent to "letters."

### Iteration

Iteration is *not* supported for strings because of the ambiguity between
bytes, codepoints, and graphemes. It is instead recommended that you explicitly
iterate over bytes, codepoints, graphemes, words, lines, etc:

### Subcomponents

- `string:bytes()` returns an array of `Int8` bytes
- `string:codepoints()` returns an array of `Int32` bytes
- `string:graphemes()` returns an array of grapheme cluster strings
- `string:words()` returns an array of word strings
- `string:lines()` returns an array of line strings
- `string:split(",", empty=no)` returns an array of strings split by the given delimiter

### Equality, Comparison, and Hashing

All text is compared and hashed using unicode normalization. Unicode provides
several different ways to represent the same text. For example, the single
codepoint `U+E9` (latin small e with accent) is rendered the same as the two
code points `U+65 U+301` (latin small e, acute combining accent) and has an
equivalent linguistic meaning. These are simply different ways to represent the
same "letter." In order to make it easy to write correct code that takes this
into account, Tomo uses unicode normalization for all string comparisons and
hashing. Normalization does the equivalent of converting text to a canonical
form before performing comparisons or hashing. This means that if a table is
created that has text with the codepoint `U+E9` as a key, then a lookup with
the same text but with `U+65 U+301` instead of `U+E9` will still succeed in
finding the value because the two strings are equivalent under normalization.

### Capitalization

- `x:capitalized()`
- `x:titlecased()`
- `x:uppercased()`
- `x:lowercased()`

### Patterns

- `string:has("target", at=Anywhere:enum(Anywhere, Start, End))->Bool` Check whether a pattern can be found
- `string:without("target", at=Anywhere:enum(Anywhere, Start, End))->Text`
- `string:trimmed("chars...", at=Anywhere:enum(Anywhere, Start, End))->Text`
- `string:find("target")->enum(Failure, Success(index:Int32))`
- `string:replace("target", "replacement", limit=Int.max)->Text` Returns a copy of the string with replacements
- `string:split("split")->[Text]`
- `string:join(["one", "two"])->Text`