1 # Text
   2 
   3 `Text` is Tomo's datatype to represent text. The name `Text` is used instead of
   4 "string" because Tomo text represents immutable, normalized unicode data with
   5 fast indexing that has a tree-based implementation that is efficient for
   6 concatenation. These are _not_ C-style NUL-terminated character arrays. GNU
   7 libunistring is used for full Unicode functionality (grapheme cluster counts,
   8 capitalization, etc.).
   9 
  10 ## Implementation
  11 
  12 Internally, Tomo text's implementation is based on [Raku/MoarVM's
  13 strings](https://docs.raku.org/language/unicode) and [Boehm et al's
  14 Cords/Ropes](https://www.cs.tufts.edu/comp/150FP/archive/hans-boehm/ropes.pdf).
  15 Texts store their grapheme cluster count and either a compact list of 8-bit
  16 ASCII characters (for ASCII text), a list of 32-bit normal-form grapheme cluster
  17 values (see below), a compressed form of grapheme clusters with a lookup table,
  18 or a (roughly) balanced binary tree representing a concatenation. The upside of
  19 this approach is that repeated concatenations are typically a constant-time
  20 operation, which will occasionally require a small rebalancing operation. Text
  21 is stored in a format that is highly memory-efficient and index-based text
  22 operations (like retrieving an arbitrary index or slicing) are very fast:
  23 typically a constant-time operation for arbitrary unicode text, but in the worst
  24 case scenario (text built from many concatenations), `O(log(n))` time with very
  25 generous constant factors typically amounting to only a handful of steps. Since
  26 concatenations use shared substructures, they are very memory-efficient for
  27 applications where you want to store many versions of a large text with
  28 modifications. For example, if you are implementing a text editor, you can
  29 naively store the full text contents of a file at each point in its edit
  30 history, and it will only have a small memory footprint because of shared
  31 substructures in the text.
  32 
  33 ### Normal-Form Graphemes
  34 
  35 In order to maintain compact storage, fast indexing, and fast slicing,
  36 non-ASCII text is stored as 32-bit normal-form graphemes. A normal-form
  37 grapheme is either a positive value representing a Unicode codepoint that
  38 corresponds to a grapheme cluster (most Unicode letters used in natural
  39 language fall into this category after normalization) or a negative value
  40 representing an index into an internal list of "synthetic grapheme cluster
  41 codepoints." Here are some examples:
  42 
  43 - `A` is a normal codepoint that is also a grapheme cluster, so it would
  44   be represented as the number `65`
  45 - `ABC` is three separate grapheme clusters, one for `A`, one for `B`, one
  46   for `C`.
  47 - `Å` is also a single codepoint (`LATIN CAPITAL LETTER A WITH RING ABOVE`)
  48   that is also a grapheme cluster, so it would be represented as the number
  49   `197`.
  50 - `家` (Japanese for "house") is a single codepoint (`CJK Unified
  51   Ideograph-5BB6`) that is also a grapheme cluster, so it would be represented
  52   as the number `23478`
  53 - `👩🏽‍🚀` is a single graphical cluster, but it's made up of several
  54   combining codepoints (`["WOMAN", "EMOJI MODIFIER FITZPATRICK TYPE-4", "ZERO
  55   WITDH JOINER", "ROCKET"]`). Since this can't be represented with a single
  56   codepoint, we must create a synthetic codepoint for it. If this was the 3rd
  57   synthetic codepoint that we've found, then we would represent it with the
  58   number `-3`, which can be used to look it up in a lookup table. The lookup
  59   table holds the full sequence of codepoints used in the grapheme cluster.
  60 
  61 ### Text Compression
  62 
  63 One common property in natural language text is that most text is comprised of a
  64 relatively small set of grapheme clusters that are frequently reused. To take
  65 advantage of this property, Tomo's Text implementation scans along in new text
  66 objects looking for spans of text that use 256 or fewer unique grapheme clusters
  67 with many repetitions. If such a span is found, the text is stored using a small
  68 translation table and one 8-bit unsigned integer per grapheme cluster in that
  69 chunk (which is possible when there are 256 or fewer clusters in the text). This
  70 means that, for the cost of a small array of 32-bit integers, we can store the
  71 entire text using only one byte per grapheme cluster instead of a full 32-bit
  72 integer. For example, a Greek-language text will typically use the Greek
  73 alphabet, plus a few punctuation marks. Thus, if you have a text with a few
  74 thousand Greek letters, we can efficiently store the text as a small lookup
  75 table of around a hundred 32-bit integers and use one byte per "letter" in the
  76 text. This representation is around 4x more efficient than using the UTF32
  77 representation to store each Unicode codepoint as a 32-bit integer, and it's
  78 about 2x more efficient than using UTF8 to store each non-ASCII codepoint as a
  79 multi-byte sequence. Different languages will have different efficiencies, but
  80 in general, text will be stored significantly more efficiently than UTF32 and
  81 somewhat more efficiently than UTF8. However, the big advantage of this approach
  82 is that we get the ability to do constant-time random access of grapheme
  83 clusters, while getting space efficiency that is almost always better than
  84 variable-width UTF8 encoding (which does not support fast random access of any
  85 kind).
  86 
  87 ## Syntax
  88 
  89 Text has a flexible syntax designed to make it easy to hold values from
  90 different languages without the need to have lots of escape sequences and
  91 without using printf-style string formatting.
  92 
  93 ```
  94 // Basic text:
  95 str := "Hello world"
  96 str2 := 'Also text'
  97 str3 := `Backticks too`
  98 ```
  99 
 100 ### Line Splits
 101 
 102 Long text can be split across multiple lines by having two or more dots at the
 103 start of a new line on the same indentation level that started the text:
 104 
 105 ```
 106 str := "This is a long
 107 ....... line that is split in code"
 108 ```
 109 
 110 ### Multi-line Text
 111 
 112 Multi-line text has indented (i.e. at least one tab more than the start of the
 113 text) text inside quotation marks. The leading and trailing newline are
 114 ignored:
 115 
 116 ```
 117 multi_line := "
 118     This text has multiple lines.
 119     Line two.
 120 
 121     You can split a line
 122 .... using two or more dots to make an elipsis.
 123 
 124     Remember to include whitespace after the elipsis if desired.
 125 
 126     Or don't if you're splitting a long word like supercalifragilisticexpia
 127 ....lidocious
 128 
 129         This text is indented by one level in the text
 130 
 131     "quotes" are ignored unless they're at the same indentation level as the
 132 .... start of the text.
 133 
 134     The end (no newline after this).
 135 "
 136 ```
 137 
 138 ## Text Interpolations
 139 
 140 Inside double quoted text, you can use a dollar sign (`$`) to insert an
 141 expression that you want converted to text. This is called text interpolation:
 142 
 143 ```
 144 // Interpolation:
 145 my_var := 5
 146 str := "My var is $my_var!"
 147 // Equivalent to "My var is 5!"
 148 
 149 // Using parentheses:
 150 str := "Sum: $(1 + 2)"
 151 // equivalent to "Sum: 3"
 152 ```
 153 
 154 Single-quoted text does not have interpolations:
 155 
 156 ```
 157 // No interpolation here:
 158 str := 'Sum: $(1 + 2)'
 159 ```
 160 
 161 ### Text Escapes
 162 
 163 Like other languages, backslash is a special character inside of text for
 164 escape sequences like `\n`. However, in general it is best practice to use
 165 multi-line text if you need to add a newline.
 166 
 167 ```
 168 str := "
 169     This has
 170     multiple lines and "quotes" too!
 171 "
 172 ```
 173 
 174 ### Customizable `$`-Text
 175 
 176 Sometimes you might need to use a lot of literal `$`s or quotation marks in
 177 text. In such cases, you can use the more customizable form of text. The
 178 customizable form lets you explicitly specify which character to use for
 179 interpolation and which characters to use for delimiting the text.
 180 
 181 The first character after the `$` is the custom interpolation character, which
 182 can be any of the following symbols: `~!@#$%^&*+=\?`. If none of these
 183 characters is used, the default interpolation character is `$`. Since this is
 184 the default, you can disable interpolation altogether by using `$` here (i.e. a
 185 double `$$`).
 186 
 187 The next thing in a customizable text is the character used to delimit the
 188 text. The text delimiter can be any of the following symbols: `` "'`|/;([{< ``
 189 If the text delimiter is one of `([{<`, then the text will continue until a
 190 matching `)]}>` is found, not terminating unless the delimiters are balanced
 191 (i.e. nested pairs of delimiters are considered part of the text).
 192 
 193 Here are some examples:
 194 
 195 ```
 196 $"Equivalent to normal text with dollar interps: $(1 + 2)"
 197 $@"The same, but the AT symbol interpolates: @(1 + 2)"
 198 $$"No interpolation here, $ is just a literal character"
 199 $|This text is pipe-delimited, so it can have "quotes" and 'single quotes' and interpolates with dollar sign: $(1+2)|
 200 $(This text is parens-delimited, so you can have (nested) parens without ending the text)
 201 $=[This text is square-bracket delimited [which can be nested] and uses equals for interps: =(1 + 2)]
 202 $@/look ma, regex literals!/
 203 ```
 204 
 205 When text is delimited by matching pairs (`()`, `[]`, `{}`, or `<>`), they
 206 can only be closed by a matched closing character at the same indentation
 207 level, ignoring nested pairs:
 208 
 209 ```
 210 $$(Inside parens, you can have (nested ()) parens no problem)
 211 $$"But only (), [], {}, and <> are matching pairs, you can't have nested quotes"
 212 $$(
 213     When indented, an unmatched ) won't close the text
 214     An unmatched ( won't mess things up either
 215     Only matching pairs on the same indentation level are counted:
 216 )
 217 $$(Multi-line text with nested (parens) and
 218 .. line continuation)
 219 ```
 220 
 221 As a special case, when you use the same character for interpolation and text
 222 delimiting, no interpolations are allowed:
 223 
 224 ```
 225 plain := $""This text has {no interpolations}!"
 226 ```
 227 
 228 **Note:** Normal doubly quoted text with no dollar sign (e.g. `"foo"`) are a
 229 shorthand for `${}"foo"`. Singly quoted text with no dollar sign (e.g.
 230 `'foo'`) are shorthand for `$''foo'`.
 231 
 232 ## Operations
 233 
 234 ### Concatenation
 235 
 236 Concatenation in the typical case is a fast operation: `"{x}{y}"` or `x ++ y`.
 237 
 238 Because text concatenation is typically fast, there is no need for a separate
 239 "string builder" class in the language and no need to use a list of text
 240 fragments.
 241 
 242 ### Text Length
 243 
 244 Text length gives you the number of grapheme clusters in the text, according to
 245 the unicode standard. This corresponds to what you would intuitively think of
 246 when you think of "letters" in a string. If you have text with an emoji that has
 247 several joining modifiers attached to it, that text has a length of 1.
 248 
 249 ```tomo
 250 assert "hello".length == 5
 251 assert "👩🏽‍🚀".length == 1
 252 ```
 253 
 254 ### Iteration
 255 
 256 Iteration is *not* supported for text. It is rarely ever the case that you will
 257 need to iterate over text, but if you do, you can iterate over the length of
 258 the text and retrieve 1-wide slices. Alternatively, you can split the text into
 259 its constituent grapheme clusters with `text.split()` and iterate over those.
 260 
 261 ### Equality, Comparison, and Hashing
 262 
 263 All text is compared and hashed using unicode normalization. Unicode provides
 264 several different ways to represent the same text. For example, the single
 265 codepoint `U+E9` (latin small e with accent) is rendered the same as the two
 266 code points `U+65 U+301` (latin small e, acute combining accent) and has an
 267 equivalent linguistic meaning. These are simply different ways to represent the
 268 same "letter." In order to make it easy to write correct code that takes this
 269 into account, Tomo uses unicode normalization for all text comparisons and
 270 hashing. Normalization does the equivalent of converting text to a canonical
 271 form before performing comparisons or hashing. This means that if a table is
 272 created that has text with the codepoint `U+E9` as a key, then a lookup with
 273 the same text but with `U+65 U+301` instead of `U+E9` will still succeed in
 274 finding the value because the two texts are equivalent under normalization.
 275 
 276 # API
 277 
 278 [API documentation](../api/text.md)