16 KiB
Text
Text is Tomo's datatype to represent text. The name Text is used instead of
"string" because Tomo represents text as an immutable UTF-8-encoded value that
uses the Boehm Cord library for efficient storage and concatenation. These are
not C-style NULL-terminated character arrays. GNU libunistring is used for
full Unicode functionality (grapheme cluster counts, capitalization, etc.).
Syntax
Text has a flexible syntax designed to make it easy to hold values from different languages without the need to have lots of escape sequences and without using printf-style string formatting.
// Basic text:
str := "Hello world"
str2 := 'Also text'
str3 := `Backticks too`
Line Splits
Long text can be split across multiple lines by having two or more dots at the start of a new line on the same indentation level that started the text:
str := "This is a long
....... line that is split in code"
Multi-line Text
Multi-line text has indented (i.e. at least one tab more than the start of the text) text inside quotation marks. The leading and trailing newline are ignored:
multi_line := "
This text has multiple lines.
Line two.
You can split a line
.... using two or more dots to make an elipsis.
Remember to include whitespace after the elipsis if desired.
Or don't if you're splitting a long word like supercalifragilisticexpia
....lidocious
This text is indented by one level in the text
"quotes" are ignored unless they're at the same indentation level as the
.... start of the text.
The end (no newline after this).
"
Text Interpolations
Inside double quoted text, you can use a dollar sign ($) to insert an
expression that you want converted to text. This is called text interpolation:
// Interpolation:
my_var := 5
str := "My var is $my_var!"
// Equivalent to "My var is 5!"
// Using parentheses:
str := "Sum: $(1 + 2)"
// equivalent to "Sum: 3"
Single-quoted text does not have interpolations:
// No interpolation here:
str := 'Sum: $(1 + 2)'
Text Escapes
Unlike other languages, backslash is not a special character inside of text.
For example, "x\ny" has the characters x, \, n, y, not a newline.
Instead, a series of character escapes act as complete text literals without
quotation marks:
newline := \n
crlf := \r\n
quote := \"
These text literals can be used as interpolation values with or without parentheses, depending on which you find more readable:
two_lines := "one$(\n)two"
has_quotes := "some $\"quotes$\" here"
However, in general it is best practice to use multi-line text to avoid these problems:
str := "
This has
multiple lines and "quotes" too!
"
Multi-line Text
There are two reasons for text to span multiple lines in code: either you
have text that contains newlines and you want to represent it without \n
escapes, or you have a long single-line text that you want to split across
multiple lines for readability. To support this, you can use newlines inside of
text with indentation-sensitivity. For splitting long lines, use two or more
"."s at the same indentation level as the start of the text literal:
single_line := "This is a long text that
... spans multiple lines"
For text that contains newlines, you may put multiple indented lines inside the quotes:
multi_line := "
line one
line two
this line is indented
last line
"
Text may only end on lines with the same indentation as the starting quote and nested quotes are ignored:
multi_line := "
Quotes in indented regions like this: " don't count
"
If there is a leading or trailing newline, it is ignored and not included in the text.
str := "
one line
"
>>> str == "one line"
=== yes
Additional newlines are counted though:
str := "
blank lines
"
>>> str == "{\n}blank lines{\n}"
Customizable $-Text
Sometimes you might need to use a lot of literal $s or quotation marks in
text. In such cases, you can use the more customizable form of text. The
customizable form lets you explicitly specify which character to use for
interpolation and which characters to use for delimiting the text.
The first character after the $ is the custom interpolation character, which
can be any of the following symbols: ~!@#$%^&*+=\?. If none of these
characters is used, the default interpolation character is $. Since this is
the default, you can disable interpolation altogether by using $ here (i.e. a
double $$).
The next thing in a customizable text is the character used to delimit the
text. The text delimiter can be any of the following symbols: "'`|/;([{<
If the text delimiter is one of ([{<, then the text will continue until a
matching )]}> is found, not terminating unless the delimiters are balanced
(i.e. nested pairs of delimiters are considered part of the text).
Here are some examples:
$"Equivalent to normal text with dollar interps: $(1 + 2)"
$@"The same, but the AT symbol interpolates: @(1 + 2)"
$$"No interpolation here, $ is just a literal character"
$|This text is pipe-delimited, so it can have "quotes" and 'single quotes' and interpolates with dollar sign: $(1+2)|
$(This text is parens-delimited, so you can have (nested) parens without ending the text)
$=[This text is square-bracket delimited [which can be nested] and uses equals for interps: =(1 + 2)]
$@/look ma, regex literals!/
When text is delimited by matching pairs ((), [], {}, or <>), they
can only be closed by a matched closing character at the same indentation
level, ignoring nested pairs:
$$(Inside parens, you can have (nested ()) parens no problem)
$$"But only (), [], {}, and <> are matching pairs, you can't have nested quotes"
$$(
When indented, an unmatched ) won't close the text
An unmatched ( won't mess things up either
Only matching pairs on the same indentation level are counted:
)
$$(Multi-line text with nested (parens) and
.. line continuation)
As a special case, when you use the same character for interpolation and text delimiting, no interpolations are allowed:
plain := $""This text has {no interpolations}!"
Note: Normal doubly quoted text with no dollar sign (e.g. "foo") are a
shorthand for ${}"foo". Singly quoted text with no dollar sign (e.g.
'foo') are shorthand for $''foo'.
Operations
Concatenation
Concatenation in the typical case is an O(1) operation: "{x}{y}" or x ++ y.
Because text concatenation is typically an O(1) operation, there is no need for a separate "string builder" class in the language and no need to use an array of text fragments.
Text Length
Text length is an ambiguous term in the context of UTF-8 text. There are several possible meanings, so each of these meanings is split into a separate method:
- Number of grapheme clusters:
text:num_clusters(). This is probably what you want to use, since it corresponds to the everyday notion of "letters". - Size in bytes:
text:num_bytes() - Number of unicode codepoints:
text:num_codepoints()(you probably want to use clusters, not codepoints in most applications)
Since the typical user expectation is that text length refers to "letters,"
the # length operator returns the number of grapheme clusters, which is the
closest unicode equivalent to "letters."
Iteration
Iteration is not supported for text because of the ambiguity between bytes, codepoints, and grapheme clusters. It is instead recommended that you explicitly iterate over bytes, codepoints, graphemes, words, lines, etc:
Equality, Comparison, and Hashing
All text is compared and hashed using unicode normalization. Unicode provides
several different ways to represent the same text. For example, the single
codepoint U+E9 (latin small e with accent) is rendered the same as the two
code points U+65 U+301 (latin small e, acute combining accent) and has an
equivalent linguistic meaning. These are simply different ways to represent the
same "letter." In order to make it easy to write correct code that takes this
into account, Tomo uses unicode normalization for all text comparisons and
hashing. Normalization does the equivalent of converting text to a canonical
form before performing comparisons or hashing. This means that if a table is
created that has text with the codepoint U+E9 as a key, then a lookup with
the same text but with U+65 U+301 instead of U+E9 will still succeed in
finding the value because the two texts are equivalent under normalization.
Text Functions
as_c_string
Description:
Converts a Text value to a C-style string.
Usage:
as_c_string(text: Text) -> CString
Parameters:
text: The text to be converted to a C-style string.
Returns:
A C-style string (CString) representing the text.
Example:
>> "Hello":as_c_string()
= CString("Hello")
bytes
Description:
Converts a Text value to an array of bytes.
Usage:
bytes(text: Text) -> [Int8]
Parameters:
text: The text to be converted to bytes.
Returns:
An array of bytes ([Int8]) representing the text.
Example:
>> "Amélie":bytes()
= [65_i8, 109_i8, 101_i8, -52_i8, -127_i8, 108_i8, 105_i8, 101_i8]
character_names
Description:
Returns a list of character names from the text.
Usage:
character_names(text: Text) -> [Text]
Parameters:
text: The text from which to extract character names.
Returns:
A list of character names ([Text]).
Example:
>> "Amélie":character_names()
= ["LATIN CAPITAL LETTER A", "LATIN SMALL LETTER M", "LATIN SMALL LETTER E", "COMBINING ACUTE ACCENT", "LATIN SMALL LETTER L", "LATIN SMALL LETTER I", "LATIN SMALL LETTER E"]
clusters
Description:
Breaks the text into a list of unicode graphical clusters. Clusters are what
you typically think of when you think of "letters" or "characters". If you're
in a text editor and you hit the left or right arrow key, it will move the
cursor by one graphical cluster.
Usage:
clusters(text: Text) -> [Text]
Parameters:
text: The text to be broken into graphical clusters.
Returns:
A list of graphical clusters ([Text]) within the text.
Example:
>> "Amélie":clusters()
= ["A", "m", "é", "l", "i", "e"] : [Text]
codepoints
Description:
Returns a list of Unicode code points for the text.
Usage:
codepoints(text: Text) -> [Int32]
Parameters:
text: The text from which to extract Unicode code points.
Returns:
A list of Unicode code points ([Int32]).
Example:
>> "Amélie":codepoints()
= [65_i32, 109_i32, 101_i32, 769_i32, 108_i32, 105_i32, 101_i32] : [Int32]
from_c_string
Description:
Converts a C-style string to a Text value.
Usage:
from_c_string(str: CString) -> Text
Parameters:
str: The C-style string to be converted.
Returns:
A Text value representing the C-style string.
Example:
from_c_string(CString("Hello")) // "Hello"
has
Description:
Checks if the Text contains a target substring.
Usage:
has(text: Text, target: Text, where: Where = Where.Anywhere) -> Bool
Parameters:
text: The text to be searched.target: The substring to search for.where: The location to search (Where.Anywhereby default).
Returns:
yes if the target substring is found, no otherwise.
Example:
has("Hello, world!", "world") // yes
join
Description:
Joins a list of text pieces with a specified glue.
Usage:
join(glue: Text, pieces: [Text]) -> Text
Parameters:
glue: The text used to join the pieces.pieces: The list of text pieces to be joined.
Returns:
A single Text value with the pieces joined by the glue.
Example:
join(", ", ["apple", "banana", "cherry"]) // "apple, banana, cherry"
lower
Description:
Converts all characters in the text to lowercase.
Usage:
lower(text: Text) -> Text
Parameters:
text: The text to be converted to lowercase.
Returns:
The lowercase version of the text.
Example:
lower("HELLO") // "hello"
num_bytes
Description:
Returns the number of bytes used by the text.
Usage:
num_bytes(text: Text) -> Int
Parameters:
text: The text to measure.
Returns:
The number of bytes used by the text.
Example:
num_bytes("Hello") // 5
num_clusters
Description:
Returns the number of clusters in the text.
Usage:
num_clusters(text: Text) -> Int
Parameters:
text: The text to measure.
Returns:
The number of clusters in the text.
Example:
num_clusters("Hello") // 5
num_codepoints
Description:
Returns the number of Unicode code points in the text.
Usage:
num_codepoints(text: Text) -> Int
Parameters:
text: The text to measure.
Returns:
The number of Unicode code points in the text.
Example:
num_codepoints("Hello") // 5
quoted
Description:
Formats the text as a quoted string.
Usage:
quoted(text: Text, color: Bool = no) -> Text
Parameters:
text: The text to be quoted.color: Whether to add color formatting (default isno).
Returns:
The text formatted as a quoted string.
Example:
quoted("Hello") // "\"Hello\""
replace
Description:
Replaces occurrences of a pattern in the text with a replacement string.
Usage:
replace(text: Text, pattern: Text, replacement: Text, limit: Int = -1) -> Text
Parameters:
text: The text in which to perform replacements.pattern: The substring to be replaced.replacement: The text to replace the pattern with.limit: The maximum number of replacements (default is-1, meaning no limit).
Returns:
The text with occurrences of the pattern replaced.
Example:
replace("Hello world", "world", "there") // "Hello there"
split
Description:
Splits the text into a list of substrings based on a delimiter.
Usage:
split(text: Text, split: Text) -> [Text]
Parameters:
text: The text to be split.split: The delimiter used to split the text.
Returns:
A list of substrings resulting from the split.
Example:
split("apple,banana,cherry", ",") // ["apple", "banana", "cherry"]
title
Description:
Converts the text to title case (capitalizing the first letter of each word).
Usage:
title(text: Text) -> Text
Parameters:
text: The text to be converted to title case.
Returns:
The text in title case.
Example:
title("hello world") // "Hello World"
trimmed
Description:
Trims characters from the beginning and end of the text.
Usage:
trimmed(text: Text, trim: Text = " {\n\r\t}", where: Where = Where.Anywhere) -> Text
Parameters:
text: The text to be trimmed.trim: The set of characters to remove (default is" {\n\r\t}").where: Specifies where to trim (Where.Anywhereby default).
Returns:
The trimmed text.
Example:
trimmed(" Hello ") // "Hello"
upper
Description:
Converts all characters in the text to uppercase.
Usage:
upper(text: Text) -> Text
Parameters:
text: The text to be converted to uppercase.
Returns:
The uppercase version of the text.
Example:
upper("hello") // "HELLO"
without
Description:
Removes all occurrences of a target substring from the text.
Usage:
without(text: Text, target: Text, where: Where = Where.Anywhere) -> Text
Parameters:
text: The text from which to remove substrings.target: The substring to remove.where: The location to remove the target (Where.Anywhereby default).
Returns:
The text with occurrences of the target removed.
Example:
without("Hello world", "world") // "Hello "