1 # Tomo Package Design
   2 
   3 In Tomo, the `use` statement serves multiple purposes. The first purpose is for
   4 [local imports](local-imports.md). However, the more complicated case is for
   5 package imports. Packages in Tomo are a bundle of source files that define
   6 their own namespace and symbols that can be used by other code.
   7 
   8 When you type
   9 
  10 ```tomo
  11 use mypackage
  12 ```
  13 
  14 Tomo will try to figure out where to get the package, download it from there
  15 (with user confirmation), check its authenticity, build the package, and link
  16 your code against it.
  17 
  18 ## Using Packages
  19 
  20 In a particular project, there may be many files that want to use a package.
  21 Each file that uses the package must include `use packagename` or `alias := use
  22 packagename` at the top level of the file.
  23 
  24 If a package is used without an alias, all the package's symbols are visible
  25 within the file. This is the equivalent of `from packagename import *` in a
  26 language like Python. If an alias is used, then all the package's symbols must
  27 be accessed through the alias, like `alias.foo()`.
  28 
  29 ## Installing Packages
  30 
  31 When you put `use foo` in a Tomo source file, Tomo needs to know where to
  32 find `foo`. In order to find it, Tomo will search through the following `.ini`
  33 files in order:
  34 
  35 1. `filename.tm.packages.ini`: A per-file package config, so `foo.tm` would
  36    have `foo.tm.packages.ini`. This should only rarely be used.
  37 2. `packages.ini`: A package config that is shared for every file in a
  38    directory.
  39 3. A system-wide `packages.ini` distributed with the Tomo compiler (default
  40    location: `~/.local/lib/tomo@version/packages.ini`).
  41 
  42 The format of a `packages.ini` file looks like this:
  43 
  44 ```
  45 [packagename]
  46 digest=sha256:2551b5ecc2617d884d856764a7f4f5ba394e97aaf1412110fe63f137b510f546
  47 source=https://example.com/tomo/package-v1.2.3.tar.gz
  48 source-2=https://mirror.example.com/tomo/package-v1.2.3.tar.gz
  49 source-3=https://mirror2.example.com/tomo/package-v1.2.3.tar.gz
  50 ```
  51 
  52 The important features are:
  53 
  54 1. The package name in square brackets. This is used to map `use` statements to
  55    the corresponding package.
  56 2. A hash digest (optional). This is a hash value used to check package
  57    integrity when installing a package. See [Package Digests](#Package-Digests)
  58    for more info.
  59 3. One or more sources. See [Package Sources](#Package-Sources) for more info.
  60 
  61 Each key/value pair is separated by an equals sign. Additional fields may be
  62 provided, and are ignored.
  63 
  64 ## Package Digests
  65 
  66 A package is canonically defined by its digest. The digest is a cryptographic
  67 hash digest of the archived source file for the package. The default digest
  68 algorithm used is SHA-256. When you put a digest in your `packages.ini` file,
  69 you define **precisely** what code you are depending on. Tomo does not have a
  70 notion of package versioning or package registries other than the digest.
  71 However, it is easy to play nicely with whichever versioning and distribution
  72 system you prefer to use.
  73 
  74 You do not need to manually compute your own digests. When adding a project
  75 dependency, you may list a package with just a source and no digest. Running
  76 the project with `tomo` will cause Tomo to download the package and save its
  77 digest to `packages.ini` so you can be sure future users will get the same
  78 version. You can think of this as a combined package file and lockfile. See
  79 [Package Sources](#Package-Sources) for more information.
  80 
  81 The philosophy of package digests is rooted in reproducibility and security.
  82 Because Tomo does not have a centrally managed package repository and does not
  83 plan to build one, we need a way for someone to publish a piece of Tomo code
  84 that relies on a package and have anyone be able to build that code. The code
  85 should build as consistently as possible over time, with no breaking changes or
  86 issues with local version incompatibilities. If the author of a piece of code
  87 knows that the code works with the package whose source code hash is `X`, then
  88 all users can be confident that if they have a package whose source code has is
  89 also `X`, then it can be used without issues, regardless of where they got it.
  90 
  91 
  92 ## Package Sources
  93 
  94 When you run a piece of Tomo code for the first time, Tomo will look to see if
  95 you have the necessary packages installed. To do so, Tomo will look for
  96 `~/.local/lib/tomo@<tomo-version>/<package-digest>`. If not, then Tomo will
  97 look for sources listed in your `packages.ini` files. Sources are enumerated
  98 as `source`, `source-2`, `source-3`, and so on. Each will be checked in order
  99 to find a valid version of the package.
 100 
 101 For each source, Tomo will perform the following steps:
 102 
 103 1. A source archive is downloaded from the given source:
 104   a. If the source is a file path (e.g. `./mypackage.tar.gz`), then the archive
 105      from the local filesystem is used.
 106   b. If the source is a URI (e.g. `https://example.com/mypackage-v1.2.3.tar.gz`
 107      or `ftp://example.com/...`), then [cURL](https://curl.se/) is used to
 108      download the archive file to a temporary directory. Tomo will prompt for
 109      user confirmation before downloading the file.
 110 2. If a digest is provided in your `packages.ini`, then the archive file's
 111    digest is computed and compared to the digest in `packages.ini`. If there is
 112    any mismatch, an error will be raised and the process will exit. If you
 113    experience a digest mismatch, you should consider removing the source that
 114    provided it from your sources list, as it may be compromised.
 115 3. If a digest is not provided in your `packages.ini`, Tomo will compute a
 116    digest for the newly downloaded file and save it to your `packages.ini` file
 117    so that all future compilations will know what the digest must be.
 118 4. The source archive will be extracted to `~/.local/lib/tomo@<tomo-version>/<package-digest>`
 119    and compiled with `tomo -L`.
 120 
 121 Package sources are not tied to any single distribution channel by design. You
 122 can host your packages on GitHub, BitBucket, GitLab, your own personal
 123 webserver, a cloud bucket, an FTP server in your closet, a vendored archive in
 124 your repository, a floppy disk, whatever you like. For your end users, the
 125 experience should be seamless, without being tied to a single point of failure
 126 or tech monopoly. If one of your sources becomes unavailable, Tomo will
 127 continue down the line, trying each source until it gets one that can provide
 128 a source archive with the right hash.
 129 
 130 ### Local Directory Packages
 131 
 132 As a special case, Tomo also permits using local directories as sources. If a
 133 source is listed as a file path to a local directory (instead of an archive
 134 file), Tomo will use the code in that directory for the package without saving
 135 a digest or installing the package in `~/.local/lib/tomo@<tomo-version>`. This
 136 makes it possible to easily vendor a dependency (e.g. with git submodules).
 137 
 138 ```
 139 [mypackage]
 140 source = ./vendor/mypackage
 141 ```
 142 
 143 If a digest is provided for a local directory package, Tomo will give you an
 144 error, because digests can only be computed for files, not directories. When
 145 compiling a package, Tomo produces build files which would affect the hash
 146 digest of the directory, making it difficult to compute a correct hash.
 147 Additionally, it is much better as a developer to be able to make local edits
 148 to a vendored package without having to worry about digest mismatches.
 149 
 150 ## Package Versioning
 151 
 152 Tomo's package design is deliberately version-agnostic. If you want to use a
 153 versioning system for distributing a package (a very reasonable goal), then
 154 the recommended practice is to use the source URL to convey versioning:
 155 
 156 ```
 157 [mylib]
 158 source = https://example.com/mypackage/mypackage-v1.2.3.tar.gz
 159 source-2 = https://mirror.example.com/mypackage/mypackage-v1.2.3.tar.gz
 160 ```
 161 
 162 To upgrade a dependency, simply delete the `digest` from your `packages.ini`,
 163 update the source URL, and rebuild. Tomo will update the digest for you. In
 164 this way, you may use any versioning system you like, not limited to semantic
 165 versioning.
 166 
 167 When you upgrade a dependency and rebuild, Tomo will pull in the upgraded
 168 version of the dependency's source code, including its `packages.ini` files,
 169 which will trigger a chain reaction of downloading the correct versions of its
 170 dependencies, and so on.
 171 
 172 ## Package Coresidence
 173 
 174 Because pacakges are installed to unique locations based on their source code
 175 hashes, it is possible for your project to install different versions of the
 176 same package through transitive dependencies. For example, if you use package
 177 `foo` and package `baz` and `foo` relies on `commonlib-v1.2` while package
 178 `baz` relies on `commonlib-v1.3`, then Tomo will dutifully install and link
 179 both versions of `commonlib`, without any problems. Left unchecked, this can
 180 result in a bloated binary with multiple versions of many libraries. If this is
 181 a problem that you are experiencing, you may wish to consider synchronizing
 182 your dependencies so they depend on the same common library (e.g. upgrading
 183 `foo` in this example) or considering pruning your dependency tree to reduce
 184 the number of transitive dependencies in your project.
 185 
 186 ## Creating a Package Archive
 187 
 188 A Tomo package source archive may be any of the following formats:
 189 
 190 - `.tar.gz` or `.tgz`
 191 - `.tar.xz` or `.txz`
 192 - `.tar`
 193 - `.zip`
 194 
 195 The contents of the archive should be either the files in your package, or a
 196 single folder containing the files in your package. Tomo will handle both.
 197 
 198 Tomo package archives can be created any way you like, but here are some easy
 199 options:
 200 
 201 - If you are using git, then `git archive --format=tar.gz -o mypackage.tar.gz`
 202 - If you are using GitHub, then when you create a tag and push to GitHub, a
 203   source archive will be created automatically at
 204   `https://github.com/<org>/<repo>/archive/refs/tags/<tagname>.tar.gz`
 205 - Run `tar -czf mypackage.tar.gz --exclude='*/.*' /path/to/project`