Pkg.jl: Julia package management for fun and profit

Basics

What happens when you type using MixedModels? How does Julia know what bit of code to run? Where does it look to find this code? Which version is it using? The answers to these questions are the province of the “package manager”, Pkg.jl. As always, the official documentation is the best source of infromation. But in the interest of expediency, here is a cheat sheet.

Projects in Julia

A project in Julia is like a packrat project (R) or virtualenv (python).

It is a list of the packages that are used for running some bit of julia code.

Any directory can be a project, as long as it has a Project.toml file in it.

Project.toml

Lists the dependencies for a project.

A basic Project.toml has one section, that lists the dependencies like this:

[deps]
BenchmarkTools = "6e4b80f9-dd63-53aa-95a3-0cdb28fa8baf"
CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
DataFrames = "a93c6f00-e57d-5684-b7b6-d8193f3e46c0"
GLM = "38e38edf-8417-5370-95a0-9cbb8c7f171a"

This is how Julia knows what you mean when you say using CSV. CSV is just a convenient, human-readable shorthand for the “real” name of the package, which is the UUID (Universally Unique ID), 336ed68f-0bac-5ca0-87d4-7b16caf5d00b.

You rarely if ever need to edit Project.toml by hand. It is automatically generated by the package manager the vast majority of the time, using commands in Pkg mode.

Pkg mode

To enter package mode, press ]. The julia> prompt should change to a pkg prompt, like (v1.3) pkg>.

The bit in the parentheses gives the name of the current project environment.

The default environment is the global environment for the current Julia version (in this case, 1.3).

Creating and activating a project environment

But any directory can be a project. To activate a project in a directory, you do:

julia> # press ] to enter Pkg mode...

(v1.3) pkg> activate path/to/your/project/directory
Activating new environment at `~/path/to/your/project/directory/Project.toml`

(directory) pkg>

Noticed that the Pkg mode prompt now shows that you’re in the new project (it’s named after the directory. Now, whenever you do an Pkg operations, they apply to this project.

Adding required packages to your project

You need to add every package that you directly import with using or import in your scripts/jmd files:

(directory) pkg> add StatsBase
 Resolving package versions...
  Updating `~/path/to/your/project/directory/Project.toml`
  [2913bbd2] + StatsBase v0.32.1
  Updating `~/path/to/your/project/directory/Manifest.toml`
  [9a962f9c] + DataAPI v1.1.0
  [864edb3b] + DataStructures v0.17.9
  [e1d29d7a] + Missings v0.4.3
  [bac558e1] + OrderedCollections v1.1.0
  [a2af1166] + SortingAlgorithms v0.3.1
  [2913bbd2] + StatsBase v0.32.1
  [2a0f44e3] + Base64
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [4ec0a83e] + Unicode

Two things have happened: first, the new Project.toml lists StatsBase as a dependency:

[deps]
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"

Second, a Manifest.toml file has been created, which lists the exact versions of all packages that have been installed to satisfy the requirements of this project.

# This file is machine-generated - editing it directly is not advised

[[Base64]]
uuid = "2a0f44e3-6c83-55bd-87e4-b1978d98bd5f"

[[DataAPI]]
git-tree-sha1 = "674b67f344687a88310213ddfa8a2b3c76cc4252"
uuid = "9a962f9c-6df0-11e9-0e5d-c546b8b5ee8a"
version = "1.1.0"

[[DataStructures]]
deps = ["InteractiveUtils", "OrderedCollections"]
git-tree-sha1 = "b7720de347734f4716d1815b00ce5664ed6bbfd4"
uuid = "864edb3b-99cc-5e75-8d2d-829cb0a9cfe8"
version = "0.17.9"

(many more entries)...

Because each project has its own, independent manifest, each project can have its own version of the packages it needs. This is really useful for stability, and for reproducibility.

instantiate to reproduce

If you have downloaded a directory with a Project.toml created by someone else (including “past you” or “other computer you”), you can instantiate that environment via:

julia> # press ] to enter Pkg mode...

(v1.3) pkg> activate path/to/your/project/directory/
Activating environment at `~/path/to/your/project/directory/Project.toml`

(directory) pkg> instantiate
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `git@github.com:JuliaRegistries/General.git`
 Resolving package versions...
  Updating `~/code/RePsychLing/path/to/your/project/directory/Project.toml`
  [2913bbd2] + StatsBase v0.32.1
  Updating `~/code/RePsychLing/path/to/your/project/directory/Manifest.toml`
  [9a962f9c] + DataAPI v1.1.0
  [864edb3b] + DataStructures v0.17.9
  [e1d29d7a] + Missings v0.4.3
  [bac558e1] + OrderedCollections v1.1.0
  [a2af1166] + SortingAlgorithms v0.3.1
  [2913bbd2] + StatsBase v0.32.1
  [2a0f44e3] + Base64
  [8ba89e20] + Distributed
  [b77e0a4c] + InteractiveUtils
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [de0858da] + Printf
  [9a3f8284] + Random
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [8dfed614] + Test
  [4ec0a83e] + Unicode

This effectively adds every package in the Project.toml. (It also does some fancy version resolution to find a set of versions that are all mutually compatible).

If you also have a Manifest.toml for this project, then instantiate will install the exact versions of all the packages listed in the Manfiest. This means that you have a perfectly portable version of your project.

Other Pkg operations

• rm to remove packages
• up to update all packges to latest versions
• up PkgName to update PkgName to the latest (compatible) version.
• resolve to try to find a set of compatible versions

adding specific versions of packages

• add PkgName@a.b.c add version a.b.c of package (or a compatible version).
• add PkgName#master add the master branch (usually pre-release) version of a package.
• add https://www.github.com/user/somepackage.jl add an un-released package from a git URL.

NOTE: version information is automatically recorded in Manifest.toml, but not in Project.toml.

compatibility in Project.toml

You can add compatibility restrictions in Project.toml but this is generally not necessary for one-off projects (like analysis scripts etc.) which are not intended to wide re-use (unlike a “real” package):

[deps]
StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"

[compat]
StatsBase = "0.32"

The rules of which exact versions are compatible are somewhat complex, and are documented here. The resolver obeys “semantic versioning” or “semver”, if that means anything to you, with the added rule that pre-1.0, a minor version counts as a breaking (“major”) release, so by this rule 1.0, 0.32, 0.31 are all considered incompatible.