Fetch Dependencies

The bootstrap problem. This project's specs live in Artifactory. The Artifactory CLI is itself an envy package, so it has to be installed before any of those specs can be fetched. Ordinary dependencies cannot express that. They order phases of packages whose specs envy already has. Fetch dependencies order the acquisition of the spec itself.

The shape

A source table carries both the tools it needs and the custom fetch that uses them. Here it is on a manifest BUNDLES declaration, fetching a bundle of specs out of Artifactory with a jf that envy installed:

envy.lua

BUNDLES = {
  corp = {
    identity = "corp.specs@r1",
    source = {
      -- Installed end to end, including setup, before the bundle is fetched.
      dependencies = {
        { spec = "tools.jfrog-cli@r1", product = "jf",
          source = envy.abspath("envy/tools.jfrog-cli.lua") },
      },

      fetch = function(tmp_dir)
        envy.run(envy.template([[
          {{jf}} config add ci --url={{url}} --access-token=$JF_TOKEN --interactive=false
          {{jf}} rt download --flat --fail-no-op '{{repo}}/' '{{dest}}'
        ]], {
          jf = envy.product("jf"),  -- provided by tools.jfrog-cli
          url = "https://acme.jfrog.io/artifactory",
          repo = "envy-specs/corp.specs@r1",
          dest = tmp_dir .. "/",
        }))

        envy.commit_fetch({ "envy-bundle.lua", "specs" })
      end,
    },
  },
}

PACKAGES = {
  { spec = "corp.toolchain@r2", bundle = "corp", options = { version = "15.2" } },
}

The fetch dependency declares product = "jf", so the fetch function resolves the tool with envy.product("jf") and gets an absolute path into the cache. Nothing assumes a jf on PATH, and nothing hardcodes a cache path. Wrapping the command in envy.template keeps the two jf invocations and the three varying values in one place. envy.package(identity) is available the same way when you need the dependency's whole directory rather than one product.

fetch writes into tmp_dir and calls envy.commit_fetch, which verifies any hashes and moves the files into the durable fetch directory. A bundle commits its envy-bundle.lua and spec files. A single spec commits spec.lua, plus any helper files beside it: everything committed lands in the spec directory, so envy.loadenv("helper") from inside the fetched spec finds it.

Where it can be declared

A BUNDLES declaration, in a manifest or a spec, and a spec DEPENDENCIES entry. A manifest PACKAGES entry cannot carry source.fetch, and since envy 0.3.1 that is a parse error rather than a failure part way through the run:

error: Package 'source' cannot be a { fetch = ... } table: nothing can ever call
       that function from here. Declare the spec in another spec's DEPENDENCIES,
       or the bundle in a BUNDLES table.

A spec-declared source.fetch receives (tmp_dir, options). A bundle's fetch receives (tmp_dir) only.

Everything a spec-declared fetch can see belongs to the dependency it is fetching, not to the spec whose file holds the closure:

• options are the entry's own. { spec = "corp.sdk@r1", options = { region = "eu" }, source = { fetch = ... } } hands the function { region = "eu" } whatever the declaring spec was built with.
• envy.product and envy.package resolve the entry's own source.dependencies, so a tool the entry declares is reachable and one the declaring spec declares is not.
• Each option set gets its own spec cache entry. A closure has no fingerprint to hash, so the options are what keep two variants apart.

Before envy 0.3.1 the function ran in the declaring spec's context, which handed it the wrong options table and hid its own declared prerequisites.

The ordinary case, for comparison

Once a spec is loaded, its own verbs resolve tools the same way, and needed_by decides how early that is legal:

corp.toolchain@r2.lua

local hashes -- version -> sha256, at the bottom of this file

DEPENDENCIES = {
  { spec = "tools.jfrog-cli@r1", product = "jf", source = "tools.jfrog-cli.lua",
    needed_by = "fetch" },
}

FETCH = function(tmp_dir, opts)
  envy.run(envy.template([[
    {{jf}} rt download --flat --fail-no-op 'toolchains/{{version}}.tar.zst' '{{dest}}'
  ]], {
    jf = envy.product("jf"),  -- provided by tools.jfrog-cli
    version = opts.version,
    dest = envy.path.join(tmp_dir, "toolchain.tar.zst"),
  }))

  envy.commit_fetch({ filename = "toolchain.tar.zst", sha256 = hashes[opts.version] })
end

needed_by = "fetch" is what legalizes the call in FETCH. The default, build, produces envy.product: product 'jf' needed_by 'build' but accessed during 'fetch'. A fetch dependency needs no needed_by, because it is already gated on the earliest phase there is.

On Windows

The bootstrapped tool is an executable in a cache directory, so nothing about the mechanism changes. Two things about the fetch function do:

• Resolve the tool with envy.product, then build the command line with envy.path.join. The resolved path contains backslashes, and a hand-built "/" path will not survive being handed to cmd.
• A script string inside the fetch function runs under the platform default, so PowerShell on Windows. A one-line CLI invocation is usually identical in both dialects, but redirection and quoting are not. Prefer passing the command to envy.run as a single line and letting the tool write the file itself.

Rules

• The guarantee. Every entry in source.dependencies goes through its entire lifecycle, install and setup included, before the dependent's spec is fetched. Compare needed_by = "fetch", which gates payload fetching only.

• source.dependencies requires source.fetch. If nothing custom runs, the tool was not needed. A source table with neither is an error: a URL or a path is written as a plain string, not a table.

• No needed_by on a source.dependencies entry. A fetch prerequisite always blocks the parent's spec fetch, which is the earliest gate there is. Declare the dependency in the spec's DEPENDENCIES to couple it to a later phase instead. Since envy 0.3.1 the key is refused rather than ignored.

• Strong references only, through the whole closure. Every source.dependencies entry needs its own spec and source:

  error: source.dependencies entry 'tools.jf@r1' in spec 'corp.sdk@r1' must be a
         strong reference

  The rule extends transitively. A fetch dependency and everything it pulls in run their phase ladders during resolution, so nothing in that closure may hold an unresolved weak reference either:

  error: source.dependencies closure must use strong dependencies: 'corp.tls@r1'
         holds a weak reference to 'openssl' but runs outside the window where
         weak references resolve

  The weak pass runs at a resolution barrier, after every spec fetch, including the one still waiting on this function. There is no point at which a weak reference could be ordered in time. The same applies to the PACKAGE_DEPOTS and DEFAULT_SHELL closures, which run in the same window.

• Products work inside the fetch function. source.dependencies entries are wired with needed_by = spec_fetch before it runs, so envy.product("jf") and envy.package(identity) resolve there.

• Chains work. A fetch dependency's own spec can have fetch dependencies, and bootstrap chains resolve bottom-up.

• Cycles are detected among fetch dependencies and reported.

The same machinery appears twice more without being called this: fetching bundles, and authenticated depot indexes with DEPENDS, which hands its fetch function ctx.deps[identity].pkg_path.