I have always practiced iterative development. I did it with brew and every other open source project I’ve ever created and every job I’ve ever had (where I was allowed—Apple refused to let me 🤓). I believe the only way to “find the fun”† in development is to try out your ideas and preferably with as large a community as possible. Thank you for your help in “finding the fun” with pkgx. It’s been a very instructive process but now we’re ready for version one.
In this post we talk about what has changed and why. It is a complement to our full documentation and the repo README.
1.0.0-alpha.1 is a seminal release, we welcome feedback, especially from v0 early adopters.
† the legendary developer Sid Meier described his iterative process when developing video games as “finding the fun”. Initially gameplay ideas might actually be tedious; if so; iterate until fun emerges.
Magic
Most notably we have reconsidered “magic”.
Magic was an amazing feature but the truth is it didn’t suit terminals. pkgx’s magic would automatically fetch packages as you typed, even working with complex pipelines.
It had people say wow, but truthfully it was a little unnerving. Terminals are precise, magic was… unpredictable. As engineers we want to know precisely what is happening in our terminals and pkgx’s magic led to numerous situations where our users were “wow I love magic but wtf is going on with this command example?”
So now things are explicit—but we try to make running anything as frictionless as possible:
$ node
^^ pkgx provides this, run it with `pkgx`
$ pkgx
env +node && node
Welcome to Node.js v20.6.0.
Type ".help" for more information.
>
If you type something that your machine doesn’t have but pkgx has it we let you know and then you can just type pkgx.
You can also just type x.
Shell Integration
In the above example typing pkgx expands to env +node && node via our shell integration.
env +node adds node to the environment:
$ pkgx +node
(+node) $ node --version
v20.1.2
(+node) in your prompt indicates your terminal has been supplemented with, here, node
You can keep adding packages:
(+node) $ pkgx +cargo +c++
(+node+cargo+llvm.org) $ node --version; cargo --version; c++ --version
v20.1.2
cargo 1.72.1
clang version 14.0.3
Packages added this way are available for the session. They are gone when you exit.
There was no direct way to do this in v0 though it was an idea we kept playing with because of how useful it can be to construct temporary environments that you then either commit (pkgx install) or discard (exit).
docs.pkgx.sh/shell-integration
Installing Packages
Yeah, yeah. You told us so… and you were right! 😏
We resisted adding an install command for nine long months because we wanted to reinvent package management. We believe we now succeeded and after playing with the successor for a while we realized… sometimes you just need to install shit.
$ pkgx install deno
installed: ~/.local/bin/deno
$ cat ~/.local/bin/deno
#!/bin/sh
exec pkgx +deno.land -- deno "$@"
pkgx doesn’t really install anything—packages are only ever cached in ~/.pkgx—so it follows that installing things with pkgx is really just creating a stub script that runs the package via pkgx.
oh! btw: sudo pkgx install will install to /usr/local/bin
docs.pkgx.sh/pkgx-install
pkg shortnames
While you could always have pkgx run commands just by specifying them directly, eg. pkgx node. For most other uses you would need to specify dependencies with the fully qualified names, eg. pkgx +nodejs.org@18 sh you can now in all cases just specify commands (eg. pkgx +node@18 sh).
In cases where multiple packages provide the same commands we require you to be more specific:
$ pkgx yarn
error: multiple packages provide `yarn`, pls be more specific:
pkgx +classic.yarnpkg.com yarn
pkgx +yarnpkg.com yarn
Previously we allowed YAML front matter in scripts, but with support for shortnames the shebang syntax became much more concise and we decided that the YAML front matter, in fact, obscured intent. Thus we have dropped the feature; if you had something like:
#!/usr/bin/env pkgx
#---
# args:
# python
# dependencies:
# git-scm.org: ^2
# gnu.org/tar: '*'
# python.org: ~3.11
#---
It can now be quite elegantly expressed instead as:
#!/usr/bin/env -S pkgx +git@2 +tar python@3.11
While the YAML front matter is easier to read it required you and others who read your script to understand how pkgx works and know about pkgx. Shebangs are as old as UNIX and by encoding all the information there anyone who knows scripting can understand what is going on.
We really love UNIX and the UNIX philosophy and—with everything we do—seek to supplement its amazing base.
pkgx’s +pkg syntax is now consistent across all usage. When using pkgx as a runner you can add extra packages to instantiations of tools (super handy for eg. pkgx +openssl cargo build ), as a shebang you can add all the packages you need during your script’s execution and with our shell-integration +pkg syntax adds packages directly to your terminal session.
docs.pkgx.sh/using-pkgx/running-anything docs.pkgx.sh/scripts docs.pkgx.sh/shell-integration
Developer Environments
In v0 magic automatically loaded developer environments. This violated the principle of least surprise which (especially on the command line) made using pkgx unpredictable. It also meant you were forced to use magic if you wanted developer environments and not all users wanted all parts of our magic.
The feature also had a secret binary operation. The shell hook that activated the developer environment could not take too long since it could randomly occur anytime you changed directory. Thus the packages in the environment might not actually be installed, instead pkgx relied on the command not found handler (ie. the primary form of magic) to install the packages on first use. Which led to another problem with magic: it only works at the shell prompt and no deeper, ie. your Makefile will just inexplicably fail to find the commands in your devenv.
So in v1 you must opt in to developer environments. It’s just as good a feature as before; we figure out the precise packages you need based on the keyfiles in your project and you can refine those dependency constraints to any versions you want using YAML front matter; it’s just now you have to explicitly turn it on on a per directory basis using a separate tool (that pkgx can run) called dev.
$ cd myproj
myproj $ dev
found: deno.json, .git; env +deno +git
(+deno+git) myproj $ deno --version
Deno 1.33.1
(+deno+git) myproj $ env | grep deno
PATH=~/.pkgx/deno.land/v1.33.1/bin:#…
# …
(+deno+git) myproj $ cd ..
pkgx -deno -git
$ deno --version
command not found: deno
# ^^ environments are only active inside their directories
dev environments persist. New terminal sessions will automatically load your dev environments when you step into them.
Making dev a separate tool brings clarity to the scope of pkgx. dev is entirely built on top of pkgx and the pkgx primitives. It leverages pkgx’s core features to create environments. You see this when you step into dev environments since we output the calls we make, eg. env +deno +git.
This suite of features means the –env and -E flags have been dropped. These flags were confusing and tricky to use correctly, if you had a specific use that you don’t think is covered in v1 let us know and we’ll figure out what you or we should do about it.
docs.pkgx.sh/dev
Endpoints
pkgx.app added support for package “endpoints” in a recent release. Endpoints represent an idiomatic use of the package, for example, Stable Diffusion web UI’s endpoint launches the web UI on an available port and opens your browser to show it. pkgx supports running these endpoints in a “Dockerlike” fashion:
$ pkgx run stable-diffusion-webui
With some of our other local-AI packages we make sure to download models for you which the tool may otherwise leave as an exercise for the reader, eg.
$ pkgx run llama.cpp
# ^^ grabs a compatible model and launches a chat TUI
docs.pkgx.sh/run
Dropped Features
Symlinks to pkgx
In v0 symlinks to pkgx would act like the name of the symlink. While a neat feature it was too easy to create fork bombs with complex scripts or collections of pkgx symlinks that were impossible to truly fix.
Stubs (which pkgx install uses) are also more configurable allowing precise versioning of installed tools.
If you have a lot of symlinks to pkgx currently then we apologize for the inconvenience. We appreciate your early adoptence!
Migration: pkgx install node
Magic
Magic was fun but ultimately had more problems than it was worth. The terminal is inherently a precise environment where you should be specific about your needs.
All the same if you want it back it’s one line of shell code:
command_not_found_handler() { pkgx -- "$@" }
Magical developer environments also meant we couldn’t actually install the tools you needed when you entered project directories since this would make a simple cd operation potentially pause for minutes. Thus we relied on magic to instantiate commands initially giving us a situation where things may or may not be installed which meant your projects may or may not actually work. Hardly great DX.
Supplemental Env Vars ($VERSION, $SRCROOT )
In the process of building pkgx we came to realize our true strength was making the entire open source ecosystem available to you. Open Source is a rich, treasure trove of tooling and there are better tools out there to get variables like these. pkgx’s scope has been tightened considerably and no longer provides these conveniences.
README.md as a Source of Dependency Data
If you used this we may bring it back, let us know. We still believe it is possible to have the README be both human readable and machine readable, but this feature was underused by the community and documenting it increased the perceived complexity of pkgx so we dropped it.
Running Scripts via URL
UNIX tools do one thing and do it well. curl is damn good at loading data from URLs, has over 20 years of battle testing and is super configurable.
curl foo.com/script.py | pkgx python@3.10
Other Minor Changes
- punting through to the system with eg pkgx +node make
- Unpredictable, would pkgx run a system package or pkgx’s package?
- If you still want to run system packages specify their full paths: pkgx +node /usr/bin/make
- this can be super useful, sometimes you need a system tool to have access to external packages
- pkgx +python.org node used to be an error; we guessed that since you were specifying additional packages yourself that you didn’t want us to infer node’s +pkg. This now works.
- –env /-E was a confusing flag that could lead to unexpected behaviors
- auto-symlinking installed tools to ~/.pkgx/.local/bin has been removed
- this caused unexpected behavior and would screw up developer environments
- pkgx install is the new way to add tools to the system environment
- –dry-run removed. We added this because pkgx previously would look for tools in pkgx and if it didn’t find them we would use a system tool (if available) and –dry-run allowed you to figure out what was going on. Since you now must specify full paths if you want pkgx to use a system tool we have dropped this flag.
- Our shell integration doesn’t have support for fish or other alternate shells YET.
- Please PR!
- shell integration is quite a bit more complex now unfortunately…
- –json — debatably useful, open ticket if you want it back
FAQ
How do I search for packages?
Type the command you want. If pkgx has it, it’ll say so.
If you need search that is more “waffley” then use the right tool for that job: a full web interface: https://pkgx.sh/+
How do I update packages?
Much like npx or pipx, pkgx doesn’t “install” packages, we just cache them. However rather than go to the Internet whenever you type a command we just use whatever is already cached if it satisfies the constraints you specify. Thus we also support @latest syntax:
$ pkgx node@latest --version
To be consistent we allow you to invoke pkgx this way, which can lead to some amusing commands:
$ pkgx@latest npx@latest cowsay@latest 'fancy a cuppa?'
________________
< fancy a cuppa? >
----------------
\ ^__^
\ (oo)\_______
(__)\ )\/\
||----w |
|| ||
Getting Started
brew install pkgxdev/made/pkgx
docs.pkgx.sh/installing-w/out-brew
A Thousand Tweaks
The 1.0.0-alpha release is a landmark for us here at pkgx.sh. Thousands of tiny changes were implemented based on user feedback and 9 months of intensely using the tooling across many stacks and many platforms.
Let us know what you think. discussions-link-here