neighborly Shell with bashup.events
In mid 2019, I fell in love (with a small Bash library): bashup.events.
In the short term it helped sand down some frustrations with Shell, but in the long term it's inspired/shaped a vision for modular neighborly Shell.
I've been hoping to show it off ever since, but my projects haven't made good examples. I recently wrote something related to neighborly Shell that makes a better example, so this post will unpack that vision a little, look briefly at bashup.events
itself, and then step through a real-world example of how to use it.
In part 2, I made the case that a comprehensive package manager for Shell (which is now cutting its teeth) makes it easier to build a leverage-compounding Shell ecosystem. I imagine most of us will bring some baggage to any effort to imagine this ecosystem or its building blocks, so I'll be explicit about what I have in mind.
At minimum, neighborly Shell is ready for use in a tree of modules that share an execution environment. In practice, this mainly means it should share scarce resources, avoid namespace conflicts, and support multiple consumers of its own.
At its best, it should use pro-community decision razors that reflect awareness of Shell's sharp edges. This hasn't matured into comprehensive commandments, but for example it should probably tend to:
- contain complexity rather than externalize it on consumers
- internalize complexity when it creates significant leverage for consumers
- prefer humane, expressive, near-natural-language external APIs
- use distinct naming patterns to differentiate its stable external API from unstable implementation details
- avoid mutating the global environment in ways that will affect other active modules
- consider whether consumers will need enough control to defer or repeat initialization work
- run its test suite under a matrix of shell options to ensure compatibility with common modes
Note: This intentionally focuses on properties of neighborly Shell without mentioning authors or projects. I don't think we'll have to lovingly hand-craft Shell in our underwater basketweaving studios to tick all of these boxes--but I don't know what'll be best-solved by build-time tooling, shimming tricky builtins, leveraging other neighborly libraries, and what'll fall to developers.
It would also be a moving target until the ecosystem matures. This is related to some of the thoughts on mature Shell packaging in part 2.
I've created a ~brainstorming repository to collect thinking about neighborly Shell. If you're interested, drop a line on the discussion board.
Let's zoom in on how neighborly Shell might approach a concrete problem.
Signal traps are a good example of something the neighborly approach will need to solve for modularity and re-use.
When multiple modules try to trap the same signal(s), the most-recent module will win the trap unless all of them handle traps in a neighborly way. This isn't really easy, though. It isn't all that hard to directly add traps in a neighborly manner--check the existing trap and append yours to it--but the complexity ramps up when we have to consider the ability to remove traps as well. It's just as much (if not more) of a problem if one module is clearing traps that others have set.
There are a few pernicious aspects of this problem:
- Modules affected usually cannot do anything about it. They're almost always already sourced before the trouble starts.
- It just takes one module doing it wrong to spoil it.
- It'd take a lot of duplicate effort for every module to independently get this right.
Neighborly Shell will need a fix for this, but it doesn't prescribe a specific one (and it may take more than one approach to address the majority of cases), so let's think through a few things we might be able to get away with:
-
It should be possible to detect a fraction of overlapping traps at build time, so we might be able to handle a fraction of them by replacing the module traps with code generated at build time.
- We have to be able to ~smell when this approach won't suffice (say, one module adds the trap at source time, and the other later on some condition). It's also possible that the fraction it can handle isn't big enough to merit the work.
- Since each use of the same module could need subtly different source, a precondition would be broader toolchain support for building a single unified script including dependencies (or similar).
- On the up side, it would leave no footprints whenever there isn't an overlap.
-
A neighborly library providing its own ~safe API that abstracts over trap.
- Since it would depend on other modules not ~misusing trap, it would probably be best-paired with countermeasures (like build-time tooling that can block the build if any other modules use trap directly).
- This seems ideal for new development, but ~retrofitting would come with a bit of effort to write or generate patches. I don't imagine it will be easy to convince older upstreams to adopt the dependency until they're ready to buy in to the broader concept/ecosystem. The best compromise is probably a good script/tool for patching direct trap invocations to use the API?
Aside: I don't mean to imply that there aren't existing implementations of this (I know of at least a few).
-
Improve the situation with a trap shim that doesn't entail patching existing code.
- It could be resilient to general ~misuse of trap, though it'll obviously have issues with skeptical modules that do things like test that trap is a builtin before using it.
- It has to be sourced before anyone uses trap, so there'll be some sharp edges (unless tooling applies the shim, and not authors?)
- It'll come with some performance hit. A small one, but it may matter in cases such as a loop where you need to add or remove a trap, do something, and then either remove or add the trap again.
A few months ago I ran into a specific problem with trap clobbering and decided to see if I could leverage bashup.events
to draft a quick fix based on the shim approach (#3). Before we look at the implementation, I'll give you a light overview of bashup.events
.
bashup.events
is exactly the kind of low-furniture, expressive, near-natural-language DSL I mentioned in part 1. It isn't magic, but it is a simple grammar for hanging multiple callbacks on the same named label without unintentional clobbering:
1$ event on "my_event" echo callback1
2$ event on "my_event" echo callback2
3$ event emit "my_event"
4callback1
5callback2
I think most of its power comes from mapping a familiar pattern into a clear Shell idiom, but the stringy nature of the language adds a little bonus: we can use the callbacks a bit like closures. For example, we can set up:
1start_thing_one(){
2 event once stop_the_things @1 stop_thing_one $EPOCHREALTIME # ①
3 sleep 3
4 event emit stop_the_things $EPOCHREALTIME # ②
5}
6stop_thing_one(){
7 echo Did the thing from $1 to $2 # ③
8}
9event on start_the_things start_thing_one
- The "callback" is just normal stringy Shell, so
$EPOCHREALTIME
gets baked in as the first argument to stop_thing_one when we register the single-use event. The @1
in this expression specifies how many additional arguments we can pass to the callback at emit time.
- Since we used
@1
to specify 1 emit-time argument, we can pass $EPOCHREALTIME
again.
- The callback has the start time passed as
$1
and the end time passed as $2
.
If we run it, we get:
1$ event emit start_the_things
2Did the thing from 1641154541.594984 to 1641154544.611540
I'm just touching on the basics of bashup.events
. You can see all of the options/operations in the bashup.events README.
The trap-clobbering issue I ran into a few months ago involves a tool that runs an instance of bash and appends its own EXIT trap to clean up a tmpdir. It clobbers an EXIT trap I'm hoping to use to write the history for these shells to a specific file.
My first swing at this, comity, is a trap builtin shim that gives each sourced file its own trap namespace.
Caution: comity is an experiment to see if we can shim in neighborly trap sharing to integrate modules that independently set traps in a non-neighborly way. It works for a living as a solution to the clobbering problem I described earlier, but I'm still deciding how I feel about it and haven't put much effort into making it robust. I'm just using it as a real, short, non-trivial example of how to use bashup.events
.
To get our bearings, let's focus on the broad-strokes of the shim's logic first:
1trap(){
2 # generate a separate associative array
3 # for each distinct caller by source path
4
5 # handle args
6 case $1 in
7 "''" | -l* | -p*)
8 # pass through invocations we don't care about
9 builtin trap "$@";;
10 -)
11 # if there are additional arguments
12 # disable specified signal events for this caller
13 # otherwise, disable all of its signal events
14 ;;
15 *)
16 local code="$1"
17 # if code seems to be a valid signal spec (ex `trap RETURN`)
18 # try rming this signal event for this caller and abort
19
20 # otherwise, loop for each signal listed
21 # if this caller has this signal event, clear it
22 # otherwise, ensure comity traps the signal
23 # (so it can emit signal events)
24
25 # and add this signal event for this caller
26 ;;
27 esac
28}
The main takeaways are:
- comity creates signal events for each caller
- comity sits on the traps itself in order to emit those signal events
bashup.events
handles invoking each subscriber's callback
Aside: I'm leaving the sourcing of bashup.events
out of the frame, here. If you're curious about these, you can see the source statement and nix expression.
Now, let's zoom in on the loop that adds the signal events and see how it uses bashup.events
:
lines 52-65 of comity.bash (src)
52 for signal in "$@"; do
53 normalizedC="${__comity_signal_map[$signal]}" # ①
54
55 if [[ -v "safe_map[$normalizedC]" ]]; then # ②
56 event off __comity_trapped_$normalizedC ${safe_map["$normalizedC"]}
57 unset "safe_map[$normalizedC]"
58 else
59 # shellcheck disable=SC2064
60 builtin trap "event emit __comity_trapped_$normalizedC" "$normalizedC" # ③
61 fi
62
63 event on __comity_trapped_$normalizedC $code # ④
64 safe_map[$normalizedC]="$code" # ②
65 done
- The ~build process for comity pre-generates an associative array that maps each signal name/number to a single normalized form and appends it to the prepared copy of the library. This just looks up the normalized name for this signal so that comity isn't naively clobbering itself.
- Note that this number appears twice in the source (lines 55 and 64)!
safe_map
references the distinct associative array for each caller. The array for each caller maps signals to callbacks because bashup.events
will need the code/callback in order to remove the signal event later. comity removes the existing signal event before setting a new one to ~match how trap will work.
- Set a trap that will emit an event for this signal. This is being a little lazy; by this point comity knows it'll need to trap the signal. It might already be doing so (for another caller, or because it doesn't currently remove its trap when there are no subscribers), but it's easier/faster to set the trap each time than to check first.
- Finally, add a signal event callback for this specific invocation.
What do I get out of this? bashup.events
frees me from managing a fair fraction of comity's logistic state by hiding it in an expressive abstraction. Not having to manage this state lets comity carry less code, and the expressive abstraction makes what is left easier to understand. Specifically, the code above only has to worry about signal events for the invoking module at any given time:
- It doesn't have to keep a running list of all of the modules it has wired up signal events for.
- It doesn't have to keep (or constantly re-compute) the list of all of the modules/callbacks registered for a given signal in order to invoke their callbacks.
I hope I've done bashup.events
justice, here. I sketched out my vision for neighborly Shell, attempted to model how bashup.events
both inspired and fits into that vision, and showed the role it plays in my shim around the trap
builtin.
I could've used it even more effectively to implement approach #2 (an API abstracting over trap entirely). I nearly nerd-sniped myself into implementing it for a final post in this series, but was narrowly saved by the number of irons I have waiting for me in other fires. Feel free to poke me if you have questions.