Generated brew formula test asserts a --help string fj never emits #235

New issue

Open

opened 2026-07-30 16:41:39 +00:00 by stephen · 0 comments

stephen commented

2026-07-30 16:41:39 +00:00

Owner

The generated Homebrew formula's test do block asserts against text fj --help does not emit. The assertion is a false negative waiting to happen and, worse, it is currently passing for a reason unrelated to what it claims to check.

Where

.forgejo/workflows/release.yml, in the heredoc that generates the tap formula. Line 440 at the time of filing:

test do
  assert_match "fj #{version}", shell_output("#{bin}/fj --version")
  assert_match "Command-line tool for Forgejo", shell_output("#{bin}/fj --help")
end

What is wrong

assert_match "Command-line tool for Forgejo" does not correspond to any string in fj --help output. What the binary actually emits starts:

$ fj --help
fj is a command-line tool for Forgejo, in the spirit of GitHub's gh. It spans multiple hosts,
stores tokens in your OS keychain or a 0600 file fallback, and mirrors the gh command surface
for what Forgejo exposes: ...

Two differences, either of which alone breaks the match:

Case. The help text is a sentence: a command-line tool for Forgejo, lowercase c. The assertion capitalises it.
Preceding words. The help text reads fj is a command-line tool, so even case-insensitively there is no substring Command-line tool for Forgejo starting at a word boundary the assertion would land on.

The capitalised string is the desc field of the formula (line 405, desc "Command-line tool for Forgejo, in the spirit of gh"), not the help output. The assertion looks like it was copied from the desc line a few dozen lines above it in the same heredoc, which is exactly the kind of near-miss that reads as correct in review.

Why it passes today, for the wrong reason

brew test is not run by the release workflow. The formula is generated, attached to the release, and copied into the tap by hand, so nothing in CI ever executes the test do block. The assertion has never been evaluated. It is not "passing", it is unexercised, which is why a string that cannot match has survived.

The failure surfaces the first time anyone runs brew test rasterstate/tap/fj, or the first time a tap audit does. At that point it reports the binary as broken when the binary is fine, which is the worst class of test failure to hand somebody: it points at the wrong thing.

Minimal repro

fj --help | grep -q "Command-line tool for Forgejo"; echo "exit=$?"   # exit=1
fj --help | grep -q "command-line tool for Forgejo"; echo "exit=$?"   # exit=0

Or against the real surface:

brew install rasterstate/tap/fj
brew test rasterstate/tap/fj    # fails on the --help assertion, passes on --version

Notes for whoever fixes it

The --version assertion on the line above is correct and should stay: fj --version emits fj 0.4.0, so assert_match "fj #{version}" matches.
Whatever replacement string is chosen should be one the help text is unlikely to reword. fj is a command-line tool for Forgejo is the current opening clause; a shorter anchor is less likely to drift.
Worth considering separately: nothing runs brew test anywhere, so any assertion in that block can rot the same way. Fixing the string without addressing that leaves the next one to be found by a user.

Found while working the paragon :2222 retirement chain; filing rather than fixing so it does not ride along on an unrelated PR.

The generated Homebrew formula's `test do` block asserts against text `fj --help` does not emit. The assertion is a false negative waiting to happen and, worse, it is currently passing for a reason unrelated to what it claims to check. ## Where `.forgejo/workflows/release.yml`, in the heredoc that generates the tap formula. Line 440 at the time of filing: ```ruby test do assert_match "fj #{version}", shell_output("#{bin}/fj --version") assert_match "Command-line tool for Forgejo", shell_output("#{bin}/fj --help") end ``` ## What is wrong `assert_match "Command-line tool for Forgejo"` does not correspond to any string in `fj --help` output. What the binary actually emits starts: ``` $ fj --help fj is a command-line tool for Forgejo, in the spirit of GitHub's gh. It spans multiple hosts, stores tokens in your OS keychain or a 0600 file fallback, and mirrors the gh command surface for what Forgejo exposes: ... ``` Two differences, either of which alone breaks the match: 1. **Case.** The help text is a sentence: `a command-line tool for Forgejo`, lowercase `c`. The assertion capitalises it. 2. **Preceding words.** The help text reads `fj is a command-line tool`, so even case-insensitively there is no substring `Command-line tool for Forgejo` starting at a word boundary the assertion would land on. The capitalised string is the **`desc` field** of the formula (line 405, `desc "Command-line tool for Forgejo, in the spirit of gh"`), not the help output. The assertion looks like it was copied from the `desc` line a few dozen lines above it in the same heredoc, which is exactly the kind of near-miss that reads as correct in review. ## Why it passes today, for the wrong reason `brew test` is not run by the release workflow. The formula is generated, attached to the release, and copied into the tap by hand, so nothing in CI ever executes the `test do` block. The assertion has never been evaluated. It is not "passing", it is unexercised, which is why a string that cannot match has survived. The failure surfaces the first time anyone runs `brew test rasterstate/tap/fj`, or the first time a tap audit does. At that point it reports the binary as broken when the binary is fine, which is the worst class of test failure to hand somebody: it points at the wrong thing. ## Minimal repro ```sh fj --help | grep -q "Command-line tool for Forgejo"; echo "exit=$?" # exit=1 fj --help | grep -q "command-line tool for Forgejo"; echo "exit=$?" # exit=0 ``` Or against the real surface: ```sh brew install rasterstate/tap/fj brew test rasterstate/tap/fj # fails on the --help assertion, passes on --version ``` ## Notes for whoever fixes it - The `--version` assertion on the line above is correct and should stay: `fj --version` emits `fj 0.4.0`, so `assert_match "fj #{version}"` matches. - Whatever replacement string is chosen should be one the help text is unlikely to reword. `fj is a command-line tool for Forgejo` is the current opening clause; a shorter anchor is less likely to drift. - Worth considering separately: nothing runs `brew test` anywhere, so any assertion in that block can rot the same way. Fixing the string without addressing that leaves the next one to be found by a user. Found while working the paragon `:2222` retirement chain; filing rather than fixing so it does not ride along on an unrelated PR.