Appverse Review Rubric

Related Docs: Appverse Reviewer Process | Appverse Contributor Guide | Appverse Best Practices | Appverse README Template

The criteria for Appverse app review. The automated review (Appverse Review) and the human reviewer both work from this document; the Reviewer Process says what to do with it and in what order. This page replaces the earlier Security Rubric, which is now the Security section below.

How to read this rubric

A review produces three kinds of content. Keeping them apart is the point of this document.

1. Gate criteria (pass / fail). Hard requirements an app must meet to be listed. They live in the Structure section. An app that fails one cannot be accepted until it is fixed.
2. Signals (Low / Medium / High). A per-dimension reading for a deployer on three axes: Portability, Documentation, and Upkeep. Low = the good end; High = the most to read before deploying, on every axis. Signals describe; they never decide. A High signal on its own does not reject an app, and a Low signal does not accept one.
3. The decision. Accept, accept with suggestions, request changes, or reject, derived from the gate criteria and from properties of individual findings. The rules are in the Decision rubric section at the end.

Signals do not gate; the decision rubric does. Two consequences worth stating: a fixable High-severity security finding is "request changes", not "reject"; and a Documentation or Portability signal below the target for inclusion lands in "accept with suggestions", not "request changes".

Code Quality is a fourth kind of content: findings that are neither a gate nor a signal. They are reported as findings and the decision rubric sets a target for two of them (error handling and input validation).

The README has two roles. "README present and substantive" is a Structure gate (a stub fails). README depth (Minimal / Adequate / Strong / Exemplary) is the Documentation signal. Same file, two different legs.

Each finding the automated review records carries a rule code (STR-, OODT-, QUA-, MNT-), a severity, and file:line evidence. The severity scale is defined in the plugin's finding-codes.md. Each finding also carries a result: FAIL means the criterion is not met; WARN means the tool could not confirm it or the defect is advisory — on a gate row, a WARN is for the reviewer to settle in Step 3 of the Process, not an automatic pass.

Repo shapes

A repo with a root appverse.yml is declared: the contributor states how their app(s) appear in the catalog. A repo with only a root manifest.yml is inferred: the catalog derives a single app from that file plus GitHub metadata.

A declared repo whose appverse.yml has an apps: list is a Monorepo. Review every entry in apps[] as its own app — each gets its own structure, security, and quality assessment, and its own decision. Per-app fields resolve with this precedence:

1. Inline in the root appverse.yml apps[] entry (highest)
2. <subpath>/appverse.yml
3. <subpath>/manifest.yml (name/description fallback only)

Directories listed in shared_paths are shared code: review them once at repo level and include them in the security review.

Installing any one app clones the whole repo, so a monorepo is gated as a whole before its apps are decided. Two kinds of problem decide every app the same way:

• A repo-level problem: a failed repo-level gate criterion (Request changes when it is fixable) or an unmaintained repo (Reject), as the Decision rubric below says for any repo.
• A High or Critical security finding, wherever it is: in shared_paths, at the repo root, or inside one app's directory. High sets at least Request changes and Critical sets Reject for every app (see the severity scale in finding-codes.md).

Past that gate, decisions can differ per app. For example, three apps can be accepted while a fourth gets Request changes for missing metadata. The Overall recommendation rolls the apps up: it is the strictest Per-app decision.

shared_paths is scope input for the security review; it is not a pass/fail criterion.

appverse.yml is optional when a root manifest.yml is present: the catalog infers a single app from the manifest. A repo with neither file has no identifiable shape and fails the Structure gate below.

Structure (gate criteria)

All of these are gates: every one must pass for the app to be listed. The tool records each as PASS or FAIL; where it could not confirm a row it records WARN or NOT CHECKED, and the reviewer settles that row in Step 3 of the Reviewer Process before deciding (see "How to read this rubric").

Repository structure

Every repo, regardless of shape:

README and LICENSE are checked once at repo level. In a monorepo the apps share them; an app-level README at a subpath is welcome but is not required by this gate.

Inferred repos (root manifest.yml, no appverse.yml):

Declared repos (root appverse.yml), checked per app after field-precedence resolution. The same fields are required whether the repo declares one app or several:

Batch Connect layout

Standard OOD structure: form.yml or form.yml.erb, submit.yml.erb, and a template/ directory containing the job script (script.sh.erb, or script.sh when nothing is templated). OOD renders form.yml.erb at request time, so an app shipping only the .erb variant is complete; shipping both is unusual and worth a look.

Passenger and companion apps substitute their own layout check: an entry point that exists and parses (config.ru for Ruby/Rack, passenger_wsgi.py for Python/WSGI), a manifest.yml role consistent with that layout, and a dependency manifest (Gemfile.lock, package-lock.json, requirements.txt) consistent with the dependency file (STR-08 if missing or inconsistent).

Validity

Security

Security is a list of findings, never a rating. The tool does not assert a security level; an app with no FAIL or WARN findings is reported as having no tool-detectable issues in the checked tiers, which is not a claim that it is safe. Some finding properties feed the decision rubric directly. Every finding is classified under the OODT taxonomy and rated on the five-level severity scale.

Decision role: a finding tagged potentially malicious, or one that cannot be fixed without redesigning the app, is a reject trigger. A fixable High finding is "request changes". Low and Medium findings do not by themselves block "accept with suggestions" (see Decision rubric).

How the automated review checks security

The security review has three tiers, named in every report so a thinner review looks thinner rather than identical to a full one:

• Tier 1 — Static. Source-level reading: the candidate loop and the open-ended pass below, then the capability profile. Runs anywhere, including CI on a submitted pull request.
• Tier 2 — Tooling. Static analysis tools (shellcheck, bandit, semgrep, trivy, and others per the plugin's security-tools.md). Needs installed binaries, not a running app. Best-effort: the review proceeds without them. Run before the review by the plugin's pre-review script; the report's tool table lists what ran.
• Tier 3 — Runtime. Boot the app and exercise it, only in an isolated, disposable environment (never on a reviewer's own machine or a shared host). Reported as NOT CHECKED — no isolated execution environment when none is available, which is the usual case.

Tier 1 is a loop, not an open-ended read. The plugin's pre-review script lists every candidate site in the app: each ERB interpolation of a form value in submit.yml.erb and template/ (with whether it is quoted and guarded), each unquoted expansion of a form value, each eval/exec, credential-shaped string, chmod/chown/umask, network call, write outside the job directory, security-relevant config flag (--no-auth, --disable-ssl, 0.0.0.0, a CORS wildcard), and binary file under template/. Each candidate names its file, line and the OODT code it would fall under. The review answers every candidate with its own row in the report: FAIL or WARN with category and severity, or PASS with one line of why it is acceptable. The candidate list is the same on every run, so every run is asked about the same sites. After the loop, an open-ended pass looks for what the enumeration cannot see (a missing CSRF defense, partial authentication, container isolation, an unusual way of phoning home) and records each under "Additional observations (review)", saying the enumeration missed it. "No tool-detectable issues in the checked tiers" is the claim when no row in either table is FAIL or WARN; never write it when any row is FAIL or WARN, and never write "safe".

Two complementary methods feed the same classification:

• Capability profiling catalogs what an app does — what system access it needs, what network calls it makes, what files it reads and writes, what processes it spawns. For batch connect apps, with their narrow expected profile, unusual capabilities are a strong signal of malicious intent. For Passenger apps, with legitimately broad capabilities, the profile is a transparency tool — it tells reviewers and deployers exactly what they are putting on their system so they can make an informed decision. An app is never penalized for doing what it is designed to do.
• Pattern checks catch capabilities used unsafely, regardless of app type. Form values reaching shell commands without sanitization, eval() on config-supplied code, a service with --no-auth, credentials stored in plain text. Normal tools, unsafe usage.

Capability baseline: Batch Connect apps

The expected profile is narrow. Anomalies are a strong signal.

Capability baseline: Passenger apps

The expected profile is wide. The full profile is reported for transparency. Capabilities in the "Flagged" column generate findings.

Dashboard apps and widgets — no detailed baseline yet; there are not enough examples in the catalog. Apply the pattern checks and report whatever capabilities are found. Baselines will be built out as more of these enter the catalog.

Pattern checks (all app types)

These catch capabilities used unsafely, across all app types.

OODT — Open OnDemand App Threats

OODT is local to Appverse review and describes threats specific to the Open OnDemand execution model. It is not related to OWASP's separate OAT catalog.

Every finding is classified under one of these eight threat types. The risk level is about blast radius, not raw severity: the Per-User Nginx (PUN) architecture already contains a lot of single-user damage, so threats that escape the user's own sandbox (cross-user, cross-site, cross-session) rank above self-harm.

Rating findings

For on-demand review, rate each finding's severity using the five-level scale defined in finding-codes.md:

• Critical — unfixable without redesigning the feature, or tagged potentially malicious (e.g., curl|bash on user-supplied URLs). Warrants a Reject recommendation.
• High — a real vulnerability, fixable with targeted changes (e.g., CORS open to all origins). Warrants Request changes.
• Medium — a genuine concern but lower blast radius or harder to exploit.
• Low — defensive-coding gap or minor hygiene issue.
• Info — positive observations or context (not used for security findings in practice; available for completeness).

Base the rating on blast radius (cross-user or cross-site outranks self-harm) and how easily it is triggered. Tag each finding as unintentional or potentially malicious.

A finding is a capability used unsafely, not a capability that is part of the app's design. A job composer that runs shell commands is doing its job; running them with CORS open to all origins is a finding. Report designed capabilities in the capability profile; score only the unsafe usage.

Severity does not mechanically determine the review decision. A fixable misconfiguration — even High severity, such as CORS open to all origins — points to "request changes," not "reject." Reserve Critical and Reject for findings tagged potentially malicious, or exposures that cannot be fixed without redesigning the app.

The numeric severity/exploitability scoring, the commit-polling audit pipeline, and the catalog security badges are defined separately, not in this rubric. Both the on-demand skill and that pipeline classify findings with the same OODT taxonomy and pattern checks above.

Portability

How much a deployer at another site has to change before the app runs.

Portability signal: Portable = Low; Partially portable = Medium; Not portable = High.

Target for inclusion: Partially portable or above. Below the target is "accept with suggestions" when the gate criteria pass; suggest the contributor move site-specific values into form attributes so a site can set them in its own config.

Where to look: submit.yml.erb, form.yml, form.yml.erb, and template/ scripts, for cluster names, partitions, accounts, absolute site paths, and module versions.

Named checks (recorded as QUA-02 findings with file:line evidence):

• check: hardcoded-site-paths — scratch, project, or home paths, cluster or partition names, or a module version that appears in a script or template rather than in a form.yml attribute or a documented configuration value. Documented site-specific values in the README's configuration table count toward Partially portable; undocumented ones count against it.

Documentation

How well the README answers a deployer who has never seen the app. The README should follow the Appverse README Template.

Documentation signal: Strong or Exemplary = Low; Adequate = Medium; Minimal or Below minimal = High.

Evidence: the rating rests on one evidence line per requirement in the table above (what it launches through architecture), each citing the README heading and line the pre-review's readme.json maps to that requirement, or none. A heading whose section is only the README template's placeholder text counts as none. A screenshots or environment-variables heading with no image or variable under it is "heading only", weaker evidence the reviewer checks. The reviewer may judge that a listed section does not satisfy its requirement. A requirement readme.json shows no heading for is met by a README line that delivers it, cited as content: README.md:N (the path is readme.json's file, the app's own README in a monorepo); the line must be text, not a heading, code, placeholder, contact or badge line, one line meets one requirement, and without one the requirement is none. This content search applies even when the requirement's own heading exists but doesn't deliver it — check other headings' content (configuration met by a line under Environment Variables, say) before settling on none. With no Overview-type heading, a descriptive paragraph under the README's title satisfies "what it launches" (readme.json records it as the intro). The level is the highest one whose requirements, and every lower level's, all have evidence. A README that is not a stub but has no evidence even for Minimal is rated Below minimal, with a QUA-01 docs-minimal FAIL, as for Minimal (both miss the Adequate target). Whether a README is a stub is decided once, by the pre-review facts (readme.json stub: fewer than 100 characters of content, or every section body template placeholder text with no real text above the first heading), never by the rating: a stub is the Structure gate failure (QUA-01 docs-stub), not a rating, and the report says "Minimal — not supported (stub README; see QUA-01)".

Target for inclusion: Adequate or above. Below the target is "accept with suggestions" when the gate criteria pass. A README that fails the Structure gate (stub or unfilled template) is a gate failure, not a Minimal rating. The four questions above are how the signal is rated; they are not a gate. This was previously listed under "Required Criteria" as Documentation Minimum, and the earlier reviewer checklist and report disagreed about it; the split here is the resolution.

Red flags (recorded as QUA-01 findings):

• README references a different institution's paths without noting they need to change
• No Configuration section — deployers can't tell what to customize

Code Quality

Findings about how the app is written. They are not a signal and not a gate. They are reported with file:line evidence, and the decision rubric sets a target for the first two.

Correctness and polish defects are also Code Quality findings: copy-paste artifacts from the template an app was cloned from (a MATLAB reference in a SAS app, a CHANGELOG describing a different app), duplicate YAML keys (valid YAML, but last-wins, so the parser does not catch them and the author almost certainly did not intend it), wrong help text, and README typos.

Draft feedback guidance. Never advise removing a comment or help text that states a real constraint (a partition that requires a GPU, a limit); advise rewording it.

Named checks:

• check: numeric-field-bounds — every numeric form field (number_field, or a text field that is parsed as a number in submit.yml.erb) has min and max, and free-text fields that reach the scheduler have a pattern or are otherwise constrained. A missing bound is an input-validation finding.
• check: icon-matches-target-os — desktop and panel configuration under template/ (for example xfce4-panel.xml button-icon, .desktop Icon=) references an icon that exists on the OS the README says the app was tested on. A Fedora icon in an app documented for Rocky Linux is a polish finding.

Upkeep

Signals about the project's health. Upkeep is a signal on the repo and, in a declared monorepo, on each app as well: apps in one repo can be maintained by different people at different paces. An app's upkeep reads its own folder at the reviewed commit (its last change by a person, its human contributors, a CHANGELOG of its own) by the same rule below, with releases, CI and issues taken from the repo. Bot commits do not count. It says whether people have changed the folder lately, not why, so a repo-wide change by a person (a license header, a lint pass) counts for every app it touches. An app's upkeep is a signal for deployers only; the decision role below stays with the repo's.

Upkeep signal: active within 12 months and two or more good-practice signals = Low; active within 12 months = Medium; inactive over 12 months = High. The good-practice signals are releases, a current CHANGELOG, CI, multiple contributors, and responded-to issues. A repo with no open issues is neutral on that last signal: it counts neither for nor against, because there is no evidence of responsiveness either way.

Target for inclusion: active within 12 months. For a brand-new app, waive the history requirement and say so in the evidence phrase; the follow-up review of brand-new apps will be scheduled by the catalog once reviews are stored there, so no manual tracking is needed today.

Decision role: an abandoned or unmaintained repo is a reject trigger. Good-practice signals never push the axis down and never fail an app.

The other signals — tagged releases, issue responsiveness, contributors, CHANGELOG, CI — are good-practice indicators, not requirements. There is no requirement to cut releases or use versioning; a release is a positive signal that the team follows good practices, and its absence is worth a suggestion, never a failure. CI is the weakest of these for an app repo: most Appverse apps are config-and-template repos with little to test beyond YAML validity and shell/ERB lint, which Appverse Review already checks — so weight a missing CI workflow low.

Decision rubric

The reviewer's decision. Derived from the gate criteria and from properties of individual findings, never from the signal levels.

In a monorepo, apply the repo-level criteria and every High or Critical security finding to the whole repo first, then decide each app (see Repo shapes). check-decisions.py fails a review whose decisions fall below the floors those findings set, or whose recommendation is milder than its strictest Per-app decision. Documentation, portability and code-quality findings never set a floor.

Any Accept is conditional on the catalog checks (duplicate check, Software entry, vocabulary terms). The automated review performs the catalog reads and records what they returned; the duplicate decision and any Software entry creation stay with the reviewer, as the Reviewer Process describes.

Appendix: common issues in real apps

Based on analysis of apps currently in the Appverse: