CI output

How a scan's results reach your CI: the exit code, a summary and annotations in GitHub Actions, JUnit for test views, SARIF, Markdown and JSON.

Run it in CI

Store a key as a secret named VALIQAI_API_KEY (which key), then run the scan after your deploy step, against the address you just deployed:

$ npx -y @valiqai/cli scan "$STAGING_URL" --fail-on blocker

valiqai init writes the job for GitHub Actions or GitLab CI. Each CI system has a guide with where the secret goes and a deploy-then-scan example: GitHub Actions, GitLab CI, CircleCI, Jenkins, Bitbucket Pipelines, Buildkite and Azure Pipelines.

Each scan from CI shows where it came from on its results page, like GitHub Actions · main · a1b2c3d, linking the job. The CLI reads that from the variables your CI already sets; add --label nightly for a note of your own.

Results in your CI

Besides the log, results show where your team already looks:

• GitHub Actions, with nothing to set up. The run's summary page gets the verdict, counts by severity, the tests and the worst open problems, each linked. Each open problem at --fail-on or above becomes an error annotation on the run, and other blockers and highs become warnings. GitHub shows up to 10 of each per step. Later steps can read steps.valiqai.outputs.run-url, scan-id, score, blockers, high, medium, status and exit-code. --no-github turns this off.
• GitLab, Jenkins, CircleCI, Bitbucket and Buildkite read JUnit. --output valiqai-junit.xml writes it, and the GitLab job above shows each test in the merge request. ValiqAI shows the steps for the others under Integrations › CI.

In the JUnit file, each test the scan ran is a test case, grouped by part of the app (sign-up and sign-in, onboarding, features). A test fails when it found an open problem at --fail-on or above. A test that couldn't reach a result is skipped, with the reason. A problem no single test found, like a missing security header, at --fail-on or above is a failing case under Site-wide checks.

Reports

--format picks a report to produce when the scan ends, and --output writes it to a file instead of stdout:

With --output and no --format, the file's extension decides: .json, .xml (JUnit), .sarif or .md. Progress still goes to the log. Without --output, the report goes to stdout and the progress to stderr, so valiqai scan --format markdown > comment.md works.

JSON output

With --json, the CLI prints one JSON object on stdout when it's done, and its progress on stderr, so you can pipe the result:

$ valiqai scan https://staging.example.com --json | jq '.counts'

{
  "exitCode": 1,
  "reason": "Found 1 blocker (failing on blocker and above).",
  "failOn": "blocker",
  "id": "cmh2x8k1p0003",
  "url": "https://staging.example.com",
  "type": "e2e",
  "status": "failed",
  "score": 64,
  "counts": { "blocker": 1, "major": 2, "minor": 0, "info": 0 },
  "dismissed": 0,
  "runUrl": "https://www.valiqai.com/runs/cmh2x8k1p0003",
  "problems": [
    {
      "severity": "blocker",
      "title": "Password reset email never arrives",
      "category": "functionality",
      "count": 1,
      "urls": ["https://staging.example.com/reset"],
      "status": "open"
    }
  ]
}

counts and problems use the CLI's severity names: major is High and minor is Medium in ValiqAI. problems lists every problem, worst first, including the ones dismissed in ValiqAI (status: "dismissed"). When the scan never started, only exitCode, reason and failOn are there. With --no-wait it has waited: false, the scan's id, type and runUrl, and no counts.