Skip to content

Running in CI ​

The point of running Atlas from CI is to catch a performance regression the same way you catch a failing unit test — automatically, on the build that caused it.

A complete example ​

sh
export ATLAS_URL=https://atlas.cloudbeat.io
export ATLAS_TOKEN=atlas_pat_...

# Only if the plan or its data live in this repository and may have changed.
atlas files upload -p SHOP ./checkout.jmx ./data/users.csv

atlas run -p SHOP --jmx checkout.jmx --servers 5 \
          --gate "error_rate<1,p95<2000" \
          --junit results.xml --origin ci

--origin ci tells Atlas the run comes from a pipeline, and the run gets a CI badge in the Results list. Most CI systems set a CI variable and the CLI then assumes it anyway; say it when yours doesn't. (CLI 0.8.0 and later.)

If your test plan is maintained in Atlas rather than in the repository, drop the upload step — the run command works on its own.

Use CLI 0.6.0 or later

Results are private, so the CLI reads them with your token. Versions before 0.6.0 do not send it for results, and --gate and --junit then fail with exit code 2 even though the test ran. Pin @cloudbeat/atlas-cli@^0.6.0 in your pipeline.

Pass/fail thresholds ​

--gate fails the command when a threshold is breached, checked against the run's overall totals.

MetricMeaning
error_ratepercentage of failed requests
avgaverage response time, ms
p90, p95, p99percentile response times, ms
maxslowest response, ms
throughputrequests per second
requeststotal requests

Use <, <=, > or >=. Separate several with commas, or repeat --gate.

sh
--gate "error_rate<1,p95<2000,throughput>50"

Setting a gate implies --wait — there is nothing to check until the test has finished.

Which thresholds to start with

error_rate and a percentile. Averages hide the tail: a p95 catches the slow requests a handful of users actually felt, which an average smooths away.

Exit codes ​

CodeMeaning
0the test ran and every threshold passed
1the test ran but a --gate threshold was breached
2something was wrong with the request — bad option, unknown project, not signed in
3the test failed or timed out, or the server could not be reached

The distinction that matters: 1 means your system was too slow, 2 and 3 mean the test never produced a verdict. Treating them the same makes a broken pipeline look like a performance regression.

JUnit report ​

sh
atlas run -p SHOP --servers 5 --junit results.xml

Writes a standard JUnit file with one test case per threshold, so CI shows each one as a separate pass or fail. The run's headline numbers are included as properties. Point your CI's test reporter at it and gates appear alongside your other tests.

Add --ai-summary and the run's AI Summary is printed at the end of the log, and its headline goes into the JUnit report — so a failed gate comes with a sentence on why. The summary never changes the exit code: the gates decide pass or fail.

The gates' verdict is also recorded as the run's outcome in Atlas — passed when every gate passes, failed when one fails, with the breached gate noted — so the team sees it in the Results list without opening the CI log. Anyone can change it later, and --no-set-outcome leaves it alone. (CLI 0.8.0 and later; before that, add --set-outcome to record it.)

Example: GitHub Actions ​

yaml
- name: Load test
  env:
    ATLAS_URL: https://atlas.cloudbeat.io
    ATLAS_TOKEN: ${{ secrets.ATLAS_TOKEN }}
  run: |
    npm install -g @cloudbeat/atlas-cli
    atlas run -p SHOP --servers 5 \
              --gate "error_rate<1,p95<2000" \
              --junit results.xml --origin ci

- name: Publish results
  if: always()
  uses: actions/upload-artifact@v4
  with:
    name: load-test
    path: results.xml

Things worth knowing ​

Load tests take minutes, not seconds. Give the job a generous timeout, and consider running it on a schedule or on merges to main rather than on every push.

Machines may be busy. If another test holds the generators you asked for, your run is queued and starts when they free up. Set --timeout to bound how long you're willing to wait.

A queued or failed run is not a regression. That's the exit-code distinction above — don't let a 3 open a performance bug.

Atlas — load testing by CloudBeat · Cookie settings