Smoke Test Suite

A com­plete, plug-and-play Play­wright smoke test suite for Craft CMS, main­tained against cur­rent Craft releas­es, with CI and deploy-plat­form inte­gra­tions already debugged.

Point it at your site, edit one file, wire it to your deploy, and get pinged in Slack the moment some­thing breaks. It answers the ques­tion you actu­al­ly care about after every deploy: ​“Is the site up, ren­der­ing, and not throw­ing errors?” with­out you hav­ing to remem­ber to check.

Com­pat­i­bil­i­ty #

The suite tests your site from the out­side, so it is not ver­sion-locked to Craft.

What it is #

• Play­wright under the hood, which uses real Chromi­um and real page loads.
• One con­fig file (smoke.config.js) dri­ves every test. You should nev­er need to touch the test files or playwright.config.js.
• Sev­en focused checks, each of which you can turn on/​off by fill­ing in (or leav­ing emp­ty) its sec­tion of the config:

  Test	What it guards against
  availability	5xx, white screens, error text on crit­i­cal URLs
  templates	“200 but the lay­out exploded”
  console	JS errors that break interactivity
  links	bro­ken inter­nal links
  seo	miss­ing title / meta / canon­i­cal, unreach­able robots & sitemap
  forms	contact/​lead forms that silent­ly stop submitting
  cp	con­trol pan­el login down; option­al authen­ti­cat­ed check

• Runs on deploy (via repository_dispatch) or on a sched­ule (cron).
• Slack noti­fi­ca­tions built in but need to be con­fig­ured by you.

Quick­start #

Spend less than 10 min­utes to get to a pass­ing smoke run against your home­page. Then grow it one check at a time.

Pre­req­ui­sites

• Node 20 or new­er (node --version).
• A site with a pub­lic URL (pro­duc­tion or staging).

Step 1: Install (about 2 minutes)

npm install
npm run install:browsers   # downloads Chromium for Playwright

Step 2: Point it at your site (about 1 minute)

Open smoke.config.js and change two lines:

baseUrl: 'https://your-live-site.com',   // no trailing slash
siteName: 'Your Site Name',

Don’t want to edit yet? Run against any URL with an env var:

SMOKE_BASE_URL='https://your-live-site.com' npm test

Step 3: First run (about 1 minute)

npm test

Out of the box, only the checks that make sense with an emp­ty con­fig will run:

• avail­abil­i­ty on /
• con­sole on /
• links on /
• seo on / (+ robots.txt and sitemap)
• cp login page

templates and forms are skipped until you con­fig­ure them. You should see most­ly green. If not, jump to Trou­bleshoot­ing — the first run sur­faces the usu­al sus­pects (wrong CP path, no sitemap, redi­rect on /).

Step 4: See the report

npm run report

This opens the Play­wright HTML report with screen­shots for any failure.

Step 5: Add your crit­i­cal paths

Back in smoke.config.js:

criticalPaths: [
  '/',
  '/about',
  '/services',
  '/blog',
  '/contact',
],

Run again. These now get the full avail­abil­i­ty + error-text treatment.

Step 6: Add a tem­plate check

Con­firm a page didn’t just return 200 but actu­al­ly ren­dered its key elements:

templates: [
  { name: 'Homepage', path: '/', expect: ['header', 'main', 'footer'] },
  { name: 'Blog index', path: '/blog', expect: ['.post-card'] },
],

Step 7: Wire it to your deploy

Once it’s green local­ly, make it run automatically:

1. Copy ci/github-actions/smoke-on-deploy.yml into .github/workflows/.
2. Fol­low your platform’s trig­ger guide in Wire it to your deploy.
3. Set up Slack noti­fi­ca­tions.

That’s it. Every deploy now smoke-tests itself and pings you only if some­thing is wrong.

Con­fig­u­ra­tion ref­er­ence #

Every­thing you con­fig­ure lives in smoke.config.js. This sec­tion doc­u­ments every option. Options read from process.env can also be set in CI with­out edit­ing the file.

Top lev­el

errorText

The defaults catch the com­mon Craft/​Twig/​Yii/​PDO fatal errors. Keep this list tight — an over­ly broad string (like error) will match legit­i­mate con­tent (“404 error page”) and cause false fail­ures. Add strings spe­cif­ic to your stack if you have cus­tom error output.

templates[]

Each entry loads a page and asserts every selec­tor is visible.

templates: [
  { name: 'Blog entry', path: '/blog/hello', expect: ['article h1', 'time', '.entry-body'] },
],

Leave the array emp­ty to skip tem­plate tests.

console

console: { paths: ['/'], ignore: ['favicon.ico'] },

A test fails if any console.error or uncaught excep­tion fires that isn’t ignored. Pages are loaded to networkidle so late errors are caught.

links

links: { paths: ['/'], checkExternal: false, ignore: ['mailto:', 'tel:', '#'] },

Each link is checked with HEAD (falling back to GET if the serv­er rejects HEAD). A sta­tus ≥ 400 fails the test.

seo

seo: {
  paths: ['/'],
  requireCanonical: true,
  requireMetaDescription: true,
  robotsPath: '/robots.txt',
  sitemapPath: '/sitemap.xml',
},

forms[]

forms: [{
  name: 'Contact form',
  path: '/contact',
  formSelector: 'form#contact',
  fields: { fromName: 'Smoke Test', fromEmail: '[email protected]', message: 'Please ignore.' },
  honeypot: 'freeform_form_handle',
  submitSelector: 'button[type="submit"]',
  successText: 'Thanks',
}],

Point form tests at stag­ing or use forms that tol­er­ate test sub­mis­sions — a pass­ing test means a real email/​entry was created.

cp

cp: {
  loginPath: '/admin/login',
  dashboardPath: '/admin/dashboard',
  username: process.env.SMOKE_CP_USERNAME || null,
  password: process.env.SMOKE_CP_PASSWORD || null,
},

Nev­er com­mit cre­den­tials. Set SMOKE_CP_USERNAME / SMOKE_CP_PASSWORD as CI secrets.

timeouts

timeouts: { action: 15000, test: 30000 },

notifications.slack

notifications: {
  slack: {
    webhookUrl: process.env.SMOKE_SLACK_WEBHOOK || null,
    notifyOn: 'failure',
  },
},

Envi­ron­ment vari­ables summary

Wire it to your deploy #

Once it’s green local­ly, run it auto­mat­i­cal­ly after every deploy. The pat­tern is iden­ti­cal on every plat­form: after a suc­cess­ful deploy, POST a repository_dispatch event to GitHub, which the smoke-on-deploy.yml work­flow lis­tens for.

First, copy the work­flow you want into .github/workflows/:

• ci/github-actions/smoke-on-deploy.yml — run after each deploy
• ci/github-actions/smoke-scheduled.yml — run on a cron (uptime monitoring)

Then pick your plat­form below.

Cre­ate a GitHub token (all platforms)

Cre­ate a fine-grained per­son­al access token (or a GitHub App token) with:

• Repos­i­to­ry access: the repo hold­ing this smoke suite.
• Per­mis­sion: Con­tents: read and Actions: read/​write (dis­patch needs the contents:write-equiv­a­lent repository_dispatch scope; a clas­sic token needs the repo scope).

Store it wher­ev­er your deploy plat­form keeps secrets, typ­i­cal­ly as SMOKE_DISPATCH_TOKEN. A suc­cess­ful dis­patch returns HTTP 204 with no body.

Tip: run the curl local­ly first with your token to con­firm a 204 No Content response before wiring it into your platform.

Servd

Servd runs post-deploy com­mands via the In-App Com­mands / project shell.

1. Store the token in Servd as an envi­ron­ment vari­able, e.g. SMOKE_DISPATCH_TOKEN.
2. In your Servd project → Set­tings → Post-Deploy Com­mands, add:

curl -sS -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer $SMOKE_DISPATCH_TOKEN" \
  https://api.github.com/repos/YOUR_ORG/craftquest-smoke-suite/dispatches \
  -d '{"event_type":"site-deployed","client_payload":{"base_url":"https://your-live-site.com"}}'

Replace YOUR_ORG/craftquest-smoke-suite and base_url. Deploy once; with­in a few sec­onds the Smoke — on deploy work­flow should appear in the repo’s Actions tab.

Bud­dy

1. Add the token to Bud­dy under Pipeline → Vari­ables (or work­space-lev­el) as SMOKE_DISPATCH_TOKEN, marked as encrypted/​secret.
2. At the end of your deploy pipeline, add a Run shell com­mand (Local shell) action:

curl -sS -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer $SMOKE_DISPATCH_TOKEN" \
  https://api.github.com/repos/YOUR_ORG/craftquest-smoke-suite/dispatches \
  -d '{"event_type":"site-deployed","client_payload":{"base_url":"'"$SMOKE_BASE_URL"'"}}'

Set SMOKE_BASE_URL as a pipeline vari­able so the same pipeline can tar­get dif­fer­ent envi­ron­ments. In the action’s Run set­tings, choose Run this action only if all pre­vi­ous actions passed so a failed deploy doesn’t trig­ger a false smoke run.

Lar­avel Forge

Forge deploys Craft sites via its Deploy Script. Because Forge halts the deploy script on the first fail­ing com­mand, the curl is only reached when the deploy itself succeeded.

1. Site → Envi­ron­ment — add SMOKE_DISPATCH_TOKEN=ghp_xxx and SMOKE_BASE_URL=https://your-site.com.
2. Site → Deploy Script, at the very bot­tom (after your Craft steps):

# Kick off external smoke tests — only reached if the deploy above succeeded.
curl -sS -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer $SMOKE_DISPATCH_TOKEN" \
  https://api.github.com/repos/YOUR_ORG/craftquest-smoke-suite/dispatches \
  -d "{\"event_type\":\"site-deployed\",\"client_payload\":{\"base_url\":\"$SMOKE_BASE_URL\"}}"

Deploy­HQ

Deploy­HQ can run SSH com­mands or fire a web­hook after a suc­cess­ful deployment.

Option A — Post-deploy SSH com­mand (rec­om­mend­ed). In Deploy­HQ: Project → Set­tings → SSH Com­mands, add a com­mand that runs After Changes are Applied:

curl -sS -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer YOUR_GITHUB_TOKEN" \
  https://api.github.com/repos/YOUR_ORG/craftquest-smoke-suite/dispatches \
  -d '{"event_type":"site-deployed","client_payload":{"base_url":"https://your-live-site.com"}}'

Deploy­HQ SSH com­mands do not expand your server’s env vars reli­ably, so it is sim­plest to inline the token and URL here. Restrict the token’s scope accord­ing­ly, and rotate it if the project is shared.

Option B — Web­hook. DeployHQ’s built-in web­hook posts its own JSON pay­load, which the GitHub repository_dispatch API will reject (wrong shape). If you pre­fer web­hooks, point Deploy­HQ at a tiny relay (see the cus­tom bash pat­tern below) that reshapes the pay­load. For most mem­bers, Option A is few­er mov­ing parts.

Any bash deploy

If your deploy is ​“SSH in and run a script”, or any plat­form not cov­ered above, put this at the end of your deploy script and use set -e so it is only reached when every pri­or step succeeded:

#!/usr/bin/env bash
set -euo pipefail

# ... your existing deploy steps (composer install, craft up, etc.) ...

# Only reached if everything above succeeded:
curl -sS -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer $SMOKE_DISPATCH_TOKEN" \
  https://api.github.com/repos/YOUR_ORG/craftquest-smoke-suite/dispatches \
  -d '{"event_type":"site-deployed","client_payload":{"base_url":"https://your-live-site.com"}}'

If one script deploys mul­ti­ple envi­ron­ments, pass the URL through:

BASE_URL="$1"   # e.g. ./deploy.sh https://staging.example.com
curl -sS -X POST \
  -H "Authorization: Bearer $SMOKE_DISPATCH_TOKEN" \
  https://api.github.com/repos/YOUR_ORG/craftquest-smoke-suite/dispatches \
  -d "{\"event_type\":\"site-deployed\",\"client_payload\":{\"base_url\":\"$BASE_URL\"}}"

The work­flow reads client_payload.base_url into SMOKE_BASE_URL, over­rid­ing smoke.config.js.

Debug­ging a dis­patch that ​“does nothing”

1. Run the curl by hand. Any­thing oth­er than 204 is your prob­lem — the body of a 401/422 explains it.
2. Con­firm event_type match­es types: [site-deployed] in smoke-on-deploy.yml.
3. repository_dispatch only trig­gers work­flows on the repo’s default branch. Make sure the work­flow file is merged to main.
4. Con­firm the repo path in the curl URL is exact­ly owner/repo, and that it points at the smoke suite repo — not your site’s repo.

Slack noti­fi­ca­tions #

Get pinged in Slack the moment a smoke run fails.

Pre­req­ui­site: you need per­mis­sion to cre­ate an app in your Slack work­space. If your work­space restricts this, ask a work­space admin to either grant it or cre­ate the web­hook and hand you the URL. Incom­ing Web­hooks are a free, built-in Slack fea­ture — there is noth­ing to install or pay for.

1. Cre­ate an Incom­ing Webhook

1. Go to https://​api​.slack​.com/apps → Cre­ate New App → From scratch.
2. Name it (e.g. ​“Smoke Suite”) and pick your workspace.
3. In the app, open Incom­ing Web­hooks → tog­gle On.
4. Click Add New Web­hook to Work­space, choose the chan­nel (e.g. #deploys), and Allow.
5. Copy the web­hook URL. It looks like https://hooks.slack.com/services/T000/B000/XXXX.

2. Store it as a secret

Nev­er com­mit the web­hook. Add it to GitHub:

• Repo → Set­tings → Secrets and vari­ables → Actions → New repos­i­to­ry secret
• Name: SMOKE_SLACK_WEBHOOK
• Val­ue: the web­hook URL

Or from the com­mand line (the val­ue is entered inter­ac­tive­ly, so it nev­er lands in your shell history):

gh secret set SMOKE_SLACK_WEBHOOK

The work­flows already ref­er­ence secrets.SMOKE_SLACK_WEBHOOK, and smoke.config.js reads process.env.SMOKE_SLACK_WEBHOOK.

3. Choose when to notify

In smoke.config.js:

notifications: {
  slack: {
    webhookUrl: process.env.SMOKE_SLACK_WEBHOOK || null,
    notifyOn: 'failure', // 'failure' (default) or 'always'
  },
},

• 'failure' — only ping when the suite goes red (rec­om­mend­ed).
• 'always' — ping on every run, green or red (use­ful while dial­ing in).

4. Test it locally

SMOKE_SLACK_WEBHOOK='https://hooks.slack.com/services/...' \
  node ci/notify.js --status failure

You should see a mes­sage land in your chan­nel. If not:

• A 403/404 from Slack means the web­hook is wrong or was revoked.
• notify: no Slack webhook configured, skipping. means the env var was empty.

What the mes­sage looks like

🔴 Smoke suite failure — My CraftQuest Site
URL: https://your-live-site.com
Passed: 6 · Failed: 1 · Flaky: 0 · Skipped: 2
View run

The sum­ma­ry line comes from Playwright’s results.json; the View run link points at the GitHub Actions run when available.

Writ­ing your own tests #

The built-in tests cov­er the ​“is it up and ren­der­ing” basics for any Craft site. Soon­er or lat­er you’ll want to assert some­thing spe­cif­ic to your site — that the cart shows a total, that a gat­ed page redi­rects, that a search returns results. This is how.

Two ways to extend

1. Con­fig-only (pre­ferred when it fits): add entries to templates or forms in smoke.config.js. No JavaScript required. Cov­ers ​“this page ren­ders these ele­ments” and ​“this form submits”.

2. A new spec file: drop a *.spec.js into tests/. Play­wright picks it up auto­mat­i­cal­ly. Use this for behav­ior the con­fig can’t express.

Anato­my of a spec

const { test, expect } = require('@playwright/test');
const smoke = require('../smoke.config');

test('cart shows a running total', async ({ page }) => {
  await page.goto('/shop/cart');
  const total = page.locator('[data-cart-total]');
  await expect(total).toBeVisible();
  await expect(total).toContainText('$');
});

Key points:

• baseURL is already set from smoke.config.js, so page.goto('/shop/cart') resolves against your site. Use rel­a­tive paths.
• Import smoke if you want to reuse con­fig val­ues (base URL, cre­den­tials, site-spe­cif­ic lists you add).
• Pre­fer sta­ble selec­tors. Add data-* hooks in your tem­plates (data-cart-total) rather than depend­ing on class names that change with CSS.

Com­mon patterns

Assert a redirect

test('members-only page redirects anonymous visitors', async ({ page }) => {
  const res = await page.goto('/members/dashboard');
  expect(page.url()).toContain('/login');
});

Search returns results

test('site search finds a known entry', async ({ page }) => {
  await page.goto('/search?q=craft');
  await expect(page.locator('.search-result').first()).toBeVisible();
});

An API/JSON end­point responds

test('menu JSON endpoint returns items', async ({ request }) => {
  const res = await request.get('/actions/menu/get');
  expect(res.status()).toBeLessThan(400);
  const body = await res.json();
  expect(Array.isArray(body.items)).toBe(true);
});

Authen­ti­cat­ed flows

Reuse the CP cre­den­tials pat­tern: read from env, skip when absent.

const hasCreds = smoke.cp.username && smoke.cp.password;
(hasCreds ? test : test.skip)('logged-in user sees their name', async ({ page }) => {
  // ... perform login, then assert ...
});

Keep your addi­tions upgrade-safe

When a new ver­sion of the Suite ships, you’ll typ­i­cal­ly replace the stock spec files. To avoid los­ing your work:

• Put all cus­tom specs in clear­ly named files, e.g. tests/site-checkout.spec.js, tests/site-membership.spec.js. Nev­er edit the stock availability.spec.js etc. — add along­side them.
• Dri­ve any­thing reusable through addi­tions to smoke.config.js (you own that file; upgrades won’t over­write your val­ues, only the shape — the changel­og notes any con­fig changes).

Run­ning just your file

npx playwright test tests/site-checkout.spec.js
npx playwright test -g "cart shows"   # by test title
npm run test:ui                        # interactive runner, great for authoring

A word on flakiness

Smoke tests should be deter­min­is­tic. If a test only pass­es sometimes:

• Wait for state, don’t sleep. Use expect(locator).toBeVisible() (auto-waits), not fixed timeouts.
• Load to the right point: waitUntil: 'networkidle' for JS-heavy pages.
• Avoid assert­ing on con­tent that changes every request (time­stamps, ran­dom fea­tured items) unless that’s the point.

Trou­bleshoot­ing #

The fail­ures every­one hits, in rough­ly the order peo­ple hit them. Each has the symp­tom, the cause, and the fix.

availability: / returned 301/302

Symp­tom: the home­page ​“fails” with a redi­rect sta­tus. Cause: your baseUrl redi­rects (http→https, non-www→www, or a trail­ing slash rule). Fix: set baseUrl to the canon­i­cal URL your site actu­al­ly serves — the one with no redi­rect. Check with curl -I https://your-site.com. If the final URL is https://www.your-site.com/, use that.

cp: login page loads fails with 404

Symp­tom: the con­trol pan­el test can’t find the login page. Cause: your cpTrigger isn’t admin, or the CP is on a sub­do­main. Fix: set cp.loginPath to match your Craft con­fig. If cpTrigger is cp, use /cp/login. If the CP is on cms.your-site.com, either run a sep­a­rate con­fig against that host or set SMOKE_BASE_URL for the CP run.

seo: sitemap is reachable fails

Symp­tom: sitemap returns 404. Cause: Craft has no sitemap by default — it comes from a plu­g­in (SEO­mat­ic, craft-sitemap, etc.), each with its own URL. Fix: set seo.sitemapPath to your plugin’s actu­al sitemap URL (SEO­mat­ic is often /sitemaps-1-sitemap.xml). No sitemap plu­g­in? Remove the check by point­ing it at a URL you know 200s, or accept the skip.

console: / has no JS errors fails on third-par­ty noise

Symp­tom: fail­ures from Google Maps, ana­lyt­ics, chat wid­gets, or favicon.ico. Cause: third-par­ty scripts log errors you don’t con­trol. Fix: add a sub­string to console.ignore in smoke.config.js. Keep it spe­cif­ic ('Google Maps JavaScript API'), not broad. Don’t ignore errors com­ing from your own JS — fix those.

links fails on links that work in a browser

Symp­tom: a link reports 403 or 405 but loads fine when you click it. Cause: some servers/​CDNs block HEAD requests or bot-like user agents, or the link requires a ses­sion. Fix: the test already falls back from HEAD to GET on ⁴⁰⁵⁄₅₀₁. For 403 from a CDN, add the link (or a path frag­ment) to links.ignore. For auth-gat­ed links, ignore them — smoke tests run anonymously.

Forms test cre­ates real sub­mis­sions / emails

Symp­tom: run­ning the suite floods an inbox or CRM. Cause: the form test gen­uine­ly sub­mits the form. Fix: point forms[].path at stag­ing, or use a form that routes test sub­mis­sions some­where harm­less. Make successText match a string that appears only on suc­cess so a val­i­da­tion error doesn’t read as a pass.

Every­thing pass­es local­ly, fails on CI right after deploy

Symp­tom: green local­ly, red in the on-deploy run, green if you re-run. Cause: the CDN/​edge cache is still warm­ing, or the deploy plat­form fired the trig­ger before the new release was live. Fix: retries is already 1 on CI to absorb this. If it per­sists, add a short delay before the dis­patch in your deploy trig­ger, or increase timeouts.action. Con­firm your plat­form trig­gers after the release is pro­mot­ed, not at build time.

Executable doesn't exist / brows­er not installed

Symp­tom: Play­wright com­plains the Chromi­um bina­ry is miss­ing. Cause: browsers weren’t installed. Fix: run npm run install:browsers. On CI the work­flow already does this; if you cus­tomized it, ensure playwright install --with-deps chromium runs before npm test.

Slack noti­fi­ca­tion nev­er arrives

Symp­tom: runs go red but no Slack mes­sage. Caus­es & fixes:

• notify: no Slack webhook configured — the SMOKE_SLACK_WEBHOOK secret isn’t set in the repo. Add it (see Slack noti­fi­ca­tions).
• Mes­sage on suc­cess expect­ed but notifyOn is 'failure' — set it to 'always'.
• Slack returns ⁴⁰³⁄₄₀₄ — the web­hook was revoked or mistyped; regen­er­ate it.
• The noti­fy step didn’t run — con­firm the work­flow has the if: failure() noti­fy step and that npm test actu­al­ly failed (a skipped suite is not a failure).

repository_dispatch fires but no work­flow runs

Symp­tom: your deploy curl returns 204 but noth­ing shows in Actions. Caus­es & fixes:

• The work­flow file must be on the repo’s default branch for repository_dispatch to trig­ger it. Merge it to main.
• event_type in your curl must match types: [site-deployed] in the workflow.
• The token lacks the scope to dis­patch — a fine-grained token needs Con­tents and Actions read/​write on that repo.
• You’re look­ing at the wrong repo — dis­patch tar­gets the repo in the curl URL, which should be the smoke suite repo, not your site’s repo.

Still stuck?

See Sup­port­ed con­fig­u­ra­tions & maintenance for how to report a break. Include the fail­ing test name, the full error out­put, your smoke.config.js (redact secrets), your Node ver­sion, and whether it’s local or CI.

Sup­port­ed con­fig­u­ra­tions & main­te­nance #

The main­te­nance is the prod­uct. This sec­tion defines it pre­cise­ly — what is com­mit­ted, what is delib­er­ate­ly not, and which con­fig­u­ra­tions are in scope.

Sup­port­ed at v1.0

• Craft CMS: sin­gle-site installs, ver­sions 3 through 6, with stan­dard front-end login pat­terns. The tests only touch the front end + CP login, so the Craft ver­sion is large­ly irrelevant.
• Node: 20 or new­er (uses the built-in fetch).
• CI run­ner: GitHub Actions.
• Deploy trig­gers: Servd, Bud­dy, Forge, Deploy­HQ, and any cus­tom bash deploy.
• OS (local runs): macOS, Lin­ux, Win­dows (Play­wright installs its own browser).

Com­mit­ted

• Ver­i­fied against every new Craft GA minor release with­in 14 days, with a changel­og entry either con­firm­ing com­pat­i­bil­i­ty or ship­ping the fix­es required.
• Play­wright and depen­den­cy updates at least quarterly.
• Secu­ri­ty-rel­e­vant depen­den­cy patch­es as needed.
• Break­ing changes in the suite are announced in the changel­og with migra­tion notes (and bump the major version).

Explic­it­ly not committed

• No cus­tom test devel­op­ment for indi­vid­ual mem­ber sites. Writ­ing the tests spe­cif­ic to your tem­plates and flows is yours — see Writ­ing your own tests.
• No debug­ging of mem­ber-spe­cif­ic CI envi­ron­ments beyond the doc­u­ment­ed platforms.
• Unsup­port­ed con­fig­u­ra­tions are roadmap items, not obligations.

Doc­u­ment­ed as unsup­port­ed at launch

These are roadmap can­di­dates, not com­mit­ments: mul­ti-site installs, head­less / decou­pled front ends, Git­Lab CI / Bit­buck­et Pipelines, and SSO or cus­tom con­trol-pan­el authentication.

Ver­sion­ing

The suite fol­lows semver, decou­pled from Craft’s ver­sion num­bers — there is no ​“Smoke Suite 5.x for Craft 5”. The com­pat­i­bil­i­ty matrix at the top com­mu­ni­cates Craft sup­port instead.

• MAJOR — break­ing changes to smoke.config.js for­mat, or required Node/​Playwright ver­sion bumps.
• MINOR — new tests, new CI/​trigger inte­gra­tions, new con­fig options.
• PATCH — fix­es, Play­wright bumps, com­pat­i­bil­i­ty updates for new Craft releases.

Report­ing a break

Mem­bers don’t have repo access, so there are no GitHub Issues. Use the sup­port form on the suite’s mem­ber page and pick the right cat­e­go­ry — bug in the suite, com­pat­i­bil­i­ty report, or roadmap request. Report­ed bugs that get fixed become entries on the pub­lic changelog.

Include: the fail­ing test name, the full error out­put, your smoke.config.js (redact secrets), your Node ver­sion, and whether it’s local or CI. The com­mon first-wave fail­ures are already answered in Trou­bleshoot­ing — check there first.

License #

This suite is pro­vid­ed under the mem­ber license: active mem­bers may use it on unlim­it­ed projects includ­ing client work; no redis­tri­b­u­tion or resale; code already deployed to client repos keeps work­ing, but updates require an active mem­ber­ship.