# Requirements: QSP-39305-2 — E2E Test Mode Separation: Local Dev Server vs Deployed App

## Status

📋 Draft — awaiting approval

---

## Summary

Today all 10 E2E spec files run in an inconsistent mixed mode: `e2e.yml` unconditionally starts a local dev server by passing `START_DEV_SERVER` via `custom-secrets-config`, while 5 of the spec files additionally hardcode `FORCE_LOCAL_DEV_SERVER_IN_CI = 'true'` and `test.setTimeout(120_000)` per-file. The remaining 5 spec files have neither setting, yet they still run against a local server because of the workflow-level `START_DEV_SERVER`. This creates a setup where the CI workflow and individual spec files share responsibility for the same concern, making it unclear which tests require a local server and which do not.

This ticket introduces a clean, complete separation. A new `unit-integration-tests.yml` workflow runs all 10 spec files against the local dev server — with the 120 s timeout and `FORCE_LOCAL_DEV_SERVER_IN_CI` configured centrally via a dedicated Playwright project (`local-dev`) and a workflow-level `env:` variable. The existing `e2e.yml` is updated to remove `START_DEV_SERVER` so it runs all 10 spec files exclusively against the deployed app. The per-file `test.setTimeout(120_000)`, `process.env.FORCE_LOCAL_DEV_SERVER_IN_CI = 'true'`, and `test.afterAll` cleanup blocks are removed from the 5 spec files that currently contain them.

---

## Acceptance Criteria

- [ ] AC-1: `playwright.config.ts` contains a Playwright project named `local-dev` with `timeout: 120_000`, using Desktop Chrome, `storageState`, and `dependencies: ['setup']`. It inherits the global `testDir: 'e2e'` and `testMatch: /\.spec\.ts$/` so it targets all 10 spec files.
- [ ] AC-2: `package.json` contains a `test:unit-integration` script that runs `playwright test --project=local-dev`.
- [ ] AC-3: `.github/workflows/unit-integration-tests.yml` exists, is triggered on `schedule: cron: '0 0 * * *'` and `workflow_dispatch`, passes `START_DEV_SERVER` via `custom-secrets-config`, sets `FORCE_LOCAL_DEV_SERVER_IN_CI: 'true'` as an environment variable, and runs the `test:unit-integration` script.
- [ ] AC-4: `.github/workflows/e2e.yml` no longer passes `START_DEV_SERVER` via `custom-secrets-config` — it runs all 10 spec files against the deployed app without starting a local server.
- [ ] AC-5: All 5 spec files that previously hardcoded `test.setTimeout(120_000)` have that line (and its accompanying comment) removed.
- [ ] AC-6: All 5 spec files that previously hardcoded `process.env.FORCE_LOCAL_DEV_SERVER_IN_CI = 'true'` in `beforeEach` have that line removed.
- [ ] AC-7: All 5 spec files that previously had `test.afterAll(() => { delete process.env.FORCE_LOCAL_DEV_SERVER_IN_CI; })` have that block removed.
- [ ] AC-8: After the cleanup, `grep -r "FORCE_LOCAL_DEV_SERVER_IN_CI" e2e/tests/` returns no results.
- [ ] AC-9: All test logic (assertions, interactions, page-object usage) in all 10 spec files remains unchanged.

---

## Key Decisions

| # | Decision | Rationale |
|---|---|---|
| D-1 | New workflow triggers: `schedule: cron: '0 0 * * *'` + `workflow_dispatch` | Same cadence as `e2e.yml` — keeps both suites in sync; on-demand dispatch retained for manual reruns |
| D-2 | `START_DEV_SERVER` removed from `e2e.yml` | The deployed-app workflow must never start a local server; without this env var the `webServer` block in `playwright.config.ts` does not activate |
| D-3 | `FORCE_LOCAL_DEV_SERVER_IN_CI=true` set as `env:` at workflow level in new workflow | It is not a vault secret — it is a plain feature flag; placing it in `env:` makes the intent explicit and avoids unnecessary vault lookups |
| D-4 | New npm script named `test:unit-integration` (not `test:e2e-local`) | Communicates the purpose — integration tests that exercise the app against a local dev server — and avoids confusion with the existing `test:e2e` family |
| D-5 | Central 120 s timeout via a Playwright project named `local-dev` with `timeout: 120_000` | Eliminates per-file `test.setTimeout(120_000)` duplication; the project-level override is the correct Playwright mechanism for project-wide timeouts |
| D-6 | All 10 spec files run in both modes (local dev server and deployed app) | Uniform coverage across both workflows; no test is exclusive to one mode |

---

## Data Model

Not applicable — this ticket covers test infrastructure and CI workflow configuration only. No TypeScript interfaces, domain types, or Grail event schemas are modified.

---

## UX / UI Specification

Not applicable — no user-facing changes.

---

## API Contracts

Not applicable.

---

## DQL Queries

Not applicable.

---

## Implementation Plan

### Consumer Impact Analysis

No TypeScript interfaces or data models are modified. Changes are isolated to the files listed below. No consumers need to be traced.

| File changed | Reason |
|---|---|
| `playwright.config.ts` | Add `local-dev` Playwright project |
| `package.json` | Add `test:unit-integration` script |
| `.github/workflows/e2e.yml` | Remove `START_DEV_SERVER` from `custom-secrets-config` |
| `.github/workflows/unit-integration-tests.yml` | New workflow (create) |
| `e2e/tests/app-landing-page/app-header-navigation.spec.ts` | Remove per-file env/timeout settings |
| `e2e/tests/components-page/components-page-elements.spec.ts` | Remove per-file env/timeout settings |
| `e2e/tests/components-page/datatable-search.spec.ts` | Remove per-file env/timeout settings |
| `e2e/tests/overview-page/overview-page-elements.spec.ts` | Remove per-file env/timeout settings |
| `e2e/tests/app-score-page/app-score-page-signals.spec.ts` | Remove per-file env/timeout settings |

---

### Step 1 — Add `local-dev` project to `playwright.config.ts`

- **File**: `playwright.config.ts`
- **Changes**: Add a new entry to the `projects` array inside `stableConfig`:

```typescript
{
  name: 'local-dev',
  timeout: 120_000,
  use: {
    ...devices['Desktop Chrome'],
    viewport: viewportSize,
    storageState: STORAGE_STATE_PATH(),
  },
  dependencies: ['setup'],
},
```

The project inherits the global `testDir: 'e2e'` and `testMatch: /\.spec\.ts$/` without override, so it targets all 10 spec files. `timeout: 120_000` overrides the global `timeout: 60_000` for this project only. Auth setup runs first because of `dependencies: ['setup']`.

- **Verification**: Run `npx playwright test --project=local-dev --list` — should list exactly the same 10 spec files as `--project=chromium`.

---

### Step 2 — Add `test:unit-integration` script to `package.json`

- **File**: `package.json`
- **Changes**: Add to the `scripts` block (after `test:e2e-ci`):

```json
"test:unit-integration": "playwright test --project=local-dev"
```

- **Verification**: `npm run test:unit-integration -- --list` lists all 10 spec files. Running `npm run typecheck` continues to pass (no TS changes).

---

### Step 3 — Update `.github/workflows/e2e.yml` — remove `START_DEV_SERVER`

- **File**: `.github/workflows/e2e.yml`
- **Changes**: Remove the `with: custom-secrets-config: | START_DEV_SERVER` block entirely. The reusable workflow is called without any `with:` inputs.

Expected result after change:

```yaml
jobs:
  e2e:
    uses: dynatrace-apps/actions/.github/workflows/e2e.yml@v0
```

- **Why**: Without `START_DEV_SERVER` in the environment, the `webServer` condition in `playwright.config.ts` (`process.env.CI && process.env.START_DEV_SERVER`) evaluates to `false` and no local server is started. All 10 tests run against the deployed app URL.
- **Verification**: Confirm the `with:` block is absent and the reusable workflow call has no inputs.

---

### Step 4 — Create `.github/workflows/unit-integration-tests.yml`

- **File**: `.github/workflows/unit-integration-tests.yml` (new file)
- **Changes**: New workflow that mirrors `e2e.yml`'s triggers but adds `START_DEV_SERVER` and `FORCE_LOCAL_DEV_SERVER_IN_CI`, and runs `test:unit-integration`.

Intended shape:

```yaml
on:
  schedule:
    - cron: '0 0 * * *'
  workflow_dispatch:

jobs:
  unit-integration:
    uses: dynatrace-apps/actions/.github/workflows/e2e.yml@v0
    with:
      custom-secrets-config: |
        START_DEV_SERVER
      env: |
        FORCE_LOCAL_DEV_SERVER_IN_CI=true
```

> **Implementation note for the Coding Agent**: GitHub Actions does not support a top-level `env:` on a job that uses `uses:` (reusable workflow call). If the `dynatrace-apps/actions` reusable workflow does not expose an `env` or `extra-env` input, `FORCE_LOCAL_DEV_SERVER_IN_CI` must be passed via `custom-secrets-config` instead (requires a repository secret `FORCE_LOCAL_DEV_SERVER_IN_CI` with value `true`). The Coding Agent must inspect the reusable workflow's accepted inputs at `dynatrace-apps/actions/.github/workflows/e2e.yml` before choosing the mechanism. If neither works, the job must be inlined (not using `uses:`).

> **Implementation note on test command**: The reusable workflow from `dynatrace-apps/actions` may run a fixed npm script (e.g. `test:e2e-ci`). If it does not accept a script override input, the `test:unit-integration` script cannot be called and the `--project=local-dev` flag must be passed another way. The Coding Agent must verify the reusable workflow's inputs before implementing.

- **Verification**: Workflow file is valid YAML. `START_DEV_SERVER` activates `webServer` in `playwright.config.ts`. `FORCE_LOCAL_DEV_SERVER_IN_CI=true` is visible in the test process environment.

---

### Step 5 — Remove per-file env/timeout settings from 5 spec files

Each of the following 5 files contains exactly three items to remove. No other lines in these files change.

| File | Items to remove |
|---|---|
| `e2e/tests/app-landing-page/app-header-navigation.spec.ts` | `test.setTimeout(120_000)` + its comment, `process.env.FORCE_LOCAL_DEV_SERVER_IN_CI = 'true'` in `beforeEach`, `test.afterAll` cleanup block |
| `e2e/tests/components-page/components-page-elements.spec.ts` | Same three removals |
| `e2e/tests/components-page/datatable-search.spec.ts` | Same three removals |
| `e2e/tests/overview-page/overview-page-elements.spec.ts` | Same three removals |
| `e2e/tests/app-score-page/app-score-page-signals.spec.ts` | Same three removals |

The exact patterns to remove from each file:

```typescript
// Remove this line at the top of the file:
test.setTimeout(120_000); //Needed as start-up of local-dev-server takes ~20s

// Remove this line from beforeEach:
process.env.FORCE_LOCAL_DEV_SERVER_IN_CI = 'true';

// Remove this entire block:
test.afterAll(() => {
  delete process.env.FORCE_LOCAL_DEV_SERVER_IN_CI;
});
```

- **Verification**: `grep -r "FORCE_LOCAL_DEV_SERVER_IN_CI" e2e/tests/` returns no output. `grep -r "test.setTimeout" e2e/tests/` returns no output. All 10 spec files compile without errors (`npm run typecheck`).

---

## Unit Test Plan

Not applicable — this ticket modifies CI workflow configuration and Playwright project configuration only. No Jest unit tests are added or changed. Manual verification steps are specified inline in each Implementation Plan step above.

---

## Edge Cases & Error Handling

- **`webServer` not starting in the new workflow**: If `START_DEV_SERVER` is not correctly resolved from vault, the `webServer` block will not activate and all tests in the `unit-integration` run will fail with connection errors to `localhost:3000`. The failure is loud and self-explanatory; no silent fallback exists.
- **`FORCE_LOCAL_DEV_SERVER_IN_CI` not propagating into the reusable workflow**: GitHub Actions does not forward a top-level `env:` into a called reusable workflow. This is flagged as an implementation risk in Step 4; the Coding Agent must verify the mechanism before implementing.
- **5 spec files that have never run against the local dev server**: `basic-platform-test.spec.ts`, `overview-page-tests.spec.ts`, `overview-to-aggregation-navigation.spec.ts`, `overview-aggregation-page.spec.ts`, and `app-score-page-elements.spec.ts` have only ever run against the deployed app. When they run against the local dev server in the new workflow, they may encounter missing Grail data or timing differences. Initial failures are expected and should be treated as bugs to fix in follow-up tickets.
- **Auth setup against local dev server**: The `setup` project (auth-setup.ts) authenticates against the Dynatrace platform, not the local dev server. It must complete successfully before `local-dev` tests start. Platform auth is still required even when `FORCE_LOCAL_DEV_SERVER_IN_CI=true` — the env var only affects `platformDtPage.openAppViaUrl`.
- **Comment co-location**: The `test.setTimeout(120_000)` line in each spec file has a trailing comment `//Needed as start-up of local-dev-server takes ~20s`. The comment must be removed together with the line, not left behind as an orphaned comment.

---

## Out of Scope

- Running either workflow on pull requests — both `e2e.yml` and `unit-integration-tests.yml` are scheduled/on-demand workflows; PR-based E2E is handled by `pr-check.yml` and is not modified.
- Migrating the `integration-tests/` Jest suite — `test:it-ci` and `integration-tests/tests/dummy.test.ts` are unchanged.
- Adding a `TestTag.LOCAL_DEV` or any new tag to spec files — project-level Playwright separation is sufficient; no new tags are introduced.
- Firefox support in the `local-dev` project — only Chromium is used, matching the developer `test:e2e` workflow.
- Fixing test failures that emerge when the 5 previously-deployed-app-only tests first run against the local dev server — those are follow-up bugs.
- Changes to `playwright-ct.config.ts`, `playwright-a11y.config.ts`, or component/unit test infrastructure.
- Version pinning of the `dynatrace-apps/actions` reusable workflow — the existing `@v0` reference is unchanged.

---

## Dependencies / Risks

| # | Risk | Likelihood | Mitigation |
|---|---|---|---|
| R-1 | `FORCE_LOCAL_DEV_SERVER_IN_CI` cannot be set as plain `env:` in a `uses:` reusable workflow job (GitHub Actions limitation) | High | Pass via `custom-secrets-config` as a repo secret with value `true`, or verify whether the reusable workflow exposes an env/extra-env input |
| R-2 | The reusable workflow from `dynatrace-apps/actions` does not support a script override input, preventing `test:unit-integration` from being called | Medium | Verify the reusable workflow's inputs; if no override exists, inline the job steps instead of using `uses:` |
| R-3 | 5 spec files that were never tested against a local dev server may fail on first run | Medium | Accept initial failures; treat as bugs to investigate in follow-up tickets; do not block this ticket on them |
| R-4 | Auth `setup` project fails when `FORCE_LOCAL_DEV_SERVER_IN_CI=true` is set | Low | The setup project connects to the Dynatrace platform directly, not through localhost; the env var only affects `platformDtPage.openAppViaUrl` |

---

## Open Questions

None — all clarifying questions have been answered.

---

## Changelog

- **2026-03-26**: Requirements drafted. All clarifying questions answered. Spec complete — awaiting human approval.

---

## Refinement Log

### Step 1 — Code investigation

**Code investigation findings:**

- `playwright.config.ts`: `webServer` block only activates when **both** `CI=true` AND `START_DEV_SERVER` env var are set. Timeout is 60s, retries 2 on CI, 2 parallel workers on CI.
- `e2e.yml`: **currently** passes `START_DEV_SERVER` via `custom-secrets-config` — meaning ALL scheduled/manual e2e runs already start the local dev server. This is the source of the problem: the separation doesn't exist yet.
- `e2e/tests/` contains **10 spec files** across 5 directories:
  - **5 files WITH** `FORCE_LOCAL_DEV_SERVER_IN_CI` + `test.setTimeout(120_000)` (hardcoded per-test):
    - `app-landing-page/app-header-navigation.spec.ts`
    - `components-page/components-page-elements.spec.ts`
    - `components-page/datatable-search.spec.ts`
    - `overview-page/overview-page-elements.spec.ts`
    - `app-score-page/app-score-page-signals.spec.ts`
  - **5 files WITHOUT** these settings (purely deployed-app tests):
    - `app-landing-page/basic-platform-test.spec.ts`
    - `overview-page/overview-page-tests.spec.ts`
    - `overview-page/overview-to-aggregation-navigation.spec.ts`
    - `overview-aggregation/overview-aggregation-page.spec.ts`
    - `app-score-page/app-score-page-elements.spec.ts`
- `e2e/src/utils/metadata/test-tag.ts`: Existing tags are `EXAMPLE_TAG`, `E2E`, `COMPONENT`.
- `package.json` scripts: `test:e2e` → `playwright test --project=chromium`; `test:e2e-ci` → `playwright test`; `test:it-ci` → jest integration tests.
- No existing `unit-integration-tests.yml` workflow — only `e2e.yml`, `main.yml`, `pr-check.yml`.
- `integration-tests/tests/dummy.test.ts` — only a placeholder test exists there now.

**Questions asked:**

1. **Workflow trigger**: The new `unit-integration-tests.yml` workflow should run the local dev server + e2e tests. What should trigger it — on PR (`pull_request`) like `main.yml`, on a schedule like `e2e.yml`, or on-demand (`workflow_dispatch`)? Or all three?

2. **`e2e.yml` change**: Since the new workflow will handle the "local dev server + e2e" mode, should `e2e.yml` be updated to **remove** `START_DEV_SERVER` from `custom-secrets-config` so it runs purely against the deployed app without starting a local server?

3. **Setting `FORCE_LOCAL_DEV_SERVER_IN_CI`**: In the new workflow, should `FORCE_LOCAL_DEV_SERVER_IN_CI=true` be set as a plain `env:` variable in the YAML (it's not a secret — it's a feature flag), or does it also need to go through `custom-secrets-config`?

4. **New npm script**: Should a new script (e.g. `test:e2e-local`) be added to `package.json` for running e2e tests against the local dev server locally by a developer, or is just setting the env var and using `test:e2e` sufficient?

5. **Central timeout**: The `test.setTimeout(120_000)` should move out of individual test files. Should this be set centrally in `playwright.config.ts` (e.g. `timeout: 120_000` when `FORCE_LOCAL_DEV_SERVER_IN_CI=true`), or through a dedicated Playwright project?

6. **Scope of "all tests in both modes"**: The context says "execute all e2e tests once against the local and once against the deployed app." Does "all" literally mean **all 10 spec files**, or only the 5 currently using `FORCE_LOCAL_DEV_SERVER_IN_CI`? The 5 without it (e.g. `overview-page-tests.spec.ts`) query live Grail data — would they behave correctly against a local dev server with mocked data?


