Production API E2E Testing Discovery

Production API E2E Testing Discovery

Recommendation

Use TypeScript + Playwright Test.

For this system it is the best fit because:

• You already know Playwright and prefer TypeScript.
• Playwright's APIRequestContext is designed for direct REST testing, test-data setup, and checking API state after UI actions. See Playwright API testing.
• The same runner can hold API tests and a small number of real browser journeys.
• Projects let you run different subsets against sandbox and production with different configuration. See Playwright projects.
• It includes fixtures, parallel workers, retries, HTML/JUnit reports, traces, and good CI support.
• TypeScript is particularly convenient for generated OpenAPI types and AI-assisted maintenance.

Java with REST Assured would align with the Spring backend, and Python with pytest/httpx would also work, but neither gives you a meaningful advantage here. They would mostly introduce another testing ecosystem.

Exact Environments Discovered

The URL you want is:

• Mutation-friendly sandbox: https://aitesters.byst.re
• Stable production-like playground: https://awesome.byst.re

Do not use aitesters.awesome.byst.re; it is only an nginx alias and the repository warns that its nested hostname does not match the normal wildcard TLS certificate.

On 8 August 2026, login, OpenAPI, images, and health returned 200 on both environments.

Property	aitesters.byst.re	awesome.byst.re
Backend/frontend images	Same deployed images	Same deployed images
Database	In-memory H2	PostgreSQL
Email	Local test outbox	ActiveMQ + consumer
Ollama	Deterministic mock	Deterministic mock
Seeded demo data	Yes	Production bootstrap/data
Destructive tests	Intended use	Use controlled test identities
Reset	Daily backend recreation	Persistent state

The deployment details are documented in awesome-localstack/docs/PROFILE_URLS.md and awesome-localstack/docker-compose.server.yml. The sandbox-specific H2, seeding, local outbox, and disabled SSO configuration is in test-secure-backend/src/main/resources/application-aitesters.yml.

The sandbox reset is configured for 03:00 in the server's local timezone. The repository does not explicitly pin that timezone, so verify the VPS timezone and avoid scheduling tests around the reset window.

Important Architectural Conclusion

Testing only aitesters.byst.re does not prove the complete production infrastructure works.

It verifies:

• Public DNS/TLS/gateway routing
• Deployed frontend and backend images
• Authentication and application logic
• H2 persistence
• Ollama mock integration
• Test email outbox

It does not verify the actual PostgreSQL, ActiveMQ, and email-consumer path used by awesome.byst.re.

Therefore, use two test lanes:

1. Full destructive regression against aitesters.byst.re.
2. Small, carefully isolated canary suite against awesome.byst.re.

The repository already has shallow blackbox monitoring for login and OpenAPI every 30 seconds. The new canary should add authenticated business behavior, not merely duplicate those HTTP probes.

Current API Size

The live sandbox OpenAPI currently contains:

• 40 paths
• 51 operations
• 15 user operations
• 6 order operations
• 6 MFA operations
• 5 cart operations
• 5 product operations
• 4 Ollama operations
• Password reset, QR, email, traffic, and local-outbox operations

See the live sandbox OpenAPI.

One discovery worth noting: OpenAPI security annotations are incomplete. For example, the outbox appears public in the generated document, but its controller requires an admin JWT plus X-Local-Outbox-Key, and the live unauthenticated request correctly returns 401. Use OpenAPI for paths and types, but write an explicit authorization matrix rather than trusting its security metadata.

What to Test

1. Deployment Smoke Tests

Run on both environments:

• HTTPS, certificate, and gateway availability
• /login
• /v3/api-docs
• /actuator/health
• Representative static image
• Protected endpoint returns 401 without a token
• Valid login
• /users/me
• Product catalogue read
• One deterministic Ollama mock request

Approximately 8–12 tests.

2. Critical Sandbox Regression

Run on aitesters.byst.re:

• Signup -> signin -> /me
• Refresh-token rotation
• Reuse of an old refresh token is rejected
• Logout revokes refresh tokens
• Client versus admin authorization
• User editing and account deletion
• Product create/update/delete
• Cart add/update/remove/clear
• Order creation, retrieval, cancellation, status transitions, and stock effects
• Password reset through the local email outbox
• MFA setup, confirmation, MFA login, recovery code, and disable
• Chat/system-prompt isolation between users
• QR response content type and PNG signature
• Ollama generate/chat/tool calls and final SSE chunk
• Traffic-log redaction of tokens, passwords, and email data
• Representative validation, duplicate, not-found, 401, and 403 cases

Approximately 35–60 tests for a strong first version.

3. Production Canary

Against awesome.byst.re, use a uniquely named user for every run:

• Signup
• Signin
• /me
• Product read
• Add to cart
• Create and retrieve an order
• Verify an email event if available
• Delete the user through right-to-be-forgotten, which should also remove its cart, orders, tokens, and related data

Run this serially with one worker. Avoid admin catalogue mutation in the frequent canary unless you create and delete a uniquely named product within the same test.

4. Minimal Browser E2E

Keep the UI suite deliberately small:

• Password login
• Browse -> cart -> order
• One Ollama/chat journey
• Possibly password reset if the complete frontend flow is important

Chromium alone is sufficient initially. Cross-browser testing adds cost but little value to an API-focused project.

SSO is disabled in both current public server profiles, so positive SSO E2E belongs in the local Keycloak stack, not the public production suite.

Suggested Repository Layout

Because the workspace root is an aggregator rather than a Git repository, create a separate sibling repository, for example test-secure-e2e, rather than place the tests inside the backend:

test-secure-e2e/
  package.json
  package-lock.json
  playwright.config.ts
  .env.example
  fixtures/
    auth.fixture.ts
    test-user.fixture.ts
    admin.fixture.ts
  clients/
    auth.client.ts
    products.client.ts
    cart.client.ts
    orders.client.ts
  support/
    test-data.ts
    sse.ts
    cleanup.ts
  tests/
    smoke/
    api/
      auth/
      users/
      products/
      cart/
      orders/
      mfa/
      password-reset/
      ollama/
    ui/

Use small domain clients and fixtures rather than page-object-style abstractions for API calls.

Recommended dependencies:

• @playwright/test
• typescript
• zod for important runtime response validation
• otplib for MFA/TOTP
• eventsource-parser or a small internal SSE parser
• Optionally openapi-typescript for compile-time request/response types

For ordinary HTTP calls, use Playwright's request fixture. For SSE endpoints, native Node fetch() is likely more convenient because it exposes the response stream directly.

Test-Data Strategy

This is the part that determines whether the suite becomes reliable.

• Generate a unique username, email, product name, and run ID from testInfo.testId, timestamp, and worker index.
• Never mutate shared client, client2, or client3 accounts.
• Use the seeded admin only to create and clean up unique records.
• Every destructive test should clean up what it owns in finally or fixture teardown.
• Cleanup should tolerate 404, because a daily reset might already have removed the data.
• Start with one or two workers. Playwright explicitly recommends unique backend data for parallel tests. See Playwright parallelism and isolation.
• Keep rate-limit tests in a separate serial suite; otherwise they can break unrelated authentication tests.
• Do not depend on fixed product or order IDs after other tests have mutated the sandbox.

Secrets such as the admin password and local-outbox key should come from CI secrets, even where a demo credential is currently documented. Also be careful with traces and reports: JWTs, reset links, and email bodies may appear in network artifacts.

CI Schedule

A production suite should test the code that is actually deployed. Running it on every backend pull request against the public URL would only test the previous deployment.

Recommended schedule:

• Pull request: local/lightweight compatibility tests against the candidate code.
• Immediately after deployment: full sandbox regression.
• Every 15–60 minutes: small production canary.
• Nightly: MFA, password reset, streaming, authorization matrix, and slower negative tests.
• Manually: rate-limit, concurrency, and aggressive mutation scenarios.

Initially use no retries for API regression so intermittent failures remain visible. Once isolation is proven, one diagnostic retry can be enabled, but flaky-on-retry should still be treated as a CI problem. Playwright otherwise categorizes such tests as flaky rather than failed. See Playwright retries.

Difficulty and Realistic Effort

This is not difficult to start, but moderately difficult to make genuinely production-grade.

Area	Difficulty
CRUD and response assertions	Easy
JWT fixtures and role separation	Medium
Parallel data isolation and cleanup	Medium
Orders and stock transitions	Medium
Password reset through outbox	Medium-hard
MFA/TOTP and recovery codes	Hard
SSE and tool-calling responses	Hard
Reliable scheduled production execution	Medium-hard

A realistic estimate:

• Useful 10–15-test smoke suite: 1–2 days
• Critical API regression: 3–5 additional days
• MFA, password reset, SSE, schema checks, and CI hardening: 3–6 additional days
• Mature first release: roughly 1–2 working weeks

Codex can generate most of the mechanics quickly. The time-consuming work will be discovering actual response semantics, eliminating shared-state races, and making failures diagnostically useful.

One final repository note: the backend already contains 433 local @Test methods. The new suite should emphasize deployment configuration, authentication boundaries, integration wiring, and full business journeys rather than reproducing every Java unit-test edge case. The backend instructions also reference an external lesson-12 compatibility repository, but that URL returned 404 from this environment; its current location or access should be resolved before depending on it. The backend README additionally references a sibling playwright-2025 project that is not present in this workspace, so it may be worth locating and reusing before scaffolding from zero.

Final Choice

Create a dedicated TypeScript Playwright repository, run the full destructive API regression on aitesters.byst.re, run a thin serial canary on awesome.byst.re, and keep only 3–5 browser tests.