Raport pokrycia i statusu dla zespołu

L18 API Coverage & Quality Status

Last updated: 2026-08-30

Audience: developers, QA, and technical leads

Live Swagger audited: https://aitesters.byst.re/swagger-ui/index.html

Authoritative course API: https://awesome.byst.re

This is the single current developer report for L18. It replaces the former active/history test plans. Detailed defect evidence lives in bug-reports/; stale implementation chronology belongs in git history.

Stakeholder view: api-coverage-report.html

Current Status

Signal	Result
Live route templates with automation	38 / 44 (86.4%)
Live endpoint operations with automation	48 / 55 (87.3%)
Completely unautomated operations	7
Operations with every documented status automated	25 / 55 (45.5%)
Documented response paths automated	141 / 191 (73.8%)
Partially covered operations	23
Missing documented response paths	50
Automated API checks	170
Playwright spec files	53
Non-admin checks against the course API	129
Admin checks against staging	41
Latest full L18 run	170 passed
Confirmed open backend/contract defects	3

Assessment: the customer and admin regression suite is green. All six 2FA operations now have endpoint-focused client automation against the authoritative course API. Seven operations remain completely unautomated and 23 more are only partially covered at the documented response-code level. Inventory, SSO, and the local outbox are the remaining material breadth gaps.

For the course-authoritative awesome.byst.re contract, which omits the two local outbox operations, the equivalent figures are 38 / 43 route templates (88.4%), 48 / 53 operations (90.6%), and 141 / 188 response paths (75.0%).

2FA Automated Coverage

All six 2FA operations have endpoint-focused Playwright coverage in the regular api project against the course-authoritative https://awesome.byst.re. No 2FA spec is placed under tests/api/admin, and the tests do not target the staging base URL.

• 28 checks in six spec files: status, setup, confirmation, second-factor sign-in, recovery-code replacement, and disable.
• Lifecycle coverage: disabled/enabled status, Base32 enrollment secret, OTPAuth URI and PNG QR structure, TOTP confirmation, password challenge without premature tokens, TOTP and recovery-code sign-in, challenge replay prevention, recovery-code rotation, old/invalid factor rejection, disable, and restored password-only sign-in.
• Negative coverage: stable 400, 401, and 409 paths, including missing fields, missing authentication, missing setup, enabled/not-enabled conflicts, TOTP replay, unknown challenge, consumed challenge, and invalid recovery code.
• Implementation: typed MfaClient and DTOs, exact-pinned otplib@13.5.0, a 30-second-step-aware TOTP helper, structural MFA assertions, and an isolated disposable enrolled-user fixture. HTTP clients are initialized in test.beforeEach and status groups are ordered ascending.
• Data safety: every stateful check owns a generated user. Account deletion removes MFA credentials, challenges, and recovery codes. MFA specs disable Playwright tracing because setup and confirmation responses contain secrets.
• Timing: stateful MFA specs allow 90 seconds because the remote signup, signin, enrollment, and teardown calls take materially longer than local tests; fresh-TOTP checks intentionally wait for a later 30-second step to respect replay prevention.

The checked-in repository api-docs.json remains stale at 42 operations and has no 2FA paths, so live awesome.byst.re/v3/api-docs is the contract source. The live error bodies still do not match their generated OpenAPI schemas; behavioral tests assert verified runtime bodies while the contract defect remains tracked in bug-reports/2fa-openapi-error-schemas-use-success-dtos.md.

The reusable curl explorer remains at scripts/explore-2fa-poc.sh, defaults to awesome.byst.re, redacts sensitive output, and deletes its generated account.

What “Covered” Means

An endpoint operation is one HTTP method + route template. It counts as covered when it has a dedicated Playwright regression spec for verified live behavior, including its primary success or stable negative path. Route-template coverage is also reported separately so a multi-method route cannot inflate the result.

Operation coverage does not mean every documented status or edge case is safe to assert. The coverage exceptions table is therefore part of the status, not a footnote.

Response-path coverage compares the distinct documented status codes for every method + path with status groups present in endpoint-focused specs. Multiple test scenarios for the same status count once in this metric.

Endpoints Without Automated Coverage

These seven live Swagger operations have no endpoint-focused L18 automation:

Endpoint operation	Documented statuses	Gap / recommended next action
GET /api/v1/admin/inventory	200, 401, 403	Add admin inventory list, auth, and authorization coverage
GET /api/v1/admin/inventory/{productId}	200, 401, 403	Add generated-product detail coverage and cleanup
GET /api/v1/admin/inventory/{productId}/movements	200, 401, 403	Add movement-history coverage around a controlled adjustment
POST /api/v1/admin/inventory/{productId}/adjustments	201, 401, 403	Add reversible adjustment coverage with before/after verification
POST /api/v1/users/sso/exchange	200, 400, 401, 404, 409	Define a test identity-provider token strategy, then automate
GET /api/v1/local/email/outbox	200	aitesters-only: confirm intended test scope and key handling
DELETE /api/v1/local/email/outbox	200, 500	aitesters-only: isolate outbox state before adding a destructive check

Partially Covered Endpoints

These 23 operations have automation, but not for every live documented status:

Endpoint operation	Automated statuses	Missing documented statuses	Why missing	Best next action
DELETE /api/v1/cart/items/{productId}	200, 401, 404	400	Added after the repository contract snapshot	Explore malformed product-id behavior and add the stable case
DELETE /api/v1/products/{id}	204, 401, 403, 404	400	Added after the repository contract snapshot	Add malformed-id admin coverage
GET /api/v1/orders/{id}	200, 401, 404	400	Added after the repository contract snapshot	Add malformed-id customer coverage
GET /api/v1/products	200, 401	404	Contract now documents an unusual collection 404	Reproduce live and align contract or add a stable scenario
GET /api/v1/products/{id}	200, 401, 404	400	Added after the repository contract snapshot	Add malformed-id coverage
GET /api/v1/traffic/logs	200	400	Added after the repository contract snapshot	Identify and automate the documented invalid-query trigger
POST /api/v1/products	201, 400, 401, 403	404	Contract now documents an unclear create 404	Reproduce live and clarify the missing dependency/resource case
POST /api/v1/users/signup	201, 400	429	Shared rate-limit exhaustion is stateful	Add an isolated rate-limit environment or reset hook
POST /api/v1/users/signin	200, 400, 422	429	Shared rate-limit exhaustion is stateful	Add deterministic limiter reset/control
POST /api/v1/users/signin/2fa	200, 400, 401	429	Exhausting the shared MFA IP bucket would destabilize sign-in coverage	Add an isolated limiter scope or deterministic reset hook
POST /api/v1/users/2fa/setup	200, 401, 409	429	Exhaustion is stateful and shares the user's MFA management bucket	Add a deterministic limiter reset/control
POST /api/v1/users/2fa/confirm	200, 400, 401, 409	410, 429	Remote setup expiry requires a 15-minute wait; exhaustion is stateful	Add backend clock and limiter controls, then cover both paths
POST /api/v1/users/2fa/recovery-codes	200, 400, 401, 409	429	Exhaustion is stateful and would poison the disposable user's bucket	Add a deterministic limiter reset/control
POST /api/v1/users/2fa/disable	200, 400, 401, 409	429	Exhaustion is stateful and would obstruct guaranteed teardown	Add a deterministic limiter reset/control
POST /api/v1/users/refresh	200, 400, 401	429	Shared rate-limit exhaustion is stateful	Add deterministic limiter reset/control
POST /api/v1/users/password/forgot	202, 400	429	Shared rate-limit exhaustion is stateful	Test in an isolated limiter scope
POST /api/v1/users/password/reset	400	200, 429	Success token is not observable; limiter path is stateful	Expose a safe token/outbox hook, then isolate rate-limit coverage
POST /api/v1/email	200, 400, 401	429	Exhaustion would poison the disposable user's bucket	Add deterministic limiter reset/control
POST /api/v1/qr/create	200, 400, 401	429	Shared rate-limit exhaustion is stateful	Add deterministic limiter reset/control
POST /api/v1/ollama/generate	200, 400, 401	404, 429, 500	Mock returns 200 for unknown model; limiter and 500 need safe controls	Align mock/contract; add limiter control and backend fault injection
POST /api/v1/ollama/chat	200, 400, 401	404, 429, 500	Mock returns 200 for unknown model; limiter and 500 need safe controls	Align mock/contract; add limiter control and backend fault injection
POST /api/v1/ollama/chat/tools	200, 400, 401	429, 500	Limiter and internal failure lack safe deterministic triggers	Add limiter control and backend fault-injection coverage
GET /api/v1/traffic/logs/{correlationId}	404	200	Live existing-record request returns confirmed 500	Fix backend, deploy, reverify, then add 200 test

The 50 missing response paths break down into:

• 20 paths belong to the seven completely unautomated operations;
• 10 paths were added to already automated operations after the repository OpenAPI snapshot, including three Ollama 429 paths;
• 14 paths were already known gaps in the former 42-operation baseline;
• 6 paths are intentionally deferred 2FA setup-expiry and rate-limit paths.

Six behaviors previously described as “live-only” are now documented by the live Swagger: 400 for customer/admin order filters, refresh-token validation, user update validation, and both prompt-update length validations. This is why the live re-audit, rather than the checked-in api-docs.json, is the metric baseline.

Coverage By Domain

Domain	OpenAPI operations	Automated checks	Status	Important scope
Identity, users, prompts	23 / 24	83	Gap: 1 operation	2FA is automated; SSO exchange remains unautomated
Inventory	0 / 4	0	Uncovered	Admin list, detail, adjustment, and movement history
Products	5 / 5	18	Covered	Authenticated reads and admin create/update/delete
Cart	5 / 5	16	Covered	Empty/populated state, add/update/remove/clear, validation and auth
Orders	6 / 6	31	Covered	Customer and admin flows, ownership, cancellation, status transitions
Email delivery	1 / 1	3	Covered with exclusion	Success, validation, auth; rate-limit exhaustion excluded
Local email outbox	0 / 2	0	Uncovered on aitesters	Confirm test-only scope before automation
Ollama	4 / 4	11	Covered with discrepancy	Definitions and deterministic SSE generate/chat/tool flows
QR	1 / 1	3	Covered	PNG response, validation, auth
Traffic	3 / 3	3	Degraded	Info/list covered; existing correlation detail success blocked by backend 500
OpenAPI and Swagger UI	—	2	Covered	Contract and documentation smoke checks
Total	48 / 55	170	Green suite; breadth gaps open	Two Playwright projects, fully parallel

Validation Evidence

Latest L18 validation on 2026-08-30:

• Playwright discovery: 170 tests in 53 files.
• Focused awesome-only 2FA run: 28 passed in 1.4 minutes.
• Full network-enabled npm test: 170 passed across api and admin-api in 3.2 minutes.
• No credentials, local environment files, node_modules, or test artifacts were promoted into L18.

Validation commands:

cd l18
npm test
npm run test:api
npm run test:admin

Confirmed Backend Defects

1. Traffic log detail returns 500

• Endpoint: GET /api/v1/traffic/logs/{correlationId}
• Expected: documented 200 for an existing, freshly listed correlation id.
• Actual: reproducible 500 { "message": "Internal server error" }.
• Impact: diagnostic consumers cannot open a log entry returned by the list API; the documented success path remains blocked from regression coverage.
• Report: bug-reports/traffic-log-detail-returns-500.md

2. Repeated account-forgetting request returns 500

• Endpoint: DELETE /api/v1/users/{username}/right-to-be-forgotten
• Sequence: first self-delete returns 204; retrying with the deleted user's JWT returns 500.
• Reproduction: 3 / 3 direct curl attempts plus one fixture-teardown signal.
• Impact: safe client retry/cleanup looks like an infrastructure failure after the account was already removed.
• Report: bug-reports/right-to-be-forgotten-repeat-returns-500.md

3. 2FA OpenAPI error schemas use success DTOs

• Scope: all six 2FA operations on both live deployments.
• Expected: documented 400 and 401 response content matches the runtime validation-map and {message} error shapes.
• Actual: five operations reference their success DTO for non-success responses; POST /2fa/disable declares no response schemas at all.
• Reproduction: unauthenticated status was 3 / 3 on each environment with 401 {"message":"Unauthorized"}; empty 2FA sign-in was 1 / 1 on each with a 400 field-validation map.
• Impact: generated clients and schema-driven tests cannot trust the published 2FA error content.
• Report: bug-reports/2fa-openapi-error-schemas-use-success-dtos.md

The Playwright suite asserts intended behavior and does not encode any of these defects as accepted regression results.

Coverage Constraints And Contract Discrepancies

Area	Current evidence	Coverage decision	Unblock condition
Password reset success	Authoritative deployment returns token=null and exposes no documented outbox/token read path	Keep verified 400 cases; do not invent a 200 flow	Provide a safe test token or local outbox visibility
Traffic detail 200	Freshly listed ids return 500	Keep stable unknown-id 404; success remains blocked	Backend fix deployed and live behavior reverified
Email 429	Requires exhausting a per-user rate-limit bucket	Excluded from routine regression to avoid poisoning state and unnecessary load	Isolated rate-limit environment or deterministic reset hook
Admin order unknown status	Live staging exploration returned 401 despite a valid admin JWT; adjacent valid requests succeeded	Use verified invalid-transition 400; reverify before formalizing	Reproduce safely and file/update a backend report if still present
Ollama unknown model	Controlled mock returns 200 while OpenAPI documents 404	Do not assert the documented 404	Align mock behavior or update the published contract
2FA error schemas	Live 400 validation and 401 error bodies do not match the published success DTO references	Assert verified runtime bodies; do not validate errors against the incorrect OpenAPI schema	Publish explicit common error and validation schemas, then add contract assertions

Test Architecture

• api project: non-admin specs against API_BASE_URL, defaulting to https://awesome.byst.re.
• admin-api project: tests/api/admin/** against API_ADMIN_BASE_URL, defaulting to https://aitesters.byst.re.
• Projects are independent and run in parallel.
• All 2FA specs belong to the regular api project and therefore use awesome.byst.re; none use the admin/staging project.
• HTTP clients are initialized in test.beforeEach.
• Specs use // given, // when, // then and order cases by ascending response status.
• Dynamic ids, timestamps, pagination, traffic, and streams use structural assertions; stable request-derived fields use exact assertions.
• Ollama responses use an SSE parser and deterministic mock prompts.
• MFA uses exact-pinned otplib, fresh-step generation, per-test disposable users, structural secret assertions, and tracing disabled for secret-bearing specs.

Data Safety And Cleanup

• Admin mutations use generated disposable products and users; seeded entities are never deleted.
• Customer order fixtures cancel cancellable orders before cleanup to restore inventory.
• Admin order fixtures delete the generated owner/order before the generated product and verify cleanup statuses.
• Right-to-be-forgotten coverage uses a dedicated generated-user fixture that tolerates intentional deletion while preserving primary setup/test failures.
• MFA coverage uses isolated generated users; right-to-be-forgotten teardown removes credentials, challenges, and recovery codes after each stateful check.
• Exploratory calls use generated identifiers and record cleanup outcomes.
• Credentials are loaded only from ignored local environment files.

Recent L18 Increment

• Added 28 awesome-only client checks for all six 2FA operations, including TOTP and recovery-code sign-in, rotation, replay prevention, disable, and cleanup.
• Added typed MFA client/DTO/fixture/assertion layers and exact-pinned otplib@13.5.0; secret-bearing specs disable Playwright traces.
• Kept the redacted awesome-default curl explorer and documented the OpenAPI 2FA error-schema defect without encoding incorrect schemas as expectations.
• Added endpoint-focused coverage for POST /api/v1/email: 200, 400, 401.
• Added endpoint-focused coverage for right-to-be-forgotten: 204, 401, 403, 404, including observable post-deletion signin failure.
• Hardened generated-user cleanup after independent review.
• Completed e-commerce operation coverage: Products 5/5, Cart 5/5, Orders 6/6.
• Maintained evidence for the three open backend/contract defect reports above.
• Restored explicit // given phases in legacy specs found by the style audit.
• Re-audited against both live Swagger contracts and corrected the stale 42-operation baseline to 55 operations on aitesters and 53 on the course API.

Next Actions

1. QA automation: add the four inventory operations using a generated product and a reversible adjustment with verified cleanup.
2. QA + identity: define a controlled identity-provider token strategy for POST /api/v1/users/sso/exchange, then automate its stable response paths.
3. Backend/testability: expose clock and rate-limit controls for 2FA 410 and 429 coverage without long waits or shared-bucket exhaustion.
4. Backend: fix traffic correlation-detail retrieval, deploy, reverify with list-to-detail curl, then add the documented 200 Playwright path.
5. Backend/security: map a JWT for a deleted principal to a handled 401 or documented 404; add a backend retry regression test.
6. Testability: expose a safe password-reset token/outbox mechanism and add the success flow without weakening anti-enumeration behavior.
7. Contract: reverify the ten newly documented statuses on existing operations, unknown admin order status, and Ollama missing-model behavior.
8. Scope decision: decide whether the two aitesters-only local-outbox operations belong in L18 automation; if yes, add isolated key-protected checks.
9. Maintenance: refresh the checked-in contract baseline and this report after every API surface change.

Reporting Rule

Every unexpected 5xx, contract mismatch, auth/validation inconsistency, cleanup failure, or flaky response is a defect signal. Preserve evidence, attempt up to three safe isolated reproductions, create or update a Markdown report under bug-reports/, link it here, and avoid turning broken behavior into a regression expectation.