Raport pokrycia i statusu dla zespołu

L17 API Coverage & Quality Status

Last updated: 2026-08-30

Audience: developers, QA, and technical leads

Live Swagger audited: https://aitesters.byst.re/swagger-ui/index.html

Authoritative course API: https://awesome.byst.re

This is the single current developer report for L17. It replaces the former active/history test plans. Detailed defect evidence lives in bug-reports/; stale implementation chronology belongs in git history.

Stakeholder view: api-coverage-report.html

Current Status

Signal	Result
Live route templates with automation	32 / 44 (72.7%)
Live endpoint operations with automation	42 / 55 (76.4%)
Completely unautomated operations	13
Operations with every documented status automated	24 / 55 (43.6%)
Documented response paths automated	121 / 191 (63.4%)
Partially covered operations	18
Missing documented response paths	70
Automated API checks	142
Playwright spec files	47
Non-admin checks against the course API	101
Admin checks against staging	41
Latest full L17 run	142 passed
Confirmed open backend defects	2

Assessment: the existing customer and admin regression suite is green, but its contract baseline was stale. The live Swagger now contains 55 operations across 44 route templates. Thirteen operations have no endpoint-focused automation and 18 more are only partially covered at the documented response-code level. Inventory, 2FA, SSO, and the local outbox are the material breadth gaps.

For the course-authoritative awesome.byst.re contract, which omits the two local outbox operations, the equivalent figures are 32 / 43 route templates (74.4%), 42 / 53 operations (79.2%), and 121 / 188 response paths (64.4%).

What “Covered” Means

An endpoint operation is one HTTP method + route template. It counts as covered when it has a dedicated Playwright regression spec for verified live behavior, including its primary success or stable negative path. Route-template coverage is also reported separately so a multi-method route cannot inflate the result.

Operation coverage does not mean every documented status or edge case is safe to assert. The coverage exceptions table is therefore part of the status, not a footnote.

Response-path coverage compares the distinct documented status codes for every method + path with status groups present in endpoint-focused specs. Multiple test scenarios for the same status count once in this metric.

Endpoints Without Automated Coverage

These 13 live Swagger operations have no endpoint-focused L17 automation:

Endpoint operation	Documented statuses	Gap / recommended next action
GET /api/v1/admin/inventory	200, 401, 403	Add admin inventory list, auth, and authorization coverage
GET /api/v1/admin/inventory/{productId}	200, 401, 403	Add generated-product detail coverage and cleanup
GET /api/v1/admin/inventory/{productId}/movements	200, 401, 403	Add movement-history coverage around a controlled adjustment
POST /api/v1/admin/inventory/{productId}/adjustments	201, 401, 403	Add reversible adjustment coverage with before/after verification
GET /api/v1/users/2fa/status	200, 401	Add authenticated status and unauthenticated coverage
POST /api/v1/users/2fa/setup	200, 401, 409, 429	Add disposable-user setup flow; isolate limiter state
POST /api/v1/users/2fa/confirm	200, 400, 401, 409, 410, 429	Add TOTP confirmation flow with observable setup secret
POST /api/v1/users/2fa/disable	200, 400, 401, 409, 429	Add confirmed-2FA teardown flow and guaranteed cleanup
POST /api/v1/users/2fa/recovery-codes	200, 400, 401, 409, 429	Add recovery-code lifecycle coverage without logging secrets
POST /api/v1/users/signin/2fa	200, 400, 401, 429	Add the second-stage signin flow using a disposable 2FA user
POST /api/v1/users/sso/exchange	200, 400, 401, 404, 409	Define a test identity-provider token strategy, then automate
GET /api/v1/local/email/outbox	200	aitesters-only: confirm intended test scope and key handling
DELETE /api/v1/local/email/outbox	200, 500	aitesters-only: isolate outbox state before adding a destructive check

Partially Covered Endpoints

These 18 operations have automation, but not for every live documented status:

Endpoint operation	Automated statuses	Missing documented statuses	Why missing	Best next action
DELETE /api/v1/cart/items/{productId}	200, 401, 404	400	Added after the repository contract snapshot	Explore malformed product-id behavior and add the stable case
DELETE /api/v1/products/{id}	204, 401, 403, 404	400	Added after the repository contract snapshot	Add malformed-id admin coverage
GET /api/v1/orders/{id}	200, 401, 404	400	Added after the repository contract snapshot	Add malformed-id customer coverage
GET /api/v1/products	200, 401	404	Contract now documents an unusual collection 404	Reproduce live and align contract or add a stable scenario
GET /api/v1/products/{id}	200, 401, 404	400	Added after the repository contract snapshot	Add malformed-id coverage
GET /api/v1/traffic/logs	200	400	Added after the repository contract snapshot	Identify and automate the documented invalid-query trigger
POST /api/v1/products	201, 400, 401, 403	404	Contract now documents an unclear create 404	Reproduce live and clarify the missing dependency/resource case
POST /api/v1/users/signup	201, 400	429	Shared rate-limit exhaustion is stateful	Add an isolated rate-limit environment or reset hook
POST /api/v1/users/signin	200, 400, 422	429	Shared rate-limit exhaustion is stateful	Add deterministic limiter reset/control
POST /api/v1/users/refresh	200, 400, 401	429	Shared rate-limit exhaustion is stateful	Add deterministic limiter reset/control
POST /api/v1/users/password/forgot	202, 400	429	Shared rate-limit exhaustion is stateful	Test in an isolated limiter scope
POST /api/v1/users/password/reset	400	200, 429	Success token is not observable; limiter path is stateful	Expose a safe token/outbox hook, then isolate rate-limit coverage
POST /api/v1/email	200, 400, 401	429	Exhaustion would poison the disposable user's bucket	Add deterministic limiter reset/control
POST /api/v1/qr/create	200, 400, 401	429	Shared rate-limit exhaustion is stateful	Add deterministic limiter reset/control
POST /api/v1/ollama/generate	200, 400, 401	404, 429, 500	Mock returns 200 for unknown model; limiter and 500 need safe controls	Align mock/contract; add limiter control and backend fault injection
POST /api/v1/ollama/chat	200, 400, 401	404, 429, 500	Mock returns 200 for unknown model; limiter and 500 need safe controls	Align mock/contract; add limiter control and backend fault injection
POST /api/v1/ollama/chat/tools	200, 400, 401	429, 500	Limiter and internal failure lack safe deterministic triggers	Add limiter control and backend fault-injection coverage
GET /api/v1/traffic/logs/{correlationId}	404	200	Live existing-record request returns confirmed 500	Fix backend, deploy, reverify, then add 200 test

The 70 missing response paths break down into:

• 46 paths belong to the 13 completely unautomated operations;
• 10 paths were added to already automated operations after the repository OpenAPI snapshot, including three Ollama 429 paths;
• 14 paths were already known gaps in the former 42-operation baseline.

Six behaviors previously described as “live-only” are now documented by the live Swagger: 400 for customer/admin order filters, refresh-token validation, user update validation, and both prompt-update length validations. This is why the live re-audit, rather than the checked-in api-docs.json, is the metric baseline.

Coverage By Domain

Domain	OpenAPI operations	Automated checks	Status	Important scope
Identity, users, prompts	17 / 24	55	Gap: 7 operations	2FA and SSO operations are unautomated
Inventory	0 / 4	0	Uncovered	Admin list, detail, adjustment, and movement history
Products	5 / 5	18	Covered	Authenticated reads and admin create/update/delete
Cart	5 / 5	16	Covered	Empty/populated state, add/update/remove/clear, validation and auth
Orders	6 / 6	31	Covered	Customer and admin flows, ownership, cancellation, status transitions
Email delivery	1 / 1	3	Covered with exclusion	Success, validation, auth; rate-limit exhaustion excluded
Local email outbox	0 / 2	0	Uncovered on aitesters	Confirm test-only scope before automation
Ollama	4 / 4	11	Covered with discrepancy	Definitions and deterministic SSE generate/chat/tool flows
QR	1 / 1	3	Covered	PNG response, validation, auth
Traffic	3 / 3	3	Degraded	Info/list covered; existing correlation detail success blocked by backend 500
OpenAPI and Swagger UI	—	2	Covered	Contract and documentation smoke checks
Total	42 / 55	142	Green existing suite; breadth gaps open	Two Playwright projects, fully parallel

Validation Evidence

Latest L17 checkpoint validation on 2026-08-30:

• Playwright discovery: 142 tests in 47 files.
• Full network-enabled npm test: 142 passed across api and admin-api in 1.9 minutes.
• A prior sandboxed run failed before reaching the API because DNS access was restricted; it was not an assertion or product failure.
• No credentials, local environment files, node_modules, or test artifacts were promoted into L17.

Validation commands:

cd l17
npm test
npm run test:api
npm run test:admin

Confirmed Backend Defects

1. Traffic log detail returns 500

• Endpoint: GET /api/v1/traffic/logs/{correlationId}
• Expected: documented 200 for an existing, freshly listed correlation id.
• Actual: reproducible 500 { "message": "Internal server error" }.
• Impact: diagnostic consumers cannot open a log entry returned by the list API; the documented success path remains blocked from regression coverage.
• Report: bug-reports/traffic-log-detail-returns-500.md

2. Repeated account-forgetting request returns 500

• Endpoint: DELETE /api/v1/users/{username}/right-to-be-forgotten
• Sequence: first self-delete returns 204; retrying with the deleted user's JWT returns 500.
• Reproduction: 3 / 3 direct curl attempts plus one fixture-teardown signal.
• Impact: safe client retry/cleanup looks like an infrastructure failure after the account was already removed.
• Report: bug-reports/right-to-be-forgotten-repeat-returns-500.md

The Playwright suite asserts intended first-request behavior and does not encode either 500 as an accepted regression result.

Coverage Constraints And Contract Discrepancies

Area	Current evidence	Coverage decision	Unblock condition
Password reset success	Authoritative deployment returns token=null and exposes no documented outbox/token read path	Keep verified 400 cases; do not invent a 200 flow	Provide a safe test token or local outbox visibility
Traffic detail 200	Freshly listed ids return 500	Keep stable unknown-id 404; success remains blocked	Backend fix deployed and live behavior reverified
Email 429	Requires exhausting a per-user rate-limit bucket	Excluded from routine regression to avoid poisoning state and unnecessary load	Isolated rate-limit environment or deterministic reset hook
Admin order unknown status	Live staging exploration returned 401 despite a valid admin JWT; adjacent valid requests succeeded	Use verified invalid-transition 400; reverify before formalizing	Reproduce safely and file/update a backend report if still present
Ollama unknown model	Controlled mock returns 200 while OpenAPI documents 404	Do not assert the documented 404	Align mock behavior or update the published contract

Test Architecture

• api project: non-admin specs against API_BASE_URL, defaulting to https://awesome.byst.re.
• admin-api project: tests/api/admin/** against API_ADMIN_BASE_URL, defaulting to https://aitesters.byst.re.
• Projects are independent and run in parallel.
• HTTP clients are initialized in test.beforeEach.
• Specs use // given, // when, // then and order cases by ascending response status.
• Dynamic ids, timestamps, pagination, traffic, and streams use structural assertions; stable request-derived fields use exact assertions.
• Ollama responses use an SSE parser and deterministic mock prompts.

Data Safety And Cleanup

• Admin mutations use generated disposable products and users; seeded entities are never deleted.
• Customer order fixtures cancel cancellable orders before cleanup to restore inventory.
• Admin order fixtures delete the generated owner/order before the generated product and verify cleanup statuses.
• Right-to-be-forgotten coverage uses a dedicated generated-user fixture that tolerates intentional deletion while preserving primary setup/test failures.
• Exploratory calls use generated identifiers and record cleanup outcomes.
• Credentials are loaded only from ignored local environment files.

Recent L17 Increment

• Added endpoint-focused coverage for POST /api/v1/email: 200, 400, 401.
• Added endpoint-focused coverage for right-to-be-forgotten: 204, 401, 403, 404, including observable post-deletion signin failure.
• Hardened generated-user cleanup after independent review.
• Completed e-commerce operation coverage: Products 5/5, Cart 5/5, Orders 6/6.
• Added and reconfirmed the two backend defect reports above.
• Restored explicit // given phases in legacy specs found by the style audit.
• Re-audited against both live Swagger contracts and corrected the stale 42-operation baseline to 55 operations on aitesters and 53 on the course API.

Next Actions

1. QA automation: add the four inventory operations first, using a generated product and a reversible adjustment with verified cleanup.
2. QA + security: implement the seven 2FA/SSO operations with disposable users, secret-safe evidence, and deterministic teardown.
3. Backend: fix traffic correlation-detail retrieval, deploy, reverify with list-to-detail curl, then add the documented 200 Playwright path.
4. Backend/security: map a JWT for a deleted principal to a handled 401 or documented 404; add a backend retry regression test.
5. Testability: expose a safe password-reset token/outbox mechanism and add the success flow without weakening anti-enumeration behavior.
6. Contract: reverify the ten newly documented statuses on existing operations, unknown admin order status, and Ollama missing-model behavior.
7. Scope decision: decide whether the two aitesters-only local-outbox operations belong in L17 automation; if yes, add isolated key-protected checks.
8. Maintenance: refresh the checked-in contract baseline and this report after every API surface change.

Reporting Rule

Every unexpected 5xx, contract mismatch, auth/validation inconsistency, cleanup failure, or flaky response is a defect signal. Preserve evidence, attempt up to three safe isolated reproductions, create or update a Markdown report under bug-reports/, link it here, and avoid turning broken behavior into a regression expectation.