Skip to main content

Implementation Portal Troubleshooting Runbook

Operational runbook for diagnosing backend issues in apps/backend/src/implementation-portal/.

Use this when team users or portal clients report OTP failures, blocked step actions, invoice issues, or attachment problems.

Scope​

Primary codepaths:

  • interfaces/http/*.controller.ts
  • interfaces/guards/portal-session.guard.ts
  • application/use-cases/*
  • domain/entities/*
  • domain/services/*

1) Symptom triage​

SymptomLikely layer
Client cannot request OTPRequestPortalOtpUseCase + email adapter
OTP verification failsVerifyPortalOtpUseCase + portal session repository
401 Portal session token is requiredPortalSessionGuard header/cookie mismatch
401 No token provided on portal write after OTP succeedsGlobal Firebase AuthGuard reached the route before PortalSessionGuard
Client cannot complete a document stepCompleteClientStepUseCase attachment precondition
Team cannot log timeLogTimeEntryUseCase + TimeEntry minimum-hours rule
Time entry edit/delete blockedBoardStepController invoice guard (invoiceDraftId)
Invoice send failsSendInvoiceDraftUseCase + payment adapter
Attachments fail uploadUploadStepAttachmentUseCase + storage adapter + MIME/size constraints
Board health unexpectedly redOverdueCheckerService stale/overdue thresholds
Overdue notification noise or silenceBullMQ queue implementation-portal-overdue + overdue processor state

2) OTP and portal session flow checks​

Expected flow:

  1. POST /portal/:shareToken/auth/request-otp
  2. POST /portal/:shareToken/auth/verify-otp
  3. guarded portal write endpoint with session token

Validated rules from code:

  • OTP request is intentionally non-enumerating from controller perspective (204 response path).
  • OTP TTL is 15 minutes.
  • session TTL is 24 hours.
  • session token can be provided by:
    • x-portal-session-token header
    • flowpos-portal-session cookie

If clients still fail after OTP verify:

  1. confirm token is sent on write request
  2. confirm token is not stale (>24h)
  3. confirm shareToken points to expected board

Confirmed OTP-only write blocker​

Portal session check and write handlers use PortalSessionGuard but are not marked @IsPublic():

  • GET /portal/:shareToken/auth/session
  • PATCH .../steps/:stepId/complete
  • PATCH .../steps/:stepId/client-approve
  • POST .../steps/:stepId/attachments
  • POST .../steps/:stepId/comments

For OTP-only clients (no Firebase ID token), the global AuthGuard rejects the request with 401 No token provided before PortalSessionGuard runs. This is deterministic current backend behavior, not a flaky client bug.

Intended fix: add @IsPublic() to those handlers so portal session tokens are the auth boundary for client mutations. Until that lands, OTP-only write testing requires either a temporary Firebase token or a code change.

Use GET /portal/:shareToken/auth/session only after the @IsPublic() fix (or with a Firebase token). When reachable, it returns 204 for a valid portal session token or cookie.

Portal write 401 decision tree​

ResponseCheck firstLikely owner
401 No token providedOTP-only client hit a non-@IsPublic() portal write/session route; global Firebase AuthGuard rejected firstBackend controller metadata (@IsPublic() missing)
401 Portal session token is requiredRequest reached PortalSessionGuard but neither x-portal-session-token nor flowpos-portal-session was presentClient request wiring
401 Invalid or expired portal sessionSession TTL, token copy/paste, share-token mismatchPortal session repository / client state

3) Step-status failures​

Team-side completion (board-steps endpoints)​

Common blockers:

  • document step has zero attachments (Document steps require at least one attachment...)
  • invalid state transition (for example completing from pending instead of in_progress)
  • approval flow misuse (approve/reject required for approval step types)

Client-side completion (portal endpoints)​

Common blockers:

  • step is not client-assigned
  • document step has no attachments
  • client approval called before step reached completed|approved

4) Time-entry failures​

From LogTimeEntryUseCase and TimeEntry entity:

  • step must be hourly billed (billableType=hourly)
  • minimum entry is 0.25 hours

From BoardStepController:

  • edit/delete of time entries is blocked once step has invoiceDraftId set

When users report "time log won't save":

  1. verify step billing type
  2. verify hours >= 0.25
  3. verify step has not already been linked to an invoice draft

5) Attachment failures​

From UploadStepAttachmentUseCase and StepAttachment entity:

  • max file size: 50 MB (STEP_ATTACHMENT_MAX_BYTES)
  • MIME type must be in the domain allowlist
  • storage key format: portal/{boardId}/steps/{stepId}/{timestamp}-{sanitizedFilename}
  • client portal upload requests must include multipart field boardId in addition to file

From GcsStorageAdapter:

  • requires GCS_PUBLIC_BUCKET
  • signed URLs are generated on read endpoints

Failure patterns:

  • Unsupported file type -> MIME not allowed
  • File size exceeds the 50 MB limit -> too large
  • storage configuration errors -> bucket missing or GCS credentials issue

6) Invoice failures​

Draft generation​

GenerateInvoiceDraftUseCase only includes unbilled steps and uses InvoiceCalculatorService.

Frequent causes of empty/low totals:

  • steps are non_billable
  • step statuses are not in billable completion states
  • hourly steps have no time entries
  • steps were already linked to another invoice draft (invoiceDraftId)

Send flow​

SendInvoiceDraftUseCase sequence:

  1. load draft + board
  2. create invoice in payment adapter
  3. add line items
  4. finalize
  5. mark sent
  6. send notification (best-effort)

If send fails:

  1. verify draft exists and belongs to business
  2. verify draft is still in draft status
  3. inspect payment adapter integration logs

7) Board health anomalies​

OverdueCheckerService rules:

  • green: 0 overdue
  • yellow: 1-2 overdue
  • red: 3+ overdue OR stale board (>7 days without step updates)

Terminal statuses excluded from overdue counting:

  • completed
  • approved
  • client_approved
  • rejected

If board health appears too severe, confirm stale condition before changing overdue logic.

Overdue notification queue​

The overdue processor runs through BullMQ queue implementation-portal-overdue and periodically evaluates active boards for overdue steps. It updates notification state through application use cases; overdue health itself remains domain behavior in OverdueCheckerService.

When overdue notifications are duplicated or missing:

  1. inspect repeat jobs for implementation-portal-overdue
  2. verify the processor is running in the backend worker process
  3. inspect affected board steps for due dates and terminal statuses
  4. check whether the board was already marked as notified before replaying jobs
  5. retry the queue job only after confirming the notification adapter dependency is healthy

Do not change health thresholds in the processor. Change health rules in the domain service only when the product rule itself changes.

8) Safe recovery actions​

  1. Re-issue OTP and retry portal session flow.
  2. Re-upload required document attachment before step completion.
  3. Regenerate invoice draft after resolving step/time-entry eligibility.
  4. Re-run send action once payment adapter dependency is healthy.
  5. Regenerate board share token for compromised or misrouted links.

9) Adapter dependency boundaries​

Implementation portal uses ports for storage, payment, and notifications:

  • attachment upload/read issues belong to the storage adapter after the domain MIME and size checks pass
  • invoice send issues belong to the payment adapter after draft status and billable-line eligibility pass
  • team/client notification issues belong to the notification adapter after the use case emits the expected event
  • OTP delivery is part of the portal OTP use case and email adapter path, not the generic notification port

Keep secret names and values in Doppler/deployment manifests. In this runbook, record only which adapter boundary failed and which use case reached it.