ca0f04923ceada4befb23152c0790f1e5cb9596d
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Charles BochetandGitHub
|
9e515afb13 |
feat(server): asymmetric JWT signing with kid + key rotation table (#20467)
## Context Today every JWT issued by Twenty (access, refresh, login, file, etc.) is HMAC-signed with a per-token-type secret derived from the global `APP_SECRET`. Rotating that secret invalidates **every** active token at once and there is no way to scope a leak to a subset of tokens. This PR is the first slice of a broader effort to **decouple stateful encryption (`APP_SECRET`-derived secrets) from stateless encryption (JWTs)**. It introduces an asymmetric (private/public key) signing path for `ACCESS` and `REFRESH` tokens and a signing-key registry to enable **safe rotation**: leaked keys can be revoked by flipping `revokedAt`/`isCurrent` on the matching row without invalidating tokens issued by other keys. > Out of scope (intentionally): swapping stateful encryption for `APP_SECRET`, asymmetric signing for token types other than `ACCESS`/`REFRESH`, an admin-panel rotation UI, and an enterprise re-encryption command. Those will land in follow-up PRs. ## What changes - **New `core.signingKey` table** (instance command `2.5.0` / `1778550000000`) storing both the public key (PEM, in clear) and the private key (PEM, encrypted with `APP_SECRET` via `SecretEncryptionService`). One row is marked `isCurrent = true` (enforced by a partial unique index). The row's UUID `id` is used directly as the JWT `kid`. - When a key is rotated out, its `privateKey` is nulled (we never keep historical private keys) but the `publicKey` row stays so previously issued tokens can still be verified. - **`JwtKeyManagerService`** lazily loads-or-generates the current signing key on first use: - If a row with `isCurrent = true` exists → decrypts and uses it. - Otherwise → generates a fresh EC P-256 keypair, encrypts the private key, inserts the row (UUID id = kid). Handles concurrent insert races via the unique constraint. - **`JwtWrapperService.signAsync()`** signs `ACCESS`/`REFRESH` payloads with `ES256` and a `kid` header. Falls back to `HS256` if no signing key is available (boot-time DB error, transient failure). - **Dual-path verification** in both `JwtWrapperService.verifyJwtToken` and the Passport `JwtAuthStrategy.secretOrKeyProvider`: - JWT with a `kid` header → resolve the public key PEM by id and verify with `ES256`, - otherwise → fall back to the existing `APP_SECRET`-derived `HS256` path (unchanged). - **`AccessTokenService` / `RefreshTokenService`** now sign through `signAsync` (single public surface; the routing detail stays internal to the wrapper). - **Public key cache**: a new `SigningKeyEntityCacheProviderService` plugs into `CoreEntityCacheService` (`signingKeyPublicKey` namespace) and serves PEMs by id, with the standard local-memo + Redis layering. - **PEM strings end-to-end**: `jsonwebtoken` accepts PEM strings directly for both sign and verify, so the manager never converts to a Node `KeyObject` and the cache hands the PEM straight to `jwt.verify`. ## Why ES256 (and not EdDSA / RS256) - `@nestjs/jwt` is backed by `jsonwebtoken`, which does **not** support EdDSA today. - ES256 keys are tiny (~120 bytes vs 1.6 kB for RS256), signatures are short (~64 bytes), and signing/verification is fast — important since JWT verification runs on every authenticated request. - ES256 is widely supported and standardized (RFC 7518), with mature ecosystem support. ## Why store the private key in DB (not env) - No new secret to provision: existing instances already have `APP_SECRET`, which we reuse to encrypt the private key at rest. - Self-healing: a fresh instance auto-generates its first signing key on first boot. Nothing to copy/paste. - Rotation is a SQL operation against `core.signingKey`, not a redeploy + env mutation. ## Backward compatibility - All previously-issued tokens (no `kid`) keep verifying through the legacy HS256 path with their existing `APP_SECRET`-derived secret. No forced re-login. - Token types not in scope (`WORKSPACE_AGNOSTIC`, `API_KEY`, `FILE`, `LOGIN`, `EMAIL_VERIFICATION`, etc.) keep their current HS256 behavior unchanged — they still go through the synchronous `JwtWrapperService.sign(payload, options)` with a caller-supplied secret. - `signWithAppSecret` is kept intentionally as the HS256 fallback path; it will be deprecated in a follow-up PR. - If the DB lookup/generation fails for any reason, the wrapper logs the error and falls back to HS256 — no startup crash, no silent regression. ## Rotation story 1. Bootstrap: first signing call lazily inserts a row in `core.signingKey` with `isCurrent = true`, `privateKey = encrypt(pem_A)`. New tokens carry `kid_A`. 2. Rotate: `UPDATE core."signingKey" SET "isCurrent" = false, "privateKey" = NULL WHERE id = '<kid_A>';` then insert a new row with `isCurrent = true`. New tokens carry `kid_B`. Tokens still in flight with `kid_A` keep verifying because the public-key row for `kid_A` is still there. 3. Revoke: `UPDATE core."signingKey" SET "revokedAt" = now() WHERE id = '<kid_A>';`. All tokens with `kid_A` now fail verification cleanly with `UNAUTHENTICATED` (no 500). 4. Tokens with no `kid` (legacy) are unaffected throughout. ## Test plan - [x] Unit: `JwtWrapperService` dual-path verification (HS256 no-kid vs ES256 with-kid), unknown-kid → `UNAUTHENTICATED`, `signAsync` happy path + `null` when no key, `signAsync` rejection for non-rotatable types. - [x] Unit: `JwtAuthStrategy` `secretOrKeyProvider` dual-path resolution and algorithm validation. - [x] All existing JWT/auth/application unit tests adjusted to the renamed public method. - [x] Integration (`jwt-key-rotation.integration-spec.ts`): - **Happy path**: signed-up user's `ACCESS` token has `alg=ES256` + correct UUID `kid`, the `isCurrent=true` row exists in `core.signingKey`, `getCurrentUser` resolves. - **Legacy fallback**: hand-crafted no-kid HS256 token verifies via the legacy `APP_SECRET`-derived path. - **Previous-key rotation**: token signed by a hardcoded *previous* key whose row is pre-inserted with `privateKey = NULL` (rotated-out) still verifies — proves the leaked-key revocation flow works in both directions. - **Unknown kid**: token signed with an orphan UUID `kid` is cleanly rejected (no 500). - [x] `npx nx typecheck twenty-server` - [x] `npx nx test twenty-server` - [x] `npx nx run twenty-server:lint` |
||
|
MarieandGitHub
|
4852ac401a |
Add server upgrade status on admin panel (#20107)
## Summary Adds an admin upgrade-status panel that surfaces per-instance and per-workspace migration health, backed by a Redis-cached aggregate to keep the page snappy on large fleets. <img width="827" height="880" alt="Screenshot 2026-04-28 at 10 21 03" src="https://github.com/user-attachments/assets/8f88baa9-7268-4eff-bf6a-906a7f06ca91" /> <img width="804" height="892" alt="Screenshot 2026-04-28 at 10 21 11" src="https://github.com/user-attachments/assets/1e6decf8-766a-4d0e-96b1-03a9962bba3c" /> ## Computed metrics **Instance** (`InstanceUpgradeStatus`) - `inferredVersion` — version derived from the latest non-initial instance command name - `health` — `upToDate` | `behind` | `failed`, derived from the latest attempt vs. the last expected instance step in the upgrade sequence - `latestCommand` — `{ name, status, executedByVersion, errorMessage, createdAt }` from the most recent attempt **Per-workspace** (`WorkspaceUpgradeStatus`) - `workspaceId`, `displayName` - `inferredVersion`, `health`, `latestCommand` (same shape as instance), computed against the latest expected step in the sequence **Aggregate** (`AllWorkspacesUpgradeStatus`, only across `ACTIVE` / `SUSPENDED` workspaces) - `instanceUpgradeStatus` - `totalCount`, `upToDateCount`, `behindCount`, `failedCount` - `workspacesBehindIds[]`, `workspacesFailedIds[]` - `computedAt` ## Fetching strategy All reads go through `UpgradeStatusCacheService` (cache namespace: `EngineHealth`). - **Aggregate read** (`getAllWorkspacesStatus` → `getAllWorkspacesUpgradeStatus` query): reads summary + behind-ids + failed-ids in parallel; if any of the three keys is missing, full recompute (`recomputeAllWorkspaces`) is triggered, which also primes per-workspace entries. - **Per-workspace read** (`getWorkspacesStatus(ids)` → `getUpgradeStatus(ids)` query): `mget` on workspace keys; misses are recomputed individually (`recomputeWorkspace`), and aggregates are reconciled in place (count + id list deltas) without a full recompute. - **Recompute on demand**: `refreshUpgradeStatus` mutation calls `recomputeAllWorkspaces` to bypass cache and rewrite all keys. - **Auto-invalidation**: `InstanceCommandRunnerService` (fast + slow paths) and `WorkspaceCommandRunnerService` invalidate after every run via `safeInvalidateUpgradeStatusCache()` (`flushByPattern('upgrade-status:*')`). Failures in cache invalidation are swallowed and logged so they never break the migration runner. - **TTL**: `60 * 60 * 1000` ms (1 hour) on every key — protects against stale data even if a runner crashes before invalidating. ## Introduced cache keys All under the `EngineHealth` cache-storage namespace: | Key | Type | Purpose | | --- | --- | --- | | `upgrade-status:all-workspaces:summary` | `CachedAllWorkspacesStatusSummary` | Counts + instance status + `computedAt` | | `upgrade-status:all-workspaces:behind-ids` | `string[]` | Workspace ids in `behind` state | | `upgrade-status:all-workspaces:failed-ids` | `string[]` | Workspace ids in `failed` state | | `upgrade-status:workspace:<workspaceId>` | `CachedWorkspaceUpgradeStatus` | Per-workspace status (one key per workspace) | Full invalidation uses the pattern `upgrade-status:*`. ## Index added on `upgradeMigration` (already added on prod) Migration `2-2-instance-command-fast-1777308014234-addUpgradeMigrationWorkspaceIdIndex.ts`: ```sql CREATE INDEX "IDX_upgradeMigration_workspaceId_name_attempt" ON "core"."upgradeMigration" ("workspaceId", "name", "attempt") WHERE "workspaceId" IS NOT NULL; |
||
|
EtienneandGitHub
|
3306d66f5b | cleaning - remove logs (#19445) | ||
|
EtienneandGitHub
|
579714b62f |
Investigate memory leak (#19213)
In search for cache leak + Remove old instrumentation |
||
|
Charles BochetandGitHub
|
c407341912 |
feat: optimize hot database queries with multi-layer caching (#19068)
## Summary Introduces multi-layer caching for the 5 most frequent database queries identified in production (Sentry data), targeting the JWT authentication hot path and cron job logic. ### Problem Our database is under heavy load from uncached queries on the auth hot path: - `WorkspaceEntity` lookups: **638 queries/min** - `ApiKeyEntity` lookups: **491 queries/min** - `UserEntity` lookups: **147 queries/min** - `UserWorkspaceEntity` lookups: **143 queries/min** - `LogicFunctionEntity` lookups: **1800 queries/min** (cron job) ### Solution **1. New `CoreEntityCacheService`** for non-workspace-scoped entities (Workspace, User, UserWorkspace): - Mirrors `WorkspaceCacheService` architecture (in-process Map + Redis with hash validation) - Provider pattern with `@CoreEntityCache` decorator - Keyed by entity primary key (not workspaceId) - 100ms local TTL, Redis-backed hash validation for cross-instance consistency - Three providers: `WorkspaceEntityCacheProviderService`, `UserEntityCacheProviderService`, `UserWorkspaceEntityCacheProviderService` **2. New `apiKeyMap` WorkspaceCache** for workspace-scoped API key lookups: - `WorkspaceApiKeyMapCacheService` loads all API keys for a workspace into a map by ID - Leverages existing `WorkspaceCacheService` infrastructure - Cache invalidation on API key create/update/revoke **3. `CronTriggerCronJob` refactored** to use existing `flatLogicFunctionMaps` workspace cache: - Eliminates per-workspace `LogicFunctionEntity` repository queries (~1800/min) - Filters cached data in-memory instead **4. `JwtAuthStrategy` refactored** to use caches for all entity lookups: - Workspace, User, UserWorkspace → `CoreEntityCacheService` - ApiKey → `WorkspaceCacheService` (`apiKeyMap`) - Impersonation queries kept as direct DB queries (rare path, requires relations) **5. Cache invalidation** wired into mutation paths: - `WorkspaceService` → invalidates `workspaceEntity` on save/update/delete - `ApiKeyService` → invalidates `apiKeyMap` on create/update/revoke ### Architecture ``` Request → JwtAuthStrategy ├── Workspace lookup → CoreEntityCacheService (in-process → Redis → DB) ├── User lookup → CoreEntityCacheService (in-process → Redis → DB) ├── UserWorkspace lookup → CoreEntityCacheService (in-process → Redis → DB) └── ApiKey lookup → WorkspaceCacheService (in-process → Redis → DB) CronTriggerCronJob └── LogicFunction lookup → WorkspaceCacheService (flatLogicFunctionMaps) ``` ### Expected Impact | Query | Before | After | |-------|--------|-------| | WorkspaceEntity | 638/min | ~0 (cached) | | ApiKeyEntity | 491/min | ~0 (cached) | | UserEntity | 147/min | ~0 (cached) | | UserWorkspaceEntity | 143/min | ~0 (cached) | | LogicFunctionEntity | 1800/min | ~0 (cached) | ### Not included (ongoing separately) - DataSourceEntity query optimization (IS_DATASOURCE_MIGRATED migration) - ObjectMetadataEntity query optimization (already partially cached) |