Commit Graph
4 Commits
Author SHA1 Message Date
MarieandGitHub 4852ac401a Add server upgrade status on admin panel (#20107)
## Summary

Adds an admin upgrade-status panel that surfaces per-instance and
per-workspace migration health, backed by a Redis-cached aggregate to
keep the page snappy on large fleets.

<img width="827" height="880" alt="Screenshot 2026-04-28 at 10 21 03"
src="https://github.com/user-attachments/assets/8f88baa9-7268-4eff-bf6a-906a7f06ca91"
/>
<img width="804" height="892" alt="Screenshot 2026-04-28 at 10 21 11"
src="https://github.com/user-attachments/assets/1e6decf8-766a-4d0e-96b1-03a9962bba3c"
/>


## Computed metrics

**Instance** (`InstanceUpgradeStatus`)
- `inferredVersion` — version derived from the latest non-initial
instance command name
- `health` — `upToDate` | `behind` | `failed`, derived from the latest
attempt vs. the last expected instance step in the upgrade sequence
- `latestCommand` — `{ name, status, executedByVersion, errorMessage,
createdAt }` from the most recent attempt

**Per-workspace** (`WorkspaceUpgradeStatus`)
- `workspaceId`, `displayName`
- `inferredVersion`, `health`, `latestCommand` (same shape as instance),
computed against the latest expected step in the sequence

**Aggregate** (`AllWorkspacesUpgradeStatus`, only across `ACTIVE` /
`SUSPENDED` workspaces)
- `instanceUpgradeStatus`
- `totalCount`, `upToDateCount`, `behindCount`, `failedCount`
- `workspacesBehindIds[]`, `workspacesFailedIds[]`
- `computedAt`

## Fetching strategy

All reads go through `UpgradeStatusCacheService` (cache namespace:
`EngineHealth`).

- **Aggregate read** (`getAllWorkspacesStatus` →
`getAllWorkspacesUpgradeStatus` query):
reads summary + behind-ids + failed-ids in parallel; if any of the three
keys is missing, full recompute (`recomputeAllWorkspaces`) is triggered,
which also primes per-workspace entries.
- **Per-workspace read** (`getWorkspacesStatus(ids)` →
`getUpgradeStatus(ids)` query):
`mget` on workspace keys; misses are recomputed individually
(`recomputeWorkspace`), and aggregates are reconciled in place (count +
id list deltas) without a full recompute.
- **Recompute on demand**: `refreshUpgradeStatus` mutation calls
`recomputeAllWorkspaces` to bypass cache and rewrite all keys.
- **Auto-invalidation**: `InstanceCommandRunnerService` (fast + slow
paths) and `WorkspaceCommandRunnerService` invalidate after every run
via `safeInvalidateUpgradeStatusCache()`
(`flushByPattern('upgrade-status:*')`). Failures in cache invalidation
are swallowed and logged so they never break the migration runner.
- **TTL**: `60 * 60 * 1000` ms (1 hour) on every key — protects against
stale data even if a runner crashes before invalidating.

## Introduced cache keys

All under the `EngineHealth` cache-storage namespace:

| Key | Type | Purpose |
| --- | --- | --- |
| `upgrade-status:all-workspaces:summary` |
`CachedAllWorkspacesStatusSummary` | Counts + instance status +
`computedAt` |
| `upgrade-status:all-workspaces:behind-ids` | `string[]` | Workspace
ids in `behind` state |
| `upgrade-status:all-workspaces:failed-ids` | `string[]` | Workspace
ids in `failed` state |
| `upgrade-status:workspace:<workspaceId>` |
`CachedWorkspaceUpgradeStatus` | Per-workspace status (one key per
workspace) |

Full invalidation uses the pattern `upgrade-status:*`.

## Index added on `upgradeMigration` (already added on prod)

Migration
`2-2-instance-command-fast-1777308014234-addUpgradeMigrationWorkspaceIdIndex.ts`:

```sql
CREATE INDEX "IDX_upgradeMigration_workspaceId_name_attempt"
  ON "core"."upgradeMigration" ("workspaceId", "name", "attempt")
  WHERE "workspaceId" IS NOT NULL;
2026-05-04 15:06:07 +00:00
EtienneandGitHub 3306d66f5b cleaning - remove logs (#19445) 2026-04-08 13:05:47 +00:00
EtienneandGitHub 579714b62f Investigate memory leak (#19213)
In search for cache leak + Remove old instrumentation
2026-04-02 11:23:10 +00:00
Charles BochetandGitHub c407341912 feat: optimize hot database queries with multi-layer caching (#19068)
## Summary

Introduces multi-layer caching for the 5 most frequent database queries
identified in production (Sentry data), targeting the JWT authentication
hot path and cron job logic.

### Problem
Our database is under heavy load from uncached queries on the auth hot
path:
- `WorkspaceEntity` lookups: **638 queries/min**
- `ApiKeyEntity` lookups: **491 queries/min**
- `UserEntity` lookups: **147 queries/min**
- `UserWorkspaceEntity` lookups: **143 queries/min**
- `LogicFunctionEntity` lookups: **1800 queries/min** (cron job)

### Solution

**1. New `CoreEntityCacheService`** for non-workspace-scoped entities
(Workspace, User, UserWorkspace):
- Mirrors `WorkspaceCacheService` architecture (in-process Map + Redis
with hash validation)
- Provider pattern with `@CoreEntityCache` decorator
- Keyed by entity primary key (not workspaceId)
- 100ms local TTL, Redis-backed hash validation for cross-instance
consistency
- Three providers: `WorkspaceEntityCacheProviderService`,
`UserEntityCacheProviderService`,
`UserWorkspaceEntityCacheProviderService`

**2. New `apiKeyMap` WorkspaceCache** for workspace-scoped API key
lookups:
- `WorkspaceApiKeyMapCacheService` loads all API keys for a workspace
into a map by ID
- Leverages existing `WorkspaceCacheService` infrastructure
- Cache invalidation on API key create/update/revoke

**3. `CronTriggerCronJob` refactored** to use existing
`flatLogicFunctionMaps` workspace cache:
- Eliminates per-workspace `LogicFunctionEntity` repository queries
(~1800/min)
- Filters cached data in-memory instead

**4. `JwtAuthStrategy` refactored** to use caches for all entity
lookups:
- Workspace, User, UserWorkspace → `CoreEntityCacheService`
- ApiKey → `WorkspaceCacheService` (`apiKeyMap`)
- Impersonation queries kept as direct DB queries (rare path, requires
relations)

**5. Cache invalidation** wired into mutation paths:
- `WorkspaceService` → invalidates `workspaceEntity` on
save/update/delete
- `ApiKeyService` → invalidates `apiKeyMap` on create/update/revoke

### Architecture

```
Request → JwtAuthStrategy
  ├── Workspace lookup → CoreEntityCacheService (in-process → Redis → DB)
  ├── User lookup → CoreEntityCacheService (in-process → Redis → DB)
  ├── UserWorkspace lookup → CoreEntityCacheService (in-process → Redis → DB)
  └── ApiKey lookup → WorkspaceCacheService (in-process → Redis → DB)

CronTriggerCronJob
  └── LogicFunction lookup → WorkspaceCacheService (flatLogicFunctionMaps)
```

### Expected Impact
| Query | Before | After |
|-------|--------|-------|
| WorkspaceEntity | 638/min | ~0 (cached) |
| ApiKeyEntity | 491/min | ~0 (cached) |
| UserEntity | 147/min | ~0 (cached) |
| UserWorkspaceEntity | 143/min | ~0 (cached) |
| LogicFunctionEntity | 1800/min | ~0 (cached) |

### Not included (ongoing separately)
- DataSourceEntity query optimization (IS_DATASOURCE_MIGRATED migration)
- ObjectMetadataEntity query optimization (already partially cached)
2026-03-28 22:53:34 +01:00