4ea2e32366
## 1. The `twenty-client-sdk` Package (Source of Truth)
The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`
## 2. Generation & Upload (Server-Side, at Migration Time)
**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.
**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database
## 3. Invalidation Signal
The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive
Default is `false` so existing applications without a generated client
aren't affected.
## 4a. Logic Functions — Local Driver
**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`
**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.
## 4b. Logic Functions — Lambda Driver
**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`
**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).
## 5. Front Components
Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.
SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):
**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.
**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`
**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache
This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.
## Summary Diagram
```
app:build (SDK)
└─ twenty-client-sdk stub (metadata=real, core=stub)
│
▼
WorkspaceMigrationRunnerService.run()
└─ SdkClientGenerationService.generateAndStore()
├─ Copy stub package (package.json + dist/)
├─ replaceCoreClient() → regenerate core.mjs/core.cjs
├─ Zip entire package → upload to S3
└─ Set isSdkLayerStale = true
│
┌────────┴────────────────────┐
▼ ▼
Logic Functions Front Components
│ │
├─ Local Driver ├─ GET /rest/sdk-client/:appId/core
│ └─ downloadAndExtract │ → core.mjs from archive
│ → symlink into │
│ node_modules ├─ Host (useApplicationSdkClient)
│ │ ├─ Fetch SDK modules
└─ Lambda Driver │ ├─ Create blob URLs
└─ downloadArchiveBuffer │ └─ Cache in Jotai atom family
→ reprefixZipEntries │
→ publish as Lambda ├─ GET /rest/front-components/:id
layer │ → raw .mjs (no bundling)
│
└─ Worker (browser)
├─ Fetch component .mjs
├─ Rewrite imports → blob URLs
└─ import() rewritten source
```
## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server
---------
Co-authored-by: Charles Bochet <[email protected]>
302 lines
10 KiB
JSON
302 lines
10 KiB
JSON
{
|
|
"name": "twenty-server",
|
|
"$schema": "../../node_modules/nx/schemas/project-schema.json",
|
|
"projectType": "application",
|
|
"tags": ["scope:backend"],
|
|
"targets": {
|
|
"build": {
|
|
"executor": "nx:run-commands",
|
|
"cache": true,
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"parallel": false,
|
|
"commands": [
|
|
"rimraf dist",
|
|
"nest build --path ./tsconfig.build.json",
|
|
"mkdir -p dist/assets/twenty-client-sdk && cp ../twenty-client-sdk/package.json dist/assets/twenty-client-sdk/ && cp -r ../twenty-client-sdk/dist dist/assets/twenty-client-sdk/dist"
|
|
]
|
|
},
|
|
"dependsOn": ["^build"]
|
|
},
|
|
"test:integration": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"commands": [
|
|
"NODE_ENV=test NODE_OPTIONS=\"--max-old-space-size=12288 --import tsx/esm\" nx jest --config ./jest-integration.config.ts"
|
|
]
|
|
},
|
|
"parallel": false,
|
|
"configurations": {
|
|
"with-db-reset": {
|
|
"cwd": "packages/twenty-server",
|
|
"commands": [
|
|
"NODE_ENV=test NODE_OPTIONS=\"--max-old-space-size=12288 --import tsx/esm\" nx database:reset > reset-logs.log && NODE_ENV=test NODE_OPTIONS=\"--max-old-space-size=12288 --import tsx/esm\" nx jest --config ./jest-integration.config.ts"
|
|
]
|
|
}
|
|
}
|
|
},
|
|
"build:packageJson": {
|
|
"executor": "@nx/js:tsc",
|
|
"options": {
|
|
"main": "packages/twenty-server/dist/main.js",
|
|
"tsConfig": "packages/twenty-server/tsconfig.json",
|
|
"outputPath": "packages/twenty-server/dist",
|
|
"updateBuildableProjectDepsInPackageJson": true
|
|
}
|
|
},
|
|
"typecheck": {
|
|
"dependsOn": ["^build"]
|
|
},
|
|
"start": {
|
|
"executor": "nx:run-commands",
|
|
"cache": false,
|
|
"dependsOn": ["^build"],
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "rimraf dist && NODE_ENV=development nest start --watch"
|
|
}
|
|
},
|
|
"start:ci": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "NODE_ENV=development nest start"
|
|
}
|
|
},
|
|
"start:ci-if-needed": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["build"],
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "curl -f -s http://localhost:3000/healthz > /dev/null 2>&1 && echo '✅ Server already running' || (echo '🚀 Server not running, starting...' && nohup nest start &)"
|
|
}
|
|
},
|
|
"start:debug": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["^build"],
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "rimraf dist && NODE_ENV=development nest start --watch --debug"
|
|
}
|
|
},
|
|
"reset:env": {
|
|
"executor": "nx:run-commands",
|
|
"inputs": ["{projectRoot}/.env.example"],
|
|
"outputs": ["{projectRoot}/.env"],
|
|
"cache": true,
|
|
"options": {
|
|
"cwd": "{projectRoot}",
|
|
"command": "cp .env.example .env"
|
|
}
|
|
},
|
|
"reset:env:e2e-testing-server": {
|
|
"executor": "nx:run-commands",
|
|
"inputs": ["{projectRoot}/.env.e2e-testing-server"],
|
|
"outputs": ["{projectRoot}/.env"],
|
|
"cache": true,
|
|
"options": {
|
|
"cwd": "{projectRoot}",
|
|
"command": "cp .env.e2e-testing-server .env"
|
|
}
|
|
},
|
|
"command": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["build"],
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "node dist/command/command.js"
|
|
}
|
|
},
|
|
"command-no-deps": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "node dist/command/command.js"
|
|
}
|
|
},
|
|
"worker": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["build"],
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "node dist/queue-worker/queue-worker.js"
|
|
}
|
|
},
|
|
"typeorm": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "ts-node --transpile-only -P tsconfig.json ../../node_modules/typeorm/cli.js"
|
|
},
|
|
"dependsOn": ["^build"]
|
|
},
|
|
"ts-node": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["^build"],
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "ts-node"
|
|
}
|
|
},
|
|
"ts-node-no-deps": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "ts-node"
|
|
}
|
|
},
|
|
"ts-node-no-deps-transpile-only": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "ts-node --transpile-only"
|
|
}
|
|
},
|
|
"lint": {
|
|
"executor": "nx:run-commands",
|
|
"cache": true,
|
|
"dependsOn": ["twenty-oxlint-rules:build"],
|
|
"inputs": [
|
|
"{projectRoot}/src/**/*.{ts,tsx}",
|
|
"{projectRoot}/tsconfig*.json",
|
|
"{workspaceRoot}/tsconfig.base.json",
|
|
"{projectRoot}/.oxlintrc.json",
|
|
"{workspaceRoot}/packages/twenty-oxlint-rules/dist/oxlint-plugin.mjs"
|
|
],
|
|
"options": {
|
|
"cwd": "{projectRoot}",
|
|
"command": "npx oxlint --type-aware -c .oxlintrc.json src/ && (prettier src/ --check --cache --cache-location ../../.cache/prettier/{projectRoot} --cache-strategy metadata || (echo 'ERROR: Prettier formatting check failed! Fix with: npx nx lint twenty-server --configuration=fix' && false))"
|
|
},
|
|
"configurations": {
|
|
"ci": {},
|
|
"fix": {
|
|
"command": "npx oxlint --type-aware --fix -c .oxlintrc.json src/ && prettier src/ --write --cache --cache-location ../../.cache/prettier/{projectRoot} --cache-strategy metadata"
|
|
}
|
|
}
|
|
},
|
|
"lint:diff-with-main": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "{projectRoot}",
|
|
"command": "FILES=$(git diff --name-only --relative --diff-filter=d main...HEAD -- src/ | grep -E '\\.(ts|tsx)$'); [ -z \"$FILES\" ] && echo 'No changed files.' || (npx oxlint --type-aware -c .oxlintrc.json $FILES && (prettier --check $FILES || (echo 'ERROR: Prettier formatting check failed! Fix with: npx nx lint:diff-with-main twenty-server --configuration=fix' && false)))"
|
|
},
|
|
"configurations": {
|
|
"fix": {
|
|
"command": "FILES=$(git diff --name-only --relative --diff-filter=d main...HEAD -- src/ | grep -E '\\.(ts|tsx)$'); [ -z \"$FILES\" ] && echo 'No changed files.' || (npx oxlint --type-aware --fix -c .oxlintrc.json $FILES && prettier --write $FILES)"
|
|
}
|
|
}
|
|
},
|
|
"test": {},
|
|
"test:debug": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "node --inspect-brk -r tsconfig-paths/register -r ts-node/register ../../node_modules/.bin/jest --runInBand"
|
|
}
|
|
},
|
|
"jest": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "jest"
|
|
}
|
|
},
|
|
"database:migrate": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["^build"],
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"commands": [
|
|
"nx typeorm -- migration:run -d src/database/typeorm/core/core.datasource"
|
|
]
|
|
}
|
|
},
|
|
"database:migrate:generate": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "npx nx typeorm -- migration:generate src/database/typeorm/core/migrations/common/{args.migrationName} -d src/database/typeorm/core/core.datasource.ts"
|
|
}
|
|
},
|
|
"database:migrate:revert": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["^build"],
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"commands": [
|
|
"nx typeorm -- migration:revert -d src/database/typeorm/core/core.datasource"
|
|
]
|
|
}
|
|
},
|
|
"generate:integration-test": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"commands": [
|
|
"nx ts-node-no-deps -- ./test/integration/graphql/codegen/index.ts"
|
|
],
|
|
"parallel": false
|
|
}
|
|
},
|
|
"database:reset": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["build"],
|
|
"configurations": {
|
|
"no-seed": {
|
|
"cwd": "packages/twenty-server",
|
|
"commands": [
|
|
"nx ts-node-no-deps-transpile-only -- ./scripts/truncate-db.ts",
|
|
"nx ts-node-no-deps-transpile-only -- ./scripts/setup-db.ts",
|
|
"nx database:migrate",
|
|
"nx command-no-deps -- cache:flush"
|
|
],
|
|
"parallel": false
|
|
},
|
|
"seed": {
|
|
"cwd": "packages/twenty-server",
|
|
"commands": [
|
|
"nx ts-node-no-deps-transpile-only -- ./scripts/truncate-db.ts",
|
|
"nx ts-node-no-deps-transpile-only -- ./scripts/setup-db.ts",
|
|
"nx database:migrate",
|
|
"nx command-no-deps -- cache:flush",
|
|
"nx command-no-deps -- workspace:seed:dev"
|
|
],
|
|
"parallel": false
|
|
}
|
|
},
|
|
"defaultConfiguration": "seed"
|
|
},
|
|
"clickhouse:migrate": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "nx ts-node-no-deps-transpile-only -- src/database/clickHouse/migrations/run-migrations.ts"
|
|
}
|
|
},
|
|
"clickhouse:seed": {
|
|
"executor": "nx:run-commands",
|
|
"options": {
|
|
"cwd": "packages/twenty-server",
|
|
"command": "nx ts-node-no-deps-transpile-only -- src/database/clickHouse/seeds/run-seeds.ts"
|
|
}
|
|
},
|
|
"lingui:extract": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["^build"],
|
|
"options": {
|
|
"cwd": "{projectRoot}",
|
|
"command": "lingui extract --overwrite --clean"
|
|
}
|
|
},
|
|
"lingui:compile": {
|
|
"executor": "nx:run-commands",
|
|
"dependsOn": ["^build"],
|
|
"options": {
|
|
"cwd": "{projectRoot}",
|
|
"command": "lingui compile --typescript"
|
|
}
|
|
}
|
|
}
|
|
}
|