## 1. The `twenty-client-sdk` Package (Source of Truth)
The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`
## 2. Generation & Upload (Server-Side, at Migration Time)
**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.
**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database
## 3. Invalidation Signal
The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive
Default is `false` so existing applications without a generated client
aren't affected.
## 4a. Logic Functions — Local Driver
**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`
**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.
## 4b. Logic Functions — Lambda Driver
**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`
**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).
## 5. Front Components
Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.
SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):
**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.
**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`
**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache
This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.
## Summary Diagram
```
app:build (SDK)
└─ twenty-client-sdk stub (metadata=real, core=stub)
│
▼
WorkspaceMigrationRunnerService.run()
└─ SdkClientGenerationService.generateAndStore()
├─ Copy stub package (package.json + dist/)
├─ replaceCoreClient() → regenerate core.mjs/core.cjs
├─ Zip entire package → upload to S3
└─ Set isSdkLayerStale = true
│
┌────────┴────────────────────┐
▼ ▼
Logic Functions Front Components
│ │
├─ Local Driver ├─ GET /rest/sdk-client/:appId/core
│ └─ downloadAndExtract │ → core.mjs from archive
│ → symlink into │
│ node_modules ├─ Host (useApplicationSdkClient)
│ │ ├─ Fetch SDK modules
└─ Lambda Driver │ ├─ Create blob URLs
└─ downloadArchiveBuffer │ └─ Cache in Jotai atom family
→ reprefixZipEntries │
→ publish as Lambda ├─ GET /rest/front-components/:id
layer │ → raw .mjs (no bundling)
│
└─ Worker (browser)
├─ Fetch component .mjs
├─ Rewrite imports → blob URLs
└─ import() rewritten source
```
## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server
---------
Co-authored-by: Charles Bochet <charlesBochet@users.noreply.github.com>
226 lines
6.3 KiB
TypeScript
226 lines
6.3 KiB
TypeScript
// @ts-nocheck
|
|
import type { LinkedField, LinkedType } from './types'
|
|
|
|
export interface Args {
|
|
[arg: string]: any | undefined
|
|
}
|
|
|
|
export interface Fields {
|
|
[field: string]: Request
|
|
}
|
|
|
|
export type Request = boolean | number | Fields
|
|
|
|
export interface Variables {
|
|
[name: string]: {
|
|
value: any
|
|
typing: [LinkedType, string]
|
|
}
|
|
}
|
|
|
|
export interface Context {
|
|
root: LinkedType
|
|
varCounter: number
|
|
variables: Variables
|
|
fragmentCounter: number
|
|
fragments: string[]
|
|
}
|
|
|
|
export interface GraphqlOperation {
|
|
query: string
|
|
variables?: { [name: string]: any }
|
|
operationName?: string
|
|
}
|
|
|
|
const parseRequest = (
|
|
request: Request | undefined,
|
|
ctx: Context,
|
|
path: string[],
|
|
): string => {
|
|
if (typeof request === 'object' && '__args' in request) {
|
|
const args: any = request.__args
|
|
let fields: Request | undefined = { ...request }
|
|
delete fields.__args
|
|
const argNames = Object.keys(args)
|
|
|
|
if (argNames.length === 0) {
|
|
return parseRequest(fields, ctx, path)
|
|
}
|
|
|
|
const field = getFieldFromPath(ctx.root, path)
|
|
|
|
const argStrings = argNames.map((argName) => {
|
|
ctx.varCounter++
|
|
const varName = `v${ctx.varCounter}`
|
|
|
|
const typing = field.args && field.args[argName] // typeMap used here, .args
|
|
|
|
if (!typing) {
|
|
throw new Error(
|
|
`no typing defined for argument \`${argName}\` in path \`${path.join(
|
|
'.',
|
|
)}\``,
|
|
)
|
|
}
|
|
|
|
ctx.variables[varName] = {
|
|
value: args[argName],
|
|
typing,
|
|
}
|
|
|
|
return `${argName}:$${varName}`
|
|
})
|
|
return `(${argStrings})${parseRequest(fields, ctx, path)}`
|
|
} else if (typeof request === 'object' && Object.keys(request).length > 0) {
|
|
const fields = request
|
|
const fieldNames = Object.keys(fields).filter((k) => Boolean(fields[k]))
|
|
|
|
if (fieldNames.length === 0) {
|
|
throw new Error(
|
|
`field selection should not be empty: ${path.join('.')}`,
|
|
)
|
|
}
|
|
|
|
const type =
|
|
path.length > 0 ? getFieldFromPath(ctx.root, path).type : ctx.root
|
|
const scalarFields = type.scalar
|
|
|
|
let scalarFieldsFragment: string | undefined
|
|
|
|
if (fieldNames.includes('__scalar')) {
|
|
const falsyFieldNames = new Set(
|
|
Object.keys(fields).filter((k) => !Boolean(fields[k])),
|
|
)
|
|
if (scalarFields?.length) {
|
|
ctx.fragmentCounter++
|
|
scalarFieldsFragment = `f${ctx.fragmentCounter}`
|
|
|
|
ctx.fragments.push(
|
|
`fragment ${scalarFieldsFragment} on ${
|
|
type.name
|
|
}{${scalarFields
|
|
.filter((f) => !falsyFieldNames.has(f))
|
|
.join(',')}}`,
|
|
)
|
|
}
|
|
}
|
|
|
|
const fieldsSelection = fieldNames
|
|
.filter((f) => !['__scalar', '__name'].includes(f))
|
|
.map((f) => {
|
|
const parsed = parseRequest(fields[f], ctx, [...path, f])
|
|
|
|
if (f.startsWith('on_')) {
|
|
ctx.fragmentCounter++
|
|
const implementationFragment = `f${ctx.fragmentCounter}`
|
|
|
|
const typeMatch = f.match(/^on_(.+)/)
|
|
|
|
if (!typeMatch || !typeMatch[1])
|
|
throw new Error('match failed')
|
|
|
|
ctx.fragments.push(
|
|
`fragment ${implementationFragment} on ${typeMatch[1]}${parsed}`,
|
|
)
|
|
|
|
return `...${implementationFragment}`
|
|
} else {
|
|
return `${f}${parsed}`
|
|
}
|
|
})
|
|
.concat(scalarFieldsFragment ? [`...${scalarFieldsFragment}`] : [])
|
|
.join(',')
|
|
|
|
return `{${fieldsSelection}}`
|
|
} else {
|
|
return ''
|
|
}
|
|
}
|
|
|
|
export const generateGraphqlOperation = (
|
|
operation: 'query' | 'mutation' | 'subscription',
|
|
root: LinkedType,
|
|
fields?: Fields,
|
|
): GraphqlOperation => {
|
|
const ctx: Context = {
|
|
root: root,
|
|
varCounter: 0,
|
|
variables: {},
|
|
fragmentCounter: 0,
|
|
fragments: [],
|
|
}
|
|
const result = parseRequest(fields, ctx, [])
|
|
|
|
const varNames = Object.keys(ctx.variables)
|
|
|
|
const varsString =
|
|
varNames.length > 0
|
|
? `(${varNames.map((v) => {
|
|
const variableType = ctx.variables[v].typing[1]
|
|
return `$${v}:${variableType}`
|
|
})})`
|
|
: ''
|
|
|
|
const operationName = fields?.__name || ''
|
|
|
|
return {
|
|
query: [
|
|
`${operation} ${operationName}${varsString}${result}`,
|
|
...ctx.fragments,
|
|
].join(','),
|
|
variables: Object.keys(ctx.variables).reduce<{ [name: string]: any }>(
|
|
(r, v) => {
|
|
r[v] = ctx.variables[v].value
|
|
return r
|
|
},
|
|
{},
|
|
),
|
|
...(operationName ? { operationName: operationName.toString() } : {}),
|
|
}
|
|
}
|
|
|
|
export const getFieldFromPath = (
|
|
root: LinkedType | undefined,
|
|
path: string[],
|
|
) => {
|
|
let current: LinkedField | undefined
|
|
|
|
if (!root) throw new Error('root type is not provided')
|
|
|
|
if (path.length === 0) throw new Error(`path is empty`)
|
|
|
|
path.forEach((f) => {
|
|
const type = current ? current.type : root
|
|
|
|
if (!type.fields)
|
|
throw new Error(`type \`${type.name}\` does not have fields`)
|
|
|
|
const possibleTypes = Object.keys(type.fields)
|
|
.filter((i) => i.startsWith('on_'))
|
|
.reduce(
|
|
(types, fieldName) => {
|
|
const field = type.fields && type.fields[fieldName]
|
|
if (field) types.push(field.type)
|
|
return types
|
|
},
|
|
[type],
|
|
)
|
|
|
|
let field: LinkedField | null = null
|
|
|
|
possibleTypes.forEach((type) => {
|
|
const found = type.fields && type.fields[f]
|
|
if (found) field = found
|
|
})
|
|
|
|
if (!field)
|
|
throw new Error(
|
|
`type \`${type.name}\` does not have a field \`${f}\``,
|
|
)
|
|
|
|
current = field
|
|
})
|
|
|
|
return current as LinkedField
|
|
}
|