## 1. The `twenty-client-sdk` Package (Source of Truth)
The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`
## 2. Generation & Upload (Server-Side, at Migration Time)
**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.
**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database
## 3. Invalidation Signal
The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive
Default is `false` so existing applications without a generated client
aren't affected.
## 4a. Logic Functions — Local Driver
**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`
**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.
## 4b. Logic Functions — Lambda Driver
**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`
**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).
## 5. Front Components
Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.
SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):
**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.
**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`
**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache
This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.
## Summary Diagram
```
app:build (SDK)
└─ twenty-client-sdk stub (metadata=real, core=stub)
│
▼
WorkspaceMigrationRunnerService.run()
└─ SdkClientGenerationService.generateAndStore()
├─ Copy stub package (package.json + dist/)
├─ replaceCoreClient() → regenerate core.mjs/core.cjs
├─ Zip entire package → upload to S3
└─ Set isSdkLayerStale = true
│
┌────────┴────────────────────┐
▼ ▼
Logic Functions Front Components
│ │
├─ Local Driver ├─ GET /rest/sdk-client/:appId/core
│ └─ downloadAndExtract │ → core.mjs from archive
│ → symlink into │
│ node_modules ├─ Host (useApplicationSdkClient)
│ │ ├─ Fetch SDK modules
└─ Lambda Driver │ ├─ Create blob URLs
└─ downloadArchiveBuffer │ └─ Cache in Jotai atom family
→ reprefixZipEntries │
→ publish as Lambda ├─ GET /rest/front-components/:id
layer │ → raw .mjs (no bundling)
│
└─ Worker (browser)
├─ Fetch component .mjs
├─ Rewrite imports → blob URLs
└─ import() rewritten source
```
## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server
---------
Co-authored-by: Charles Bochet <[email protected]>
280 lines
7.4 KiB
TypeScript
280 lines
7.4 KiB
TypeScript
import * as fs from 'fs-extra';
|
|
import { join } from 'path';
|
|
|
|
const SEED_API_KEY =
|
|
'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIyMDIwMjAyMC1lNmI1LTQ2ODAtOGEzMi1iODIwOTczNzE1NmIiLCJ1c2VySWQiOiIyMDIwMjAyMC1lNmI1LTQ2ODAtOGEzMi1iODIwOTczNzE1NmIiLCJ3b3Jrc3BhY2VJZCI6IjIwMjAyMDIwLTFjMjUtNGQwMi1iZjI1LTZhZWNjZjdlYTQxOSIsIndvcmtzcGFjZU1lbWJlcklkIjoiMjAyMDIwMjAtNDYzZi00MzViLTgyOGMtMTA3ZTAwN2EyNzExIiwidXNlcldvcmtzcGFjZUlkIjoiMjAyMDIwMjAtMWU3Yy00M2Q5LWE1ZGItNjg1YjUwNjlkODE2IiwidHlwZSI6IkFDQ0VTUyIsImF1dGhQcm92aWRlciI6InBhc3N3b3JkIiwiaWF0IjoxNzUxMjgxNzA0LCJleHAiOjIwNjY4NTc3MDR9.HMGqCsVlOAPVUBhKSGlD1X86VoHKt4LIUtET3CGIdik';
|
|
|
|
export const scaffoldIntegrationTest = async ({
|
|
appDirectory,
|
|
sourceFolderPath,
|
|
}: {
|
|
appDirectory: string;
|
|
sourceFolderPath: string;
|
|
}) => {
|
|
await createIntegrationTest({
|
|
appDirectory: sourceFolderPath,
|
|
fileFolder: '__tests__',
|
|
fileName: 'app-install.integration-test.ts',
|
|
});
|
|
|
|
await createSetupTest({
|
|
appDirectory: sourceFolderPath,
|
|
fileFolder: '__tests__',
|
|
fileName: 'setup-test.ts',
|
|
});
|
|
|
|
await createVitestConfig(appDirectory);
|
|
await createTsconfigSpec(appDirectory);
|
|
await createGithubWorkflow(appDirectory);
|
|
};
|
|
|
|
const createVitestConfig = async (appDirectory: string) => {
|
|
const content = `import tsconfigPaths from 'vite-tsconfig-paths';
|
|
import { defineConfig } from 'vitest/config';
|
|
|
|
export default defineConfig({
|
|
plugins: [
|
|
tsconfigPaths({
|
|
projects: ['tsconfig.spec.json'],
|
|
ignoreConfigErrors: true,
|
|
}),
|
|
],
|
|
test: {
|
|
testTimeout: 120_000,
|
|
hookTimeout: 120_000,
|
|
include: ['src/**/*.integration-test.ts'],
|
|
setupFiles: ['src/__tests__/setup-test.ts'],
|
|
env: {
|
|
TWENTY_API_KEY:
|
|
'${SEED_API_KEY}',
|
|
},
|
|
},
|
|
});
|
|
`;
|
|
|
|
await fs.writeFile(join(appDirectory, 'vitest.config.ts'), content);
|
|
};
|
|
|
|
const createTsconfigSpec = async (appDirectory: string) => {
|
|
const tsconfigSpec = {
|
|
extends: './tsconfig.json',
|
|
compilerOptions: {
|
|
composite: true,
|
|
types: ['vitest/globals'],
|
|
},
|
|
include: ['src/**/*.ts', 'src/**/*.tsx'],
|
|
exclude: ['node_modules', 'dist'],
|
|
};
|
|
|
|
await fs.writeFile(
|
|
join(appDirectory, 'tsconfig.spec.json'),
|
|
JSON.stringify(tsconfigSpec, null, 2),
|
|
);
|
|
|
|
const tsconfigPath = join(appDirectory, 'tsconfig.json');
|
|
const tsconfig = await fs.readJson(tsconfigPath);
|
|
|
|
tsconfig.references = [{ path: './tsconfig.spec.json' }];
|
|
|
|
await fs.writeFile(tsconfigPath, JSON.stringify(tsconfig, null, 2));
|
|
};
|
|
|
|
const createSetupTest = async ({
|
|
appDirectory,
|
|
fileFolder,
|
|
fileName,
|
|
}: {
|
|
appDirectory: string;
|
|
fileFolder?: string;
|
|
fileName: string;
|
|
}) => {
|
|
const content = `import * as fs from 'fs';
|
|
import * as os from 'os';
|
|
import * as path from 'path';
|
|
import { beforeAll } from 'vitest';
|
|
|
|
const TWENTY_API_URL = process.env.TWENTY_API_URL ?? 'http://localhost:2020';
|
|
const TEST_CONFIG_DIR = path.join(os.tmpdir(), '.twenty-sdk-test');
|
|
|
|
const assertServerIsReachable = async () => {
|
|
let response: Response;
|
|
|
|
try {
|
|
response = await fetch(\`\${TWENTY_API_URL}/healthz\`);
|
|
} catch {
|
|
throw new Error(
|
|
\`Twenty server is not reachable at \${TWENTY_API_URL}. \` +
|
|
'Make sure the server is running before executing integration tests.',
|
|
);
|
|
}
|
|
|
|
if (!response.ok) {
|
|
throw new Error(\`Server at \${TWENTY_API_URL} returned \${response.status}\`);
|
|
}
|
|
};
|
|
|
|
beforeAll(async () => {
|
|
await assertServerIsReachable();
|
|
|
|
fs.mkdirSync(TEST_CONFIG_DIR, { recursive: true });
|
|
|
|
const configFile = {
|
|
remotes: {
|
|
local: {
|
|
apiUrl: process.env.TWENTY_API_URL,
|
|
apiKey: process.env.TWENTY_API_KEY,
|
|
},
|
|
},
|
|
defaultRemote: 'local',
|
|
};
|
|
|
|
fs.writeFileSync(
|
|
path.join(TEST_CONFIG_DIR, 'config.json'),
|
|
JSON.stringify(configFile, null, 2),
|
|
);
|
|
});
|
|
`;
|
|
|
|
await fs.ensureDir(join(appDirectory, fileFolder ?? ''));
|
|
await fs.writeFile(join(appDirectory, fileFolder ?? '', fileName), content);
|
|
};
|
|
|
|
const createIntegrationTest = async ({
|
|
appDirectory,
|
|
fileFolder,
|
|
fileName,
|
|
}: {
|
|
appDirectory: string;
|
|
fileFolder?: string;
|
|
fileName: string;
|
|
}) => {
|
|
const content = `import { APPLICATION_UNIVERSAL_IDENTIFIER } from 'src/application-config';
|
|
import { appBuild, appDeploy, appInstall, appUninstall } from 'twenty-sdk/cli';
|
|
import { MetadataApiClient } from 'twenty-client-sdk/metadata';
|
|
import { afterAll, beforeAll, describe, expect, it } from 'vitest';
|
|
|
|
const APP_PATH = process.cwd();
|
|
|
|
describe('App installation', () => {
|
|
beforeAll(async () => {
|
|
const buildResult = await appBuild({
|
|
appPath: APP_PATH,
|
|
tarball: true,
|
|
onProgress: (message: string) => console.log(\`[build] \${message}\`),
|
|
});
|
|
|
|
if (!buildResult.success) {
|
|
throw new Error(
|
|
\`Build failed: \${buildResult.error?.message ?? 'Unknown error'}\`,
|
|
);
|
|
}
|
|
|
|
const deployResult = await appDeploy({
|
|
tarballPath: buildResult.data.tarballPath!,
|
|
onProgress: (message: string) => console.log(\`[deploy] \${message}\`),
|
|
});
|
|
|
|
if (!deployResult.success) {
|
|
throw new Error(
|
|
\`Deploy failed: \${deployResult.error?.message ?? 'Unknown error'}\`,
|
|
);
|
|
}
|
|
|
|
const installResult = await appInstall({ appPath: APP_PATH });
|
|
|
|
if (!installResult.success) {
|
|
throw new Error(
|
|
\`Install failed: \${installResult.error?.message ?? 'Unknown error'}\`,
|
|
);
|
|
}
|
|
});
|
|
|
|
afterAll(async () => {
|
|
const uninstallResult = await appUninstall({ appPath: APP_PATH });
|
|
|
|
if (!uninstallResult.success) {
|
|
console.warn(
|
|
\`App uninstall failed: \${uninstallResult.error?.message ?? 'Unknown error'}\`,
|
|
);
|
|
}
|
|
});
|
|
|
|
it('should find the installed app in the applications list', async () => {
|
|
const metadataClient = new MetadataApiClient();
|
|
|
|
const result = await metadataClient.query({
|
|
findManyApplications: {
|
|
id: true,
|
|
name: true,
|
|
universalIdentifier: true,
|
|
},
|
|
});
|
|
|
|
const installedApp = result.findManyApplications.find(
|
|
(application: { universalIdentifier: string }) =>
|
|
application.universalIdentifier ===
|
|
APPLICATION_UNIVERSAL_IDENTIFIER,
|
|
);
|
|
|
|
expect(installedApp).toBeDefined();
|
|
});
|
|
});
|
|
`;
|
|
|
|
await fs.ensureDir(join(appDirectory, fileFolder ?? ''));
|
|
await fs.writeFile(join(appDirectory, fileFolder ?? '', fileName), content);
|
|
};
|
|
|
|
const DEFAULT_TWENTY_VERSION = 'latest';
|
|
|
|
const createGithubWorkflow = async (appDirectory: string) => {
|
|
const content = `name: CI
|
|
|
|
on:
|
|
push:
|
|
branches:
|
|
- main
|
|
pull_request: {}
|
|
|
|
env:
|
|
TWENTY_VERSION: ${DEFAULT_TWENTY_VERSION}
|
|
|
|
jobs:
|
|
test:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@v4
|
|
|
|
- name: Spawn Twenty instance
|
|
id: twenty
|
|
uses: twentyhq/twenty/.github/actions/spawn-twenty-docker-image@main
|
|
with:
|
|
twenty-version: \${{ env.TWENTY_VERSION }}
|
|
github-token: \${{ secrets.GITHUB_TOKEN }}
|
|
|
|
- name: Enable Corepack
|
|
run: corepack enable
|
|
|
|
- name: Setup Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version-file: '.nvmrc'
|
|
cache: 'yarn'
|
|
|
|
- name: Install dependencies
|
|
run: yarn install --immutable
|
|
|
|
- name: Run integration tests
|
|
run: yarn test
|
|
env:
|
|
TWENTY_API_URL: \${{ steps.twenty.outputs.server-url }}
|
|
TWENTY_API_KEY: \${{ steps.twenty.outputs.access-token }}
|
|
`;
|
|
|
|
const workflowDir = join(appDirectory, '.github', 'workflows');
|
|
|
|
await fs.ensureDir(workflowDir);
|
|
await fs.writeFile(join(workflowDir, 'ci.yml'), content);
|
|
};
|