Files
twenty/.github/workflows/ci-create-app-e2e.yaml
T
4ea2e32366 Refactor twenty client sdk provisioning for logic function and front-component (#18544)
## 1. The `twenty-client-sdk` Package (Source of Truth)

The monorepo package at `packages/twenty-client-sdk` ships with:
- A **pre-built metadata client** (static, generated from a fixed
schema)
- A **stub core client** that throws at runtime (`CoreApiClient was not
generated...`)
- Both ESM (`.mjs`) and CJS (`.cjs`) bundles in `dist/`
- A `package.json` with proper `exports` map for
`twenty-client-sdk/core`, `twenty-client-sdk/metadata`, and
`twenty-client-sdk/generate`

## 2. Generation & Upload (Server-Side, at Migration Time)

**When**: `WorkspaceMigrationRunnerService.run()` executes after a
metadata schema change.

**What happens in `SdkClientGenerationService.generateAndStore()`**:
1. Copies the stub `twenty-client-sdk` package from the server's assets
(resolved via `SDK_CLIENT_PACKAGE_DIRNAME` — from
`dist/assets/twenty-client-sdk/` in production, or from `node_modules`
in dev)
2. Filters out `node_modules/` and `src/` during copy — only
`package.json` + `dist/` are kept (like an npm publish)
3. Calls `replaceCoreClient()` which uses `@genql/cli` to introspect the
**application-scoped** GraphQL schema and generates a real
`CoreApiClient`, then compiles it to ESM+CJS and overwrites
`dist/core.mjs` and `dist/core.cjs`
4. Archives the **entire package** (with `package.json` + `dist/`) into
`twenty-client-sdk.zip`
5. Uploads the single archive to S3 under
`FileFolder.GeneratedSdkClient`
6. Sets `isSdkLayerStale = true` on the `ApplicationEntity` in the
database

## 3. Invalidation Signal

The `isSdkLayerStale` boolean column on `ApplicationEntity` is the
invalidation mechanism:
- **Set to `true`** by `generateAndStore()` after uploading a new client
archive
- **Checked** by both logic function drivers before execution — if
`true`, they rebuild their local layer
- **Set back to `false`** by `markSdkLayerFresh()` after the driver has
successfully consumed the new archive

Default is `false` so existing applications without a generated client
aren't affected.

## 4a. Logic Functions — Local Driver

**`ensureSdkLayer()`** is called before every execution:
1. Checks if the local SDK layer directory exists AND `isSdkLayerStale`
is `false` → early return
2. Otherwise, cleans the local layer directory
3. Calls `downloadAndExtractToPackage()` which streams the zip from S3
directly to disk and extracts the full package into
`<tmpdir>/sdk/<workspaceId>-<appId>/node_modules/twenty-client-sdk/`
4. Calls `markSdkLayerFresh()` to set `isSdkLayerStale = false`

**At execution time**, `assembleNodeModules()` symlinks everything from
the deps layer's `node_modules/` **except** `twenty-client-sdk`, which
is symlinked from the SDK layer instead. This ensures the logic
function's `import ... from 'twenty-client-sdk/core'` resolves to the
generated client.

## 4b. Logic Functions — Lambda Driver

**`ensureSdkLayer()`** is called during `build()`:
1. Checks if `isSdkLayerStale` is `false` and an existing Lambda layer
ARN exists → early return
2. Otherwise, deletes all existing layer versions for this SDK layer
name
3. Calls `downloadArchiveBuffer()` to get the raw zip from S3 (no disk
extraction)
4. Calls `reprefixZipEntries()` which streams the zip entries into a
**new zip** with the path prefix
`nodejs/node_modules/twenty-client-sdk/` — this is the Lambda layer
convention path. All done in memory, no disk round-trip
5. Publishes the re-prefixed zip as a new Lambda layer via
`publishLayer()`
6. Calls `markSdkLayerFresh()`

**At function creation**, the Lambda is created with **two layers**:
`[depsLayerArn, sdkLayerArn]`. The SDK layer is listed last so it
overwrites the stub `twenty-client-sdk` from the deps layer (later
layers take precedence in Lambda's `/opt` merge).

## 5. Front Components

Front components are built by `app:build` with `twenty-client-sdk/core`
and `twenty-client-sdk/metadata` as **esbuild externals**. The stored
`.mjs` in S3 has unresolved bare import specifiers like `import {
CoreApiClient } from 'twenty-client-sdk/core'`.

SDK import resolution is split between the **frontend host** (fetching &
caching SDK modules) and the **Web Worker** (rewriting imports):

**Server endpoints**:
- `GET /rest/front-components/:id` —
`FrontComponentService.getBuiltComponentStream()` returns the **raw
`.mjs`** directly from file storage. No bundling, no SDK injection.
- `GET /rest/sdk-client/:applicationId/:moduleName` —
`SdkClientController` reads a single file (e.g. `dist/core.mjs`) from
the generated SDK archive via
`SdkClientGenerationService.readFileFromArchive()` and serves it as
JavaScript.

**Frontend host** (`FrontComponentRenderer` in `twenty-front`):
1. Queries `FindOneFrontComponent` which returns `applicationId`,
`builtComponentChecksum`, `usesSdkClient`, and `applicationTokenPair`
2. If `usesSdkClient` is `true`, renders
`FrontComponentRendererWithSdkClient` which calls the
`useApplicationSdkClient` hook
3. `useApplicationSdkClient({ applicationId, accessToken })` checks the
Jotai atom family cache for existing blob URLs. On cache miss, fetches
both SDK modules from `GET /rest/sdk-client/:applicationId/core` and
`/metadata`, creates **blob URLs** for each, and stores them in the atom
family
4. Once the blob URLs are cached, passes them as `sdkClientUrls`
(already blob URLs, not server URLs) to `SharedFrontComponentRenderer` →
`FrontComponentWorkerEffect` → worker's `render()` call via
`HostToWorkerRenderContext`

**Worker** (`remote-worker.ts` in `twenty-sdk`):
1. Fetches the raw component `.mjs` source as text
2. If `sdkClientUrls` are provided and the source contains SDK import
specifiers (`twenty-client-sdk/core`, `twenty-client-sdk/metadata`),
**rewrites** the bare specifiers to the blob URLs received from the host
(e.g. `'twenty-client-sdk/core'` → `'blob:...'`)
3. Creates a blob URL for the rewritten source and `import()`s it
4. Revokes only the component blob URL after the module is loaded — the
SDK blob URLs are owned and managed by the host's Jotai cache

This approach eliminates server-side esbuild bundling on every request,
caches SDK modules per application in the frontend, and keeps the
worker's job to a simple string rewrite.

## Summary Diagram

```
app:build (SDK)
  └─ twenty-client-sdk stub (metadata=real, core=stub)
       │
       ▼
WorkspaceMigrationRunnerService.run()
  └─ SdkClientGenerationService.generateAndStore()
       ├─ Copy stub package (package.json + dist/)
       ├─ replaceCoreClient() → regenerate core.mjs/core.cjs
       ├─ Zip entire package → upload to S3
       └─ Set isSdkLayerStale = true
              │
     ┌────────┴────────────────────┐
     ▼                             ▼
Logic Functions               Front Components
     │                             │
     ├─ Local Driver               ├─ GET /rest/sdk-client/:appId/core
     │   └─ downloadAndExtract     │    → core.mjs from archive
     │      → symlink into         │
     │        node_modules         ├─ Host (useApplicationSdkClient)
     │                             │    ├─ Fetch SDK modules
     └─ Lambda Driver              │    ├─ Create blob URLs
         └─ downloadArchiveBuffer  │    └─ Cache in Jotai atom family
            → reprefixZipEntries   │
            → publish as Lambda    ├─ GET /rest/front-components/:id
              layer                │    → raw .mjs (no bundling)
                                   │
                                   └─ Worker (browser)
                                        ├─ Fetch component .mjs
                                        ├─ Rewrite imports → blob URLs
                                        └─ import() rewritten source
```

## Next PR
- Estimate perf improvement by implementing a redis caching for front
component client storage ( we don't even cache front comp initially )
- Implem frontent blob invalidation sse event from server

---------

Co-authored-by: Charles Bochet <[email protected]>
2026-03-24 18:10:25 +00:00

199 lines
7.0 KiB
YAML

name: CI Create App E2E
on:
push:
branches:
- main
pull_request:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
jobs:
changed-files-check:
uses: ./.github/workflows/changed-files.yaml
with:
files: |
packages/create-twenty-app/**
packages/twenty-sdk/**
packages/twenty-client-sdk/**
packages/twenty-shared/**
packages/twenty-server/**
!packages/create-twenty-app/package.json
!packages/twenty-sdk/package.json
!packages/twenty-client-sdk/package.json
!packages/twenty-shared/package.json
!packages/twenty-server/package.json
create-app-e2e:
needs: changed-files-check
if: needs.changed-files-check.outputs.any_changed == 'true'
timeout-minutes: 30
runs-on: ubuntu-latest-4-cores
services:
postgres:
image: twentycrm/twenty-postgres-spilo
env:
PGUSER_SUPERUSER: postgres
PGPASSWORD_SUPERUSER: postgres
ALLOW_NOSSL: 'true'
SPILO_PROVIDER: 'local'
ports:
- 5432:5432
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
redis:
image: redis
ports:
- 6379:6379
steps:
- name: Fetch custom Github Actions and base branch history
uses: actions/checkout@v4
with:
fetch-depth: 10
- name: Install dependencies
uses: ./.github/actions/yarn-install
- name: Set CI version and prepare packages for publish
run: |
CI_VERSION="0.0.0-ci.$(date +%s)"
echo "CI_VERSION=$CI_VERSION" >> $GITHUB_ENV
npx nx run-many -t set-local-version -p twenty-sdk twenty-client-sdk create-twenty-app --releaseVersion=$CI_VERSION
- name: Build packages
run: |
npx nx build twenty-sdk
npx nx build twenty-client-sdk
npx nx build create-twenty-app
- name: Install and start Verdaccio
run: |
npx verdaccio --config .github/verdaccio-config.yaml &
for i in $(seq 1 30); do
if curl -s http://localhost:4873 > /dev/null 2>&1; then
echo "Verdaccio is ready"
break
fi
echo "Waiting for Verdaccio... ($i/30)"
sleep 1
done
- name: Publish packages to local registry
run: |
npm set //localhost:4873/:_authToken "ci-auth-token"
for pkg in twenty-sdk twenty-client-sdk create-twenty-app; do
cd packages/$pkg
npm publish --registry http://localhost:4873 --tag ci
cd ../..
done
- name: Scaffold app using published create-twenty-app
run: |
npm install -g create-twenty-app@$CI_VERSION --registry http://localhost:4873
create-twenty-app --version
mkdir -p /tmp/e2e-test-workspace
cd /tmp/e2e-test-workspace
create-twenty-app test-app --exhaustive --display-name "Test App" --description "E2E test app" --skip-local-instance
- name: Install scaffolded app dependencies
run: |
cd /tmp/e2e-test-workspace/test-app
echo 'npmRegistryServer: "http://localhost:4873"' >> .yarnrc.yml
echo 'unsafeHttpWhitelist: ["localhost"]' >> .yarnrc.yml
YARN_ENABLE_IMMUTABLE_INSTALLS=false yarn install --no-immutable
- name: Verify installed app versions
run: |
cd /tmp/e2e-test-workspace/test-app
echo "--- Checking package.json references correct SDK version ---"
node -e "
const pkg = require('./package.json');
const sdkVersion = pkg.devDependencies['twenty-sdk'];
if (!sdkVersion.startsWith('0.0.0-ci.')) {
console.error('Expected twenty-sdk version to start with 0.0.0-ci., got:', sdkVersion);
process.exit(1);
}
console.log('SDK version in scaffolded app:', sdkVersion);
"
- name: Verify SDK CLI is available
run: |
cd /tmp/e2e-test-workspace/test-app
npx --no-install twenty --version
- name: Setup server environment
run: npx nx reset:env:e2e-testing-server twenty-server
- name: Build server
run: npx nx build twenty-server
- name: Create and setup database
run: |
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -d postgres -c 'CREATE DATABASE "default";'
PGPASSWORD=postgres psql -h localhost -p 5432 -U postgres -d postgres -c 'CREATE DATABASE "test";'
npx nx run twenty-server:database:reset
- name: Start server
run: |
npx nx start twenty-server &
echo "Waiting for server to be ready..."
timeout 60 bash -c 'until curl -s http://localhost:3000/health; do sleep 2; done'
- name: Authenticate with twenty-server
env:
SEED_API_KEY: 'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIyMDIwMjAyMC1lNmI1LTQ2ODAtOGEzMi1iODIwOTczNzE1NmIiLCJ1c2VySWQiOiIyMDIwMjAyMC1lNmI1LTQ2ODAtOGEzMi1iODIwOTczNzE1NmIiLCJ3b3Jrc3BhY2VJZCI6IjIwMjAyMDIwLTFjMjUtNGQwMi1iZjI1LTZhZWNjZjdlYTQxOSIsIndvcmtzcGFjZU1lbWJlcklkIjoiMjAyMDIwMjAtNDYzZi00MzViLTgyOGMtMTA3ZTAwN2EyNzExIiwidXNlcldvcmtzcGFjZUlkIjoiMjAyMDIwMjAtMWU3Yy00M2Q5LWE1ZGItNjg1YjUwNjlkODE2IiwidHlwZSI6IkFDQ0VTUyIsImF1dGhQcm92aWRlciI6InBhc3N3b3JkIiwiaWF0IjoxNzUxMjgxNzA0LCJleHAiOjIwNjY4NTc3MDR9.HMGqCsVlOAPVUBhKSGlD1X86VoHKt4LIUtET3CGIdik'
run: |
cd /tmp/e2e-test-workspace/test-app
npx --no-install twenty remote add --token $SEED_API_KEY --url http://localhost:3000
- name: Deploy scaffolded app
run: |
cd /tmp/e2e-test-workspace/test-app
npx --no-install twenty deploy
- name: Install scaffolded app
run: |
cd /tmp/e2e-test-workspace/test-app
npx --no-install twenty install
- name: Execute hello-world logic function
run: |
cd /tmp/e2e-test-workspace/test-app
EXEC_OUTPUT=$(npx --no-install twenty exec --functionName hello-world-logic-function)
echo "$EXEC_OUTPUT"
echo "$EXEC_OUTPUT" | grep -q "Hello, World!"
- name: Execute create-hello-world-company logic function
run: |
cd /tmp/e2e-test-workspace/test-app
EXEC_OUTPUT=$(npx --no-install twenty exec --functionName create-hello-world-company)
echo "$EXEC_OUTPUT"
echo "$EXEC_OUTPUT" | grep -q "Created company"
- name: Run scaffolded app integration test
env:
TWENTY_API_URL: http://localhost:3000
run: |
cd /tmp/e2e-test-workspace/test-app
yarn test
ci-create-app-e2e-status-check:
if: always() && !cancelled()
timeout-minutes: 5
runs-on: ubuntu-latest
needs: [changed-files-check, create-app-e2e]
steps:
- name: Fail job if any needs failed
if: contains(needs.*.result, 'failure')
run: exit 1