Commit Graph
17 Commits
Author SHA1 Message Date
martmullandGitHub 9162685b2e Reorganize logic function files (#17766)
reorganize according to

<img width="1243" height="725" alt="Pasted Graphic"
src="https://github.com/user-attachments/assets/ba65dd10-8eec-4b13-ad49-9726edd3b79c"
/>

Not working yet
2026-02-09 12:36:39 +01:00
Charles BochetandGitHub 67a98f77e3 Refactor workflow-logic-function-interaction (#17699)
# Refactor workflow–logic function interaction

## Why

Workflow code steps and standalone logic functions shared the same build
layer and DB layer, which blurred two use cases: code steps belong to a
workflow version; standalone functions are deployable units. That made
workflow code steps harder to own and evolve.

## Goal

Treat code steps as **workflow-owned**: build and run them in workflow
context, and expose workflow-scoped APIs so the editor can load, test,
and save code step source without going through the generic
logic-function layer.
2026-02-05 00:46:55 +00:00
Charles BochetandGitHub 867b03393b Backfill package json for custom and standard app (#17681)
# Backfill application package files for custom and standard apps

- Backfill `package.json` / `yarn.lock` (and related fields) for
existing workspaces; new standard/custom apps get default dependency
files.
- Default package files under
`application/constants/default-package-files/`; util with hardcoded
checksums (comment on how to regenerate).
- New **FileFolder.Dependencies** for app dependency files;
**writeFile_v2** accepts optional `queryRunner` for transactional
writes.

Usages:
- **Upgrade command** `upgrade:1-17:backfill-application-package-files`:
standard/custom apps → default files; other apps → from logic function
layer.
- **Workspace creation**: create workspace with
`workspaceCustomApplicationId` first, then create application (enables
same-transaction insert). Migration makes workspace/application/file FKs
deferrable.
-  dev seeder
2026-02-03 21:23:29 +01:00
Charles BochetandGitHub 4a770eafa1 Rework logic function module (#17588)
core-modules/logic-function/
├── logic-function.module.ts
├── logic-function-executor/
│   ├── logic-function-executor.module.ts
│   ├── commands/
│   │   └── add-packages.command.ts
│   ├── constants/
│   │   └── logic-function-executor.constants.ts
│   ├── factories/
│   │   └── logic-function-module.factory.ts
│   ├── interfaces/
│   │   └── logic-function-executor.interface.ts
│   └── services/
│       └── logic-function-executor.service.ts
├── logic-function-build/
│   ├── logic-function-build.module.ts
│   ├── services/
│   │   └── logic-function-build.service.ts
│   └── utils/
│       └── get-logic-function-base-folder-path.util.ts
├── logic-function-drivers/
│   ├── logic-function-drivers.module.ts
│   ├── constants/
│   │   └── ...
│   ├── drivers/
│   │   ├── disabled.driver.ts
│   │   ├── lambda.driver.ts
│   │   └── local.driver.ts
│   ├── interfaces/
│   │   └── logic-function-executor-driver.interface.ts
│   ├── layers/
│   │   └── ...
│   └── utils/
│       └── ...
├── logic-function-layer/
│   ├── logic-function-layer.module.ts
│   └── services/
│       └── logic-function-layer.service.ts
└── logic-function-trigger/
    ├── logic-function-trigger.module.ts
    ├── jobs/
    │   └── logic-function-trigger.job.ts
    └── triggers/
        ├── cron/
        ├── database-event/
        └── route/
            ├── exceptions/
            ├── services/
            │   └── route-trigger.service.ts
            └── utils/
2026-01-30 19:28:20 +01:00
Charles BochetandGitHub da6f1bbef3 Rename serverlessFunction to logicFunction (#17494)
## Summary

Rename "Serverless Function" to "Logic Function" across the codebase for
clearer naming.

### Environment Variable Changes

| Old | New |
|-----|-----|
| `SERVERLESS_TYPE` | `LOGIC_FUNCTION_TYPE` |
| `SERVERLESS_LAMBDA_REGION` | `LOGIC_FUNCTION_LAMBDA_REGION` |
| `SERVERLESS_LAMBDA_ROLE` | `LOGIC_FUNCTION_LAMBDA_ROLE` |
| `SERVERLESS_LAMBDA_SUBHOSTING_URL` |
`LOGIC_FUNCTION_LAMBDA_SUBHOSTING_URL` |
| `SERVERLESS_LAMBDA_ACCESS_KEY_ID` |
`LOGIC_FUNCTION_LAMBDA_ACCESS_KEY_ID` |
| `SERVERLESS_LAMBDA_SECRET_ACCESS_KEY` |
`LOGIC_FUNCTION_LAMBDA_SECRET_ACCESS_KEY` |

### Breaking Changes

- Environment variables must be updated in production deployments
- Database migration renames `serverlessFunction` → `logicFunction`
tables
2026-01-28 01:42:19 +01:00
martmullandGitHub f92c8a98a4 2114 extensibility manage serverless execution from built code (#17317)
Summary

- Serverless functions are now built when created/updated instead of at
execution time
  - Added builtHandlerPath field to track pre-built function artifacts
- Renamed base-typescript-project to seed-project with pre-built ESM
output included
- Renamed file folder enums for consistency (Functions → BuiltFunction,
SourceCode → Source)
  - supports backwards compatibility with existing serverless functions

Tested with all combinations
- storage : local driver and S3 driver
- serverless : local driver and lambda driver
2026-01-22 12:56:44 +01:00
Félix MalfaitandGitHub 5dfb66917c Upgrade NestJS from 10.x to 11.x (#15836)
## Overview
This PR upgrades all NestJS dependencies from version 10.x to 11.x,
following the [official migration
guide](https://docs.nestjs.com/migration-guide). This builds on top of
the v9 to v10 upgrade completed in PR #15835.

## Changes

### Dependencies Updated
**Core packages (10.x → 11.x):**
- `@nestjs/common`: 10.4.16 → 11.0.8
- `@nestjs/core`: 10.4.16 → 11.0.8
- `@nestjs/platform-express`: 10.4.16 → 11.0.8
- `@nestjs/config`: 3.2.3 → 3.3.0
- `@nestjs/passport`: 10.0.3 → 11.0.0
- `@nestjs/axios`: 3.0.2 → 3.1.2
- `@nestjs/schedule`: ^3.0.0 → ^4.1.1
- `@nestjs/serve-static`: 4.0.2 → 5.0.1
- `@nestjs/cache-manager`: ^2.2.1 → ^2.3.0
- `@nestjs/jwt`: 10.2.0 → 11.0.0
- `@nestjs/typeorm`: 10.0.2 → 11.0.0
- `@nestjs/terminus`: 11.0.0 (already on v11)
- `@nestjs/event-emitter`: 2.1.0 (compatible)

**DevDependencies:**
- `@nestjs/testing`: ^10.4.16 → ^11.0.8
- `@nestjs/schematics`: ^10.1.0 → ^11.0.2
- `@nestjs/cli`: 10.3.0 → 11.0.0

### Code Changes
**Fixed: TwentyConfigModule conditional imports**
- Updated `TwentyConfigModule.forRoot()` to use spread operator for
conditional imports
- Fixes TypeScript error with NestJS 11's stricter DynamicModule type
checking

**Cleanup: Removed unused package**
- Removed `@revertdotdev/revert-react` (not being used anywhere in the
codebase)

## Breaking Changes Addressed

### 1.  Reflector Type Inference
- **Impact**: None - codebase only uses `reflector.get()` method
- **Analysis**: Does not use `getAllAndMerge()` or `getAllAndOverride()`
(the methods with breaking changes)
- **Files reviewed**: feature-flag.guard.ts,
message-queue-metadata.accessor.ts,
workspace-query-hook-metadata.accessor.ts

### 2.  Lifecycle Hooks Execution Order
- **Change**: Termination hooks (`OnModuleDestroy`,
`BeforeApplicationShutdown`, `OnApplicationShutdown`) now execute in
REVERSE order
- **Analysis**: Reviewed all lifecycle hook implementations
  - Redis client cleanup
  - Database connection cleanup (GlobalWorkspaceDataSource)
  - BullMQ queue/worker cleanup
  - Cache storage cleanup
- **Result**: Dependency order is safe - services using connections
clean up before the connections themselves

### 3.  Middleware Registration Order
- **Change**: Global middleware now executes first regardless of import
order
- **Analysis**: Middleware is not registered as global, so execution
order remains consistent
- **Files reviewed**: app.module.ts, middleware.module.ts

## Testing

All tests passing and build successful:

**Unit Tests (283+ tests):**
-  Health module: 38 tests passed
-  Auth module: 115 tests passed (passport v11 integration)
-  REST API: 90 tests passed (middleware and express platform)
-  Feature flags: 17 tests passed (Reflector usage)
-  Workspace: 23 tests passed

**Build & Quality:**
-  Type checking: Passed
-  Linting: Passed
-  Build: 3,683 files compiled successfully

## Verification

Tested critical NestJS functionality:
-  Authentication & Security (JWT, OAuth, guards)
-  HTTP Platform (Express integration, REST endpoints)
-  Dependency Injection (Services, factories, providers)
-  Cache Management (Redis with @nestjs/cache-manager)
-  GraphQL (Query runners, resolvers)
-  Configuration (Environment config)
-  Scheduling (Cron jobs with @nestjs/schedule v4)
-  Lifecycle Hooks (Module initialization and cleanup)
-  Reflector (Metadata reflection in guards)

## Related PRs
- #15835 - Upgrade NestJS from 9.x to 10.x (completed)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Upgrades NestJS to v11 and updates routing patterns, auth strategies,
GraphQL schema options, and build/dist paths (scripts, Docker, Nx,
migrations, assets), plus enables Devtools in development.
> 
> - **Backend (NestJS 11 upgrade)**:
> - Bump `@nestjs/*` packages (core, platform-express, jwt, passport,
typeorm, serve-static, schedule, cli/testing/schematics) to v11.
> - Update REST/route-trigger/file controllers to new wildcard syntax
(`*path`).
> - Refactor OAuth (Google/Microsoft) and SAML strategies (abstract base
+ explicit `validate`); minor typings.
>   - Enable `DevtoolsModule` in development.
> - **GraphQL**:
> - Add `buildSchemaOptions.orphanedTypes` for client-config types; keep
Yoga/Sentry setup.
> - **Build/Runtime & Config**:
> - Standardize dist layout (remove `src` in paths): update scripts,
Docker `CMD`, Nx `project.json`, render scripts, TypeORM migration
paths, asset resolution.
> - Adjust `nest-cli.json` (watchOptions, asset globs, migrations
outDir, monorepo/root).
> - Improve config module imports (spread conditional); tsconfig
excludes `node_modules`.
>   - Minor Nx default: `start` target caching disabled.
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
1139fd85a9. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
2025-11-16 18:20:06 +01:00
Félix MalfaitandGitHub 4f1623ff76 Seeding Attachments, Disable ORM Logs, Seeding Parallelization (#15174)
Improvements to database seeding performance and developer experience.

**Changes:**

1. **Attachment seeding**: Add sample files (PDF, XLSX, PPTX, PNG, ZIP)
to dev seeds with proper file storage
2. **Seeding parallelization**: Process entities within batches in
parallel while respecting dependencies
3. **ORM query logging**: Replace manual logger toggling with
`ORM_QUERY_LOGGING` env var
- Values: `disabled` (default), `server-only` (for local dev), `always`
- Configured once in `core.datasource.ts`, removed from all seeder
services

**For .env:**
```bash
ORM_QUERY_LOGGING=server-only
```

Net result: Faster seeding, cleaner code (-68 lines), better local dev
experience.
2025-10-17 16:50:19 +02:00
Félix MalfaitandGitHub 7b78b64bca Update clickhouse tables (#11905)
Following a discussion with @Bonapara - changing the base tables
2025-05-07 09:39:18 +02:00
Félix MalfaitandGitHub cec17a3ebe Fix clickhouse migration not in dist (#11709)
Clickhouse migrations were not correctly copied to the dist folder (for
command yarn clickhouse:migrate:prod)
2025-04-23 21:16:30 +02:00
martmullandGitHub 927b8c717e Poc lambda deployment duration (#10340)
closes https://github.com/twentyhq/core-team-issues/issues/436

## Acheivements
Improve aws lambda deployment time from ~10/15 secs to less that 1 sec

## Done
- migrate with the new code executor architecture for local and lambda
drivers
- support old and new executor architecture to avoid breaking changes
- first run is long, next runs are quick even if code step is updated

## Demo using `lambda` driver
### Before


https://github.com/user-attachments/assets/7f7664b4-658f-4689-8949-ea2c31131252


### After



https://github.com/user-attachments/assets/d486c8e2-f8f8-4dbd-a801-c9901e440b29
2025-02-20 10:49:57 +01:00
85c04c8931 Performance improvement to dev xp (#9294)
The DX is not great when you need to do a lot of database
resets/command.

Should we disable Typescript validation to speed things up? With this
and caching database:reset takes 1min instead of 2 on my machine.


See also: https://github.com/typeorm/typeorm/issues/4136

And #9291 / #9293

---------

Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
2025-01-01 17:28:45 +01:00
martmullandGitHub 62fe1d0e88 6653 serverless functions store and use environment variables in serverless function scripts (#7390)
![image](https://github.com/user-attachments/assets/a15bd4c1-3db4-4466-b748-06bdf3874354)

![image](https://github.com/user-attachments/assets/71242dfb-956b-43ed-9704-87cb0dfbc98d)
2024-10-03 13:56:17 +02:00
martmullandGitHub 7e03419c16 Serverless function improvements (#6769)
- add layer for lambda execution
- add layer for local execution
- add package resolve for the monaco editor
- add route to get installed package for serverless functions
- add layer versioning
2024-09-02 15:25:20 +02:00
ThaïsandGitHub c193663a71 chore: use Nx affected tasks in CI (#5110)
Closes #5097

- Uses "nx affected" to detect what projects need to be checked in the
current PR (for now, `ci-front` and `ci-server` workflows only).
- Caches results of certain tasks (`lint`, `typecheck`, `test`,
`storybook:build`) when a PR pipeline runs. The next runs of the same
PR's pipeline will then be able to reuse the PR's task cache to execute
tasks faster.
- Caches Yarn's cache folder to install dependencies faster in CI jobs.
- Rewrites the node modules cache/install steps as a custom, reusable
Github action.
- Distributes `ci-front` jobs with a "matrix" strategy.
- Sets common tasks config at the root `nx.json`. For instance, to
activate the `typecheck` task in a project, add `typecheck: {}` to its
`project.json` and it'll use the default config set in `nx.json` for the
`typecheck` task. Options can be overridden in each individual
`project.json` if needed.
- Adds "scope" tags to some projects: `scope:frontend`, `scope:backend`,
`scope:shared`. An eslint rule ensures that `scope:frontend` only
depends on `scope:frontent` or `scope:shared` projects, same for
`scope:backend`. These tags are used by `nx affected` to filter projects
by scope and generates different task cache keys according to the
requested scope.
- Enables checks for twenty-emails in the `ci-server` workflow.
2024-04-30 16:28:25 +02:00
Félix MalfaitandGitHub 9aa24ed803 Compile with swc on twenty-server (#4863)
Experiment using swc instead of tsc (as we did the switch on
twenty-front)

It's **much** faster (at least 5x) but has stricter requirements.
I fixed the build but there's still an error while starting the server,
opening this PR for discussion.

Checkout the branch and try `nx build:swc twenty-server`

Read: https://docs.nestjs.com/recipes/swc#common-pitfalls
2024-04-14 09:09:51 +02:00
Charles BochetandGitHub 5bdca9de6c Migrate to a monorepo structure (#2909) 2023-12-10 18:10:54 +01:00