This PR does not produce any functional changes for our users. It
prepares the branches for workflows by:
- decommissioning `output` and `context` fields or `workflowRun` records
and use newly created `state` field from front-end and back-end
- use `stepStatus` computed by `back-end` in `front-end`
- add utils and types in `twenty-shared/workflow` (not completed, a
follow-up is scheduled
https://github.com/twentyhq/core-team-issues/issues/1211)
- add concurrency to `workflowQueue` message queue to avoid weird branch
execution when using forms in workflow branches
- add a WithLock decorator for better dev experience of
`CacheLockService.withLock` usage
Here is an example of such a workflow running (front branch display is
not yet done that's why it looks ugly) ->
https://discord.com/channels/1130383047699738754/1258024460238192691/1392897615171158098
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
Follow-up on https://github.com/twentyhq/twenty/pull/12007
In this PR
- adding a filter on HttpExceptionHandlerService to filter out 4xx
errors from driver handling (as we do for graphQL errors: see
useGraphQLErrorHandler hook - only filteredIssues are sent to`
exceptionHandlerService.captureExceptions()`.)
- grouping together more missing metadata issues
- attempting to use error codes as issues names in sentry to improve UI;
for now it says "Error" all the time
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
No need to audit log workflow runs as it's already a form of audit log.
Add more audit log for other objects
Rename MessagingTelemetry to MessagingMonitoring
Merge Analytics and Audit in one (Audit)
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
related to https://github.com/twentyhq/core-team-issues/issues/601
## Done
- add a `onDbEvent` `Subscription` graphql endpoint to listen to
database_event using what we have done with webhooks:
- you can subscribe to any `action` (created, updated, ...) for any
`objectNameSingular` or a specific `recordId`. Parameters are nullable
and treated as wildcards when null.
- returns events with following shape
```typescript
@Field(() => String)
eventId: string;
@Field()
emittedAt: string;
@Field(() => DatabaseEventAction)
action: DatabaseEventAction;
@Field(() => String)
objectNameSingular: string;
@Field(() => GraphQLJSON)
record: ObjectRecord;
@Field(() => [String], { nullable: true })
updatedFields?: string[];
```
- front provide a componentEffect `<ListenRecordUpdatesEffect />` that
listen for an `objectNameSingular`, a `recordId` and a list of
`listenedFields`. It subscribes to record updates and updates its apollo
cached value for specified `listenedFields`
- subscription is protected with credentials
## Result
Here is an application with `workflowRun`
https://github.com/user-attachments/assets/c964d857-3b54-495f-bf14-587ba26c5a8c
---------
Co-authored-by: prastoin <paul@twenty.com>
## Context
When a trial ends, we receive a webhook from stripe to switch a
workspace from its previous status to SUSPENDED.
We also have some workspaces in ONGOING_CREATION that started the flow,
started the trial but did not finish completely the flow which means the
workspace has no data/metadata/schema.
Because of this, we can have workspaces that switch from
ONGOING_CREATION to SUSPENDED directly and ONGOING_CREATION workspaces
that didn't start the trial can stay in this state forever since they
are not cleaned up by the current cleaner.
To solve those 2 issues, I'm adding a new cron that will automatically
clean ONGOING_CREATION workspaces that are older than 7 days and I'm
updating the stripe webhook to only update the activationStatus to
SUSPENDED if it's already ACTIVE (then it will be deleted by the other
cron in the other case)
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
Simplifying a lot the upgrade system.
New way to upgrade:
`yarn command:prod upgrade`
New way to write upgrade commands (all wrapping is done for you)
```
override async runOnWorkspace({
index,
total,
workspaceId,
options,
}: RunOnWorkspaceArgs): Promise<void> {}
```
Also cleaning CommandModule imports to make it lighter
## Context
This config tells bullmq what to do with jobs once they have been
processed.
We used to retain 100 completed jobs and 500 failed jobs. We never
really needed that much retention and keeping succeeded job is not
useful either in our case. I'm updating the strategy so all completed
jobs are removed from the queue and we now keep 100 failed jobs in case
we need to debug. (Although we should have good monitoring with
ExceptionHandler driver)
Fixes https://github.com/twentyhq/twenty/issues/8810
Fixes https://github.com/twentyhq/twenty/issues/5268
Fixes https://github.com/twentyhq/twenty/issues/8971
- Fixing Task/Note creation not sending position during creation
- Adding a command to backfill position being null, using existing
backfill command.
- Removed unused backfill job.
- Updated workspace entities to set position non-nullable and set a
default value to make it non-required on the API
- Updated position factory to set a default position for all objects
having a POSITION field instead of only company/people
- Moved the try/catch in each resolver factory calling
GraphqlQueryRunnerException handler, makes more sense to call it in the
actual graphql-query-runner and removing some duplicate codes
- Adding validations for input in QueryRunnerArgs factories
- Allow sync-metadata to override and sync defaultValues for certain
field types (that can't be updated by users)
- Removing health-check from sync-metadata command during force mode to
improve performances
Solves (https://github.com/twentyhq/private-issues/issues/194)
**TLDR**
Updates the billingCustomer table data using stripe webhooks event, also
updates the customer's metadata in stripe, in order to contain the
workspaceId associated to this customer.
**In order to test**
Billing:
- Set IS_BILLING_ENABLED to true
- Add your BILLING_STRIPE_SECRET and BILLING_STRIPE_API_KEY
- Add your BILLING_STRIPE_BASE_PLAN_PRODUCT_ID (use the one in testMode
> Base Plan)
-
Authenticate with your account in the stripe CLI
Run the command: stripe listen --forward-to
http://localhost:3000/billing/webhooks
Run the twenty workker
Authenticate yourself on the app choose a plan and run the app normally.
In stripe and in posgress the customer table data should be added.
**Next steps**
Learn more about integrations tests and implement some for this PR.
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
- create a serverless function when creating a new workflow code step
- add code editor in workflow code step
- move workflowVersion steps management from frontend to backend
- add a custom resolver for workflow-version management
- fix optimistic rendering on frontend
- fix css
- delete serverless function when deleting workflow code step
TODO
- Don't update serverlessFunction if no code change
- Factorize what can be between crud trigger and crud step
- Publish serverless version when activating workflow
- delete serverless functions when deleting workflow or workflowVersion
- fix optimistic rendering for code updates
- Unify CRUD types
<img width="1279" alt="image"
src="https://github.com/user-attachments/assets/3d97ee9f-4b96-4abc-9d36-5c0280058be4">
## Context
bull-mq connection was not working as intended, the connection parameter
was ignored and was falling back to localhost.
This PR should fix the issue by instantiating a IORedis client following
bullmq documentation https://docs.bullmq.io/guide/connections
I also changed cache-storage module to use IORedis client as well to be
more consistent even though it was not necessary there. We could move
that instantiation to a factory class in the future.
## Test
start server + worker with correct port and wrong port with
cache-storage-type memory/redis and queue-type sync/bull-mq
Closes#7421
This pull request consolidates Redis connection parameters into a single
`REDIS_URL` environment variable across various configuration files and
code modules. The most important changes include updates to environment
variable files, Docker and Kubernetes configurations, and code modules
to utilize the new `REDIS_URL` format.
### Environment Variable Updates:
*
[`packages/twenty-docker/.env.example`](diffhunk://#diff-f4b5e7acc0dde630eafb2228390ca56bd56db0d183582be6433a9ee808088a4cL8-R8):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/.env.example`](diffhunk://#diff-c06e244338b33286ea90221635809785352a971de53f647ea65650124ae74ad2L53-R53):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
*
[`packages/twenty-server/.env.test`](diffhunk://#diff-def44a08e699c9deca2f72c9f87951de5d33d9ccf5621eab2f34978df8ad0954L16-R16):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
### Docker and Kubernetes Configuration Updates:
*
[`packages/twenty-docker/docker-compose.yml`](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL` in multiple
service definitions.
[[1]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL28-R28)
[[2]](diffhunk://#diff-545fb51ad66f93f727bb8f8b375dc6858b33348d91d5e51ad931fb0bbc1affeeL62-R61)
*
[`packages/twenty-docker/k8s/manifests/deployment-server.yaml`](diffhunk://#diff-91623ed4e8b2088947cfa9a5dad76b6013e8db0c150d84347a215fa9ad78bf04L44-R45):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/manifests/deployment-worker.yaml`](diffhunk://#diff-8532debd131ce168a2527fa6a9be6405792178576ee47e2eef0cd9f3ff4a8f8cL43-R44):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-server.tf`](diffhunk://#diff-b4c468660ab00bd03589d0d47502c477ca83f7d876ff196534880b02ba46dce0L64-R65):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-docker/k8s/terraform/deployment-worker.tf`](diffhunk://#diff-e5d6a1f68d5391e2120aef6261f22f905aa1ce0a567e31b21a27d44f313eaecaL62-R63):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
### Code Module Updates:
*
[`packages/twenty-server/src/engine/core-modules/cache-storage/cache-storage.module-factory.ts`](diffhunk://#diff-06e787a7c8a48022d5909b5df9b8c6ca192521cf32f51d7f561cee937bed6678L23-R35):
Replaced `REDIS_HOST` and `REDIS_PORT` with `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/environment/environment-variables.ts`](diffhunk://#diff-26ce615693b053eda02aa48aa2e30400381a2588dcb08d4a9dc3b0bf5bdd6fe7L378-R384):
Added validation for `REDIS_URL`.
*
[`packages/twenty-server/src/engine/core-modules/message-queue/message-queue.module-factory.ts`](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6):
Replaced `REDIS_HOST`, `REDIS_PORT`, `REDIS_USERNAME`, and
`REDIS_PASSWORD` with `REDIS_URL`.
[[1]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bR6)
[[2]](diffhunk://#diff-7bd644d28bdd86c159c1d71242753df5d5acd91c73c6e60b2e35caa53ed3836bL35-R41)
### Documentation Updates:
*
[`packages/twenty-website/src/content/developers/self-hosting/self-hosting-var.mdx`](diffhunk://#diff-c4cc78a3ce18b6edb10f1aee8990271e1d2796a8c06c1c6ae3b68db8d52278a3L37-R37):
Updated documentation to reflect the change to `REDIS_URL`.
*
[`packages/twenty-website/src/content/developers/self-hosting/upgrade-guide.mdx`](diffhunk://#diff-c7d757829f6128e1f47aa1955bde561292dce558280511fab66281afa99042a5R102-R112):
Added upgrade instructions for the new `REDIS_URL` variable.
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Charles Bochet <charles@twenty.com>
From PR: #6626Resolves#6763Resolves#6055Resolves#6782
## GTK
I retain the 'Invite by link' feature to prevent any breaking changes.
We could make the invitation by link optional through an admin setting,
allowing users to rely solely on personal invitations.
## Todo
- [x] Add an expiration date to an invitation
- [x] Allow to renew an invitation to postpone the expiration date
- [x] Refresh the UI
- [x] Add the new personal token in the link sent to new user
- [x] Display an error if a user tries to use an expired invitation
- [x] Display an error if a user uses another mail than the one in the
invitation
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
