## Context
Add an eventEmitter instance to twenty datasources so we can emit DB
events.
Add input and output formatting to twenty orm (formatData, formatResult)
Those 2 elements simplified existing logic when we interact with the
ORM, input will be formatted by the ORM so we can directly use
field-like structure instead of column-like. The output will be
formatted, for builder queries it will be in `result.generatedMaps`
where `result.raw` preserves the previous column-like structure.
Important change: We now have an authContext that we can pass when we
get a repository, this will be used for the different events emitted in
the ORM. We also removed the caching for repositories as it was not
scaling well and not necessary imho
Note: An upcoming PR should handle the onDelete: cascade behavior where
we send DESTROY events in cascade when there is an onDelete: CASCADE on
the FK.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
This PR does not produce any functional changes for our users. It
prepares the branches for workflows by:
- decommissioning `output` and `context` fields or `workflowRun` records
and use newly created `state` field from front-end and back-end
- use `stepStatus` computed by `back-end` in `front-end`
- add utils and types in `twenty-shared/workflow` (not completed, a
follow-up is scheduled
https://github.com/twentyhq/core-team-issues/issues/1211)
- add concurrency to `workflowQueue` message queue to avoid weird branch
execution when using forms in workflow branches
- add a WithLock decorator for better dev experience of
`CacheLockService.withLock` usage
Here is an example of such a workflow running (front branch display is
not yet done that's why it looks ugly) ->
https://discord.com/channels/1130383047699738754/1258024460238192691/1392897615171158098
This PR is purely technical, it does produces any functional change to
the user
- add Lock mecanism to run steps concurrently
- update `workflow-executor.workspace-service.ts` to handle multi branch
workflow execution
- stop passing `context` through steps, it causes race condition issue
- refactor a little bit
- simplify `workflow-run.workspace-service.ts` to prepare `output` and
`context` removal
- move workflowRun status computing from `run-workflow.job.ts` to
`workflow-executor.workspace-service.ts`
## NOTA BENE
When a code step depends of 2 parents like in this config (see image
below)
If the form is submitted before the "Code - 2s" step succeed, the branch
merge "Form" step is launched twice.
- once because form is submission Succeed resumes the workflow in an
asynchronous job
- the second time is when the asynchronous job is launched when "Code -
2s" is succeeded
- the merge "Form" step makes the workflow waiting for response to
trigger the resume in another job
- during that time, the first resume job is launched, running the merge
"Form" step again
This issue only occurs with branch workflows. It will be solved by
checking if the currentStepToExecute is already in a SUCCESS state or
not
<img width="505" alt="image"
src="https://github.com/user-attachments/assets/b73839a1-16fe-45e1-a0d9-3efa26ab4f8b"
/>
This PR does not produce any functional change
First step of the workflow branch feature
- add gather `workflowRun.output` and `workflowRun.context` into one
column `workflowRun.runContext`
- add a command to fill `runContext` from `output` and `context` in
existing records
- maintain `runContext` up to date during workflow runs
- new status `ENQUEUED` added. With a command to backfill
- counter in cache per workspace, managed by a new service
[workflow-run-queue.workspace-service.ts](https://github.com/twentyhq/twenty/compare/tt-improve-workflow-run-queueing?expand=1#diff-1e2de2a48cd482a3bd7e8dedf1150a19d0b200afbd9282181a24ecddddb56927)
- cron added that will run every minute to look for not started
workflows
Here is the new flow:
- When executing a workflow, we check if the queue is not full. If not,
run is created as `ENQUEUED` and the run workflow job is triggered as
usual. If full, create the run as NOT_STARTED and do not trigger the job
- Cron will look for NOT_STARTED workflows and queue some if there is
some place again
- Only MANUAL and Form submit skip the queue limit
In this PR
- Determine object record permissions on workflows objects (workflow,
workflowVersion, workflowRun) base on settings permissions @Weiko
- Add Workflow permission guards on workflow resolvers @thomtrp . **Any
method within a resolver that has the SettingsPermission Guard is only
callable by a apiKey or a user that has the permission** (so not by
external parties).
- Add checks bypass in workflow services since 1) for actions gated by
settings permissions, the gate should be done at resolver level, so it
will have been done before the call to the service 2) some service
methods may be called by workflowTriggerController which is callable by
external parties without permissions (ex:
workflowCommonWorkspaceService.getWorkflowVersionOrFail). This is
something we may want to change in the future (still to discuss), by
removing the guard at resolver-level and relying on
shouldBypassPermissionChecks at getRepository and made in a way that we
only bypass for external parties.
- Add checks bypass for actions performed by workflows since they should
not be restricted in our current vision
- Add tests
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
related to https://github.com/twentyhq/core-team-issues/issues/601
## Done
- add a `onDbEvent` `Subscription` graphql endpoint to listen to
database_event using what we have done with webhooks:
- you can subscribe to any `action` (created, updated, ...) for any
`objectNameSingular` or a specific `recordId`. Parameters are nullable
and treated as wildcards when null.
- returns events with following shape
```typescript
@Field(() => String)
eventId: string;
@Field()
emittedAt: string;
@Field(() => DatabaseEventAction)
action: DatabaseEventAction;
@Field(() => String)
objectNameSingular: string;
@Field(() => GraphQLJSON)
record: ObjectRecord;
@Field(() => [String], { nullable: true })
updatedFields?: string[];
```
- front provide a componentEffect `<ListenRecordUpdatesEffect />` that
listen for an `objectNameSingular`, a `recordId` and a list of
`listenedFields`. It subscribes to record updates and updates its apollo
cached value for specified `listenedFields`
- subscription is protected with credentials
## Result
Here is an application with `workflowRun`
https://github.com/user-attachments/assets/c964d857-3b54-495f-bf14-587ba26c5a8c
---------
Co-authored-by: prastoin <paul@twenty.com>
- Add position during workflow version / creation. It will allow to have
the versions and runs ordered
- Block the creation from generated api for versions. We use workflow
post hooks or create from draft
Currently, when filling the form, values are not saved in the action
settings. This is an issue because we do not see the response in the
node settings, only in the output of the step.
This PR:
- adds a new endpoint to update a step in the run flow output
- updates this flow when a step is updated
https://github.com/user-attachments/assets/2e74a010-a0d2-4b87-bd1f-1c91f7ca6b60
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
- create form action
- add `pendingEvent` to executor output
- when receiving pendingEvent, exit workflow execution
- let the workflow in running status for now before taking a decision
Actions will now:
- receive the complete input
- get the step they want to execute by themself
- check that the type is the right one
- resolve variables
These all share a common executor interface.
It will allow for actions with a special execution process (forms, loop,
router) to have all required informations.
Main workflow executor should:
- find the right executor to call for current step
- store the output and context from step execution
- call next step index
Should be possible to end workflow, not matter what the current status
is.
On failure before the workflow was started, this error prevent the
workflow to be marked as failed with the right error message.
We need the version trigger and steps to be stored in the output. We
should not rely on the version itself because some run are made on draft
versions. Which means versions could be edited afterwards.
## Context
In some CustomException exceptions, we were instantiating a code without
initializing it which was overriding the parent code and it was then
lost when retrieving it in filters.
Removing them to make sure we don't reproduce this pattern
Solves :
https://github.com/twentyhq/private-issues/issues/251
**TLDR:**
Clean Billing Code using feedback of the previous PR (#9865). Normalized
the metadata and names of the products, prices, and meters in Stripe so
that they can be accessed in stripe's test mode and live mode.
**In order to test:**
1. Have the environment variable IS_BILLING_ENABLED set to true and add
the other required environment variables for Billing to work
2. Do a database reset (to ensure that the new feature flag is properly
added and that the billing tables are created)
3. Run the command: npx nx run twenty-server:command
billing:sync-plans-data (if you don't do that the products and prices
will not be present in the database)
4. Run the server , the frontend, the worker, and the stripe listen
command (stripe listen --forward-to
http://localhost:3000/billing/webhooks)
5. Buy a subscription for the Acme workspace
6. Create a workflow and run it
7. After the run has been finished check in sprite the quantity of
events in the CreditMeter, you should see that there is a new occurence
with value one.
Solves :
https://github.com/twentyhq/private-issues/issues/241https://github.com/twentyhq/private-issues/issues/254
**TLDR:**
- Add BillingMeterEventService and StripeBillingMeterEventService in
order to send billing meter events to stripe.
- Plugged the service into workflow node execution for testing purposes
(more improvements on this area will be done in the next PR's)
**In order to test:**
- Have the environment variable IS_BILLING_ENABLED set to true and add
the other required environment variables for Billing to work
- Do a database reset (to ensure that the new feature flag is properly
added and that the billing tables are created)
- Run the command: npx nx run twenty-server:command
billing:sync-plans-data (if you don't do that the products and prices
will not be present in the database)
- Run the server , the frontend, the worker, and the stripe listen
command (stripe listen --forward-to
http://localhost:3000/billing/webhooks)
- Buy a subscription for the Acme workspace
- Create a workflow and run it
- After the run has been finished check in sprite the quantity of events
in the CreditMeter, you should see that there is a new occurence with
value one.
**Take into consideration:**
- I used an eventName that I have made a long time ago, so it hasn't a
significant naming. I'm updating the meters and associated prices in
stripe to use the correct meter with a more clearer eventName.
- I put some error handling in the execution of the workflow nodes, this
is still incomplete and needs some refinement, I would like the feedback
of the workflows track for a more cleaner approach
- catch error on action execution. We will log the error and return it
in the step
- catch error on workflow run
- remove the catch in the action. All actions should simply throw and
let the executor do the job
<img width="1512" alt="Capture d’écran 2025-01-14 à 17 35 53"
src="https://github.com/user-attachments/assets/dcf79567-a309-45f1-a640-c50b7ac4769b"
/>
We want to avoid infinite loops using workflows. Adding a throttler with
a limit of 10 executions / sec by default for each workflow.
We were not emitting events on workflow actions so loops could not
happen. Since throttler is there we can now and these.
Adding an error message so the user knows when it happens.
<img width="1284" alt="Capture d’écran 2024-12-27 à 17 05 20"
src="https://github.com/user-attachments/assets/dafa837b-5b4c-48be-8207-c90f5c71a236"
/>
- add `inputSchema` column in serverless function. This is an array of
parameters, with their name and type
- on serverless function id update, get the `inputSchema` + store empty
settings in step
- from step settings, build the form
TODO in next PR:
- use field type to decide what kind of form should be printed
- have a strategy to handle object as input
https://github.com/user-attachments/assets/ed96f919-24b5-4baf-a051-31f76f45e575
Workflow version :
- prevent status to be update manually
- prevent creation / deletion in another status than draft
- prevent creation if a draft already exists
Workflow:
- prevent statuses to set manually
WorkflowRun:
- prevent all manual operations
Following figma updates
https://www.figma.com/design/PNBfTgOVraw557OXChYagk/Explo?node-id=21872-7929&t=DOUzd6rzwr6lprcs-0
- No activity targets for workflow entities for now
- Adding a direct relation between workflow run et workflow
- Adding a status on the version (draft, active, deactivated)
- Adding a list of statuses on workflow
- publishedVersionId => lastPublishedVersionId
Also adding:
- the endpoint to deactivate a version
We want to avoid the nested structure of active pieces. Steps to execute
will now be separated from the trigger. It will be an array executed
sequentially.
For now a step can only be an action. But at some point it will also be
a branch or a loop
Fix https://github.com/twentyhq/twenty/issues/6669
- create a commun function `startWorkflowRun` that both create the run
object and the job for executing the workflow
- use it in both the `workflowEventJob` and the `runWorkflowVersion`
endpoint
Bonus:
- use filtering for exceptions instead of a util. It avoids doing a try
catch in all endpoint
- create a workflow run every time a workflow is triggered in
not_started status. This status will be helpful later for once workflows
will be scheduled
- update run status once workflow starts running
- complete status once the workflow finished running
- add a failed status if an error occurs
- Add global listener on database event
- Fetch event listener associated
- Trigger associated workflow
Also updated the runner so it expects the input to be in the payload
rather than the trigger
