- replace save by insert
- if the insert output is needed, cast the generatedMaps
- remove transactions from trigger services. Doing it manually would be
complex and not reliable
- move cron commands logic to a workspace service
- add commands to enqueue not started and dequeue staled workflows
- stop dispatching one job per workspace. Handle all workspaces in one
job
- run enqueue cron every 5 minutes instead of every minutes
Follow up:
- add a separated redis key to control if there are not started
workflows. It will avoid fetching workflow runs for each workspace
This Pr continues the extensibility journey
- adds a `core.databaseEventTrigger` table
- add a oneToMany relation between `core.serverlessFunction` and
`core.databaseEventTrigger`
- add a job `CallDatabaseEventTriggerJobsJob` triggered by
`EntityEventsToDbListener` triggering `serverlessFunction` based on the
`core.databaseEventTrigger.settings.eventName`
- add a new `trigger-queue` to carry this job
- renamed `DatabaseEventTriggerListener` into
`WorkflowDatabaseEventTriggerListener`
This Pr begins the extensibility journey
- adds a `core.cronTrigger` table
- add a oneToMany relation between core.serverlessFunction and
`core.cronTrigger` (one serverlessFunction can be triggered by multiple
cronTriggers)
- add a job to trigger a serverless function
- adds a cron to trigger serverlessFunction (via the trigger job) based
on the core.cronTrigger.setting.pattern
- adds a command to register the cron
- add the command in `cron-register-all.command.ts`
## Context
To simplify the way we inject our default datasource, I've recently
removed the token injection that was confusion since we only had once
configured on the module level. Now I'm removing TypeORM service which
allows us to instantiate a new Datasource with the same parameters as
the default one, it was redundant and confusing.
Today WorkflowVariablesDropdownAllItems is used for variable picker in
all workflow steps, and for step field picker in filters.
Using that component in filters prevent us from knowing if a full record
or only the id has been selected.
This PR:
- creates a new component WorkflowDropdownStepOutputItems, mostly
copying the logic of WorkflowVariablesDropdownAllItems
- call it WorkflowStepFilterFieldSelect, removing the display logic from
that parent component
- store isFullRecord in filter. So we now know if we should display a
record picker or a uuid picker
Before - selecting id makes picker behaves like when we select an object
https://github.com/user-attachments/assets/bde34dc5-8011-4983-8d0f-d8cb0cb3c045
After - selecting object and id are two different things
https://github.com/user-attachments/assets/49289990-3e6d-4ad7-abc1-e3ade2a821bb
On field selection, we need to retrieve the fieldMetadata to set the
filter type. Current code was using the current step filter
fieldMetdataId instead of the new one.
Also adding a special behaviour for Select filter: it behaves as
arrays/multiselect but the opearands are not the same.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Improve workflow enqueue cron : instead of relying on the cache to know
how many workflows we can enqueue, query the DB. Then set the cache and
process the not started workflows.
Also adding a second cron that will look for workflows enqueued one hour
ago or more and put these back in the not started status. This will
allow the first cron to start these again.
When the target is not in the direct next step ids of the source (step
or trigger), check if there is a filter between both. If yes, delete the
filter and remove it from source next step ids.
Also refactored the existing.
Add `workflow:delete-workflow-runs` command to delete workflow runs
Options:
- created-before YYYY-MM-DD default to now
- standard options from
`ActiveOrSuspendedWorkspacesMigrationCommandRunner` commands
In this PR
1. Fix delete and soft-delete repository methods for repositories where
permission checks are NOT bypassed: they need to have a selection of
columns to return by default. To match what we did for insert I set it
to `'*'` by default. But I feel this may be bug prone as developers will
not necessarily think to fill the right values in. Maybe we should
change the api to use selectable fields by default. @charlesBochet
2. Add field permission seeds and enable field permission feature flag
in dev
- handle missing composite fields
- refacto filter action to be more reliable
Remaining improvements:
- filter out some field types
- improve relations
This PR aims to improve performances as we are making A LOT of queries
after having added events emission to the ORM layer. While investigating
issues, I've come across multiple problems I will describe below
## Add index on workspace.activationStatus as we are querying it a lot
As per title
## Add logs on core datasource destroy
It seems that we have postgres connection pool destruction in
production. I cannot reproduce locally but I suspect the Query Timeouts
to be the root cause. I'm fixing most of the Query Timeouts cause in
this PR but I'm adding the logs so we have more information in case in
keeps happening in production.
## GraphQL query runner createMany
It was using a for loop on each record. This is as issue has we will
emit an event separately for each record => we should always try to
batch events.
Replacing by a save on all records. Note that this is not perfect as we
should avoid using save (bad performances), and use insert + updateMany
instead. As I have follow up discussions regarding permissions, I
haven't replaced it by insert and updateMany yet. Using save instead of
a for loop makes the problem less worrying.
## Introduce updateMany in ORM @Weiko @ijreilly FYI
.save(manyRecords) is bad as it's querying the data for no good reason
(and doing a select for each record...).
We already have .insert(), i'm introducing .updateMany()
I think our ORM layer should be simplified a lot but I'm not starting
the refacto yet
## Fixing ORM @Weiko @ijreilly FYI
- Fixing events emission in delete function
- Fixing events emission in insert function
- make sure everything is batched
## Events performance @Weiko @ijreilly FYI
Do not emit timelineActivity db events as this does not seem useful and
is quite heavy
## Messaging and Calendar performance @bosiraphael FYI
Rework many functions to make sure they are batched. This does not touch
the driver layer and I have heavily tested it. (multiple account, with
multiple channels, common thread, common messages, etc...)
## Workflow Trigger relation Fetch performance @martmull FYI
Improved performance by batching
Fixes https://github.com/twentyhq/twenty/issues/13428
We perform the call enrichment even with id undefined. It returns the
first item in DB.
Adding two more fixes related to filters:
- Conditions title
- Fixing placeholder color for select control (only other impacted field
is in Advanced filters)
- add to step output schema the information that field is a composite
sub field
- from output schema, when selecting the variable, copy all info
required to stepFilter
- from stepFilter, when selecting a value, display a special component
for composites
We will have floating steps in our workflow with the branch design.
Currently, a step without parent is considered linked to the trigger. We
need to distinguish the 2 cases. Thus this PR:
- add `nextStepIds` to workflowVersion.trigger
- create a command to migrate existing triggers
- Renamed `WorkflowActionAdapter` to `ToolExecutorWorkflowAction`
- Renamed `settingPermission` table to `permissionFlag` and `setting`
column to `flag`
- Decoupled the send email logic from workflows to tools
- Add new `Tools Permission` section in FE
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
- add fieldMetadataId to step output schema
- use it to display FormFieldInput in Filter input
- few fixes for a few fields
Next step:
- Handle composite fields
- Design review
