To avoid huge workflows to block the worker, we will enqueue a new job
every 20 steps.
This could be more than 20 if there are branches but I think this is
fine, the goal is only to have a limit set.
Also cleaning a bit the code to mark running steps as failed when
workflow fails.
I tested it on a huge workflow:
https://github.com/user-attachments/assets/d7b8e345-d1a1-4467-96fd-92117b500120
Filters should not cut the whole workflow. These should only stop the
branch. This PR:
- adds a new skipped status
- when a filter stops, it still goes to the next step
- the next step will execute if there is at least a successful step
- if only skipped step, it will be skipped as well
It allows to use filters in iterators.
https://github.com/user-attachments/assets/1cfca052-55c0-4ce5-9eb8-63736618d082
In this PR https://github.com/twentyhq/twenty/pull/14785 we deprecated
viewFilterOperand (from camelCase to capital snakeCase values), but we
had not migrated the existing workflow steps values. Hence they cannot
be executed!
Let's fix that by dealing with both operands, new and deprecated, to
mitigate the issue. Then we will add a command to migrate the values.
In the BE we have stored filter operand as IS_EMPTY, IS_NOT_EMPTY, etc.
For some reason in the FE we were manipulating IsEmpty, IsNotEmpty, etc.
(maybe because they were used before in Views before they were moved to
core)
So we were converting the operands in the FE from IS to Is
(convertViewFilterOperandFromCore) to read and manipulate viewFilters,
and then back to BE version to send mutations etc., from Is to IS
(convertViewFilterOperandToCore).
The migration is now over, so we can remove and simplify that code.
(In the 1-5:migrate-views-to-core command we still do the migration from
Is format to IS format.)
When all items have been processed, iterator should:
- store last iteration in history
- return the final result saying iteration have been successful. So it
gets stored by the executor
It was not doing the first step, so we were losing the last iteration.
Splitting into separated function + adding tests
- before reseting the step, enrich info with previous result
- reset only when a step is executed more than once. This will avoid to
store the initial `NOT_STARTED` status of the step
- replace save by insert
- if the insert output is needed, cast the generatedMaps
- remove transactions from trigger services. Doing it manually would be
complex and not reliable
Today WorkflowVariablesDropdownAllItems is used for variable picker in
all workflow steps, and for step field picker in filters.
Using that component in filters prevent us from knowing if a full record
or only the id has been selected.
This PR:
- creates a new component WorkflowDropdownStepOutputItems, mostly
copying the logic of WorkflowVariablesDropdownAllItems
- call it WorkflowStepFilterFieldSelect, removing the display logic from
that parent component
- store isFullRecord in filter. So we now know if we should display a
record picker or a uuid picker
Before - selecting id makes picker behaves like when we select an object
https://github.com/user-attachments/assets/bde34dc5-8011-4983-8d0f-d8cb0cb3c045
After - selecting object and id are two different things
https://github.com/user-attachments/assets/49289990-3e6d-4ad7-abc1-e3ade2a821bb
On field selection, we need to retrieve the fieldMetadata to set the
filter type. Current code was using the current step filter
fieldMetdataId instead of the new one.
Also adding a special behaviour for Select filter: it behaves as
arrays/multiselect but the opearands are not the same.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
In this PR
1. Fix delete and soft-delete repository methods for repositories where
permission checks are NOT bypassed: they need to have a selection of
columns to return by default. To match what we did for insert I set it
to `'*'` by default. But I feel this may be bug prone as developers will
not necessarily think to fill the right values in. Maybe we should
change the api to use selectable fields by default. @charlesBochet
2. Add field permission seeds and enable field permission feature flag
in dev
- handle missing composite fields
- refacto filter action to be more reliable
Remaining improvements:
- filter out some field types
- improve relations
- Renamed `WorkflowActionAdapter` to `ToolExecutorWorkflowAction`
- Renamed `settingPermission` table to `permissionFlag` and `setting`
column to `flag`
- Decoupled the send email logic from workflows to tools
- Add new `Tools Permission` section in FE
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
- add fieldMetadataId to step output schema
- use it to display FormFieldInput in Filter input
- few fixes for a few fields
Next step:
- Handle composite fields
- Design review
- Added a generic HTTP request tool, allowing agents and workflows to
make HTTP requests to external APIs with configurable method, headers,
and body.
- Decoupled HTTP request workflow nodes from workflow-specific types and
factories, introducing a generic tool interface.
- Updated agent system prompts to include explicit guidance for the HTTP
request tool, including when and how to use it, and how to communicate
limitations.
### Demo
https://github.com/user-attachments/assets/129bc445-a277-4a19-95ab-09f890f8f051
## Context
Add an eventEmitter instance to twenty datasources so we can emit DB
events.
Add input and output formatting to twenty orm (formatData, formatResult)
Those 2 elements simplified existing logic when we interact with the
ORM, input will be formatted by the ORM so we can directly use
field-like structure instead of column-like. The output will be
formatted, for builder queries it will be in `result.generatedMaps`
where `result.raw` preserves the previous column-like structure.
Important change: We now have an authContext that we can pass when we
get a repository, this will be used for the different events emitted in
the ORM. We also removed the caching for repositories as it was not
scaling well and not necessary imho
Note: An upcoming PR should handle the onDelete: cascade behavior where
we send DESTROY events in cascade when there is an onDelete: CASCADE on
the FK.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
This PR does not produce any functional changes for our users. It
prepares the branches for workflows by:
- decommissioning `output` and `context` fields or `workflowRun` records
and use newly created `state` field from front-end and back-end
- use `stepStatus` computed by `back-end` in `front-end`
- add utils and types in `twenty-shared/workflow` (not completed, a
follow-up is scheduled
https://github.com/twentyhq/core-team-issues/issues/1211)
- add concurrency to `workflowQueue` message queue to avoid weird branch
execution when using forms in workflow branches
- add a WithLock decorator for better dev experience of
`CacheLockService.withLock` usage
Here is an example of such a workflow running (front branch display is
not yet done that's why it looks ugly) ->
https://discord.com/channels/1130383047699738754/1258024460238192691/1392897615171158098
Instead of initializing model at start time we do it at run time to be
able to swap model provider more easily.
Also introduce a third driver for openai-compatible providers, which
among other allows for local models with Ollama
Step operand will more or less be the same as view filter operand.
This PR:
- moves `ViewFilterOperand` to twenty-shared
- use it as step operand
- check what operand should be available based on the selected field
type in filter action
- rewrite the function that evaluates filters so it uses
ViewFilterOperand instead
ViewFilterOperand may be renamed in a future PR.
I rebuilt the advanced filters used in views and workflow search for a
specific filter step.
Components structure remains the same, using `stepFilterGroups` and
`stepFilters`. But those filters are directly sent to backend.
Also re-using the same kind of states we use for advanced filters to
share the current filters used. And a context to share what's coming
from workflow props (function to update step settings and readonly)
⚠️ this PR only focusses on the content of the step. There is still a
lot to do on the filter icon behavior in the workflow
https://github.com/user-attachments/assets/8a6a76f0-11fa-444a-82b9-71fc96b18af4
This PR is purely technical, it does produces any functional change to
the user
- add Lock mecanism to run steps concurrently
- update `workflow-executor.workspace-service.ts` to handle multi branch
workflow execution
- stop passing `context` through steps, it causes race condition issue
- refactor a little bit
- simplify `workflow-run.workspace-service.ts` to prepare `output` and
`context` removal
- move workflowRun status computing from `run-workflow.job.ts` to
`workflow-executor.workspace-service.ts`
## NOTA BENE
When a code step depends of 2 parents like in this config (see image
below)
If the form is submitted before the "Code - 2s" step succeed, the branch
merge "Form" step is launched twice.
- once because form is submission Succeed resumes the workflow in an
asynchronous job
- the second time is when the asynchronous job is launched when "Code -
2s" is succeeded
- the merge "Form" step makes the workflow waiting for response to
trigger the resume in another job
- during that time, the first resume job is launched, running the merge
"Form" step again
This issue only occurs with branch workflows. It will be solved by
checking if the currentStepToExecute is already in a SUCCESS state or
not
<img width="505" alt="image"
src="https://github.com/user-attachments/assets/b73839a1-16fe-45e1-a0d9-3efa26ab4f8b"
/>
Currently, when a server query or mutation from the front-end fails, the
error message defined server-side is displayed in a snackbar in the
front-end.
These error messages usually contain technical details that don't belong
to the user interface, such as "ObjectMetadataCollection not found" or
"invalid ENUM value for ...".
**BE**
In addition to the original error message that is still needed (for the
request response, debugging, sentry monitoring etc.), we add a
`displayedErrorMessage` that will be used in the snackbars. It's only
relevant to add it for the messages that will reach the FE (ie. not in
jobs or in rest api for instance) and if it can help the user sort out /
fix things (ie. we do add displayedErrorMessage for "Cannot create
multiple draft versions for the same workflow" or "Cannot delete
[field], please update the label identifier field first", but not
"Object metadata does not exist"), even if in practice in the FE users
should not be able to perform an action that will not work (ie should
not be able to save creation of multiple draft versions of the same
workflows).
**FE**
To ease the usage we replaced enqueueSnackBar with enqueueErrorSnackBar
and enqueueSuccessSnackBar with an api that only requires to pass on the
error.
If no displayedErrorMessage is specified then the default error message
is `An error occured.`
This PR does not produce any functional change
First step of the workflow branch feature
- add gather `workflowRun.output` and `workflowRun.context` into one
column `workflowRun.runContext`
- add a command to fill `runContext` from `output` and `context` in
existing records
- maintain `runContext` up to date during workflow runs
Previous logic was using the previous step output and filtering items
that were passing filters.
What we actually want is:
- send filters, right operand being always a step output key, left
operand being either a key, either a value
- resolve those filter variables
- apply the filters to decide whether the condition is passed or not
