Previous refacto was creating output schema V2 which has more specific
schemas based on the step type. Before we were using one common schema,
which was too complex when searching for variable informations.
This PR migrate the deprecated schemas and remove the old code:
- mark previous `BaseOutputSchema` as deprecated
- remove other previous schemas
- use V2 everywhere
- icon should not be stored in schema. Instead it should be generated
based on the fieldmetadata or the item type
- replace save by insert
- if the insert output is needed, cast the generatedMaps
- remove transactions from trigger services. Doing it manually would be
complex and not reliable
Today WorkflowVariablesDropdownAllItems is used for variable picker in
all workflow steps, and for step field picker in filters.
Using that component in filters prevent us from knowing if a full record
or only the id has been selected.
This PR:
- creates a new component WorkflowDropdownStepOutputItems, mostly
copying the logic of WorkflowVariablesDropdownAllItems
- call it WorkflowStepFilterFieldSelect, removing the display logic from
that parent component
- store isFullRecord in filter. So we now know if we should display a
record picker or a uuid picker
Before - selecting id makes picker behaves like when we select an object
https://github.com/user-attachments/assets/bde34dc5-8011-4983-8d0f-d8cb0cb3c045
After - selecting object and id are two different things
https://github.com/user-attachments/assets/49289990-3e6d-4ad7-abc1-e3ade2a821bb
When the target is not in the direct next step ids of the source (step
or trigger), check if there is a filter between both. If yes, delete the
filter and remove it from source next step ids.
Also refactored the existing.
- add to step output schema the information that field is a composite
sub field
- from output schema, when selecting the variable, copy all info
required to stepFilter
- from stepFilter, when selecting a value, display a special component
for composites
- add fieldMetadataId to step output schema
- use it to display FormFieldInput in Filter input
- few fixes for a few fields
Next step:
- Handle composite fields
- Design review
# Introduction
Following `FieldMetadataInterface` deprecation in
https://github.com/twentyhq/twenty/pull/13264
As for the previous PR will rename and remove all the file in a
secondary PR to avoid conflicts and over loading this one
## Improvements
Removed optional properties from the `objectMetadataEntity` model and
added utils to retrieve test data
## Notes
By touching to `ObjectMetadataDTO` I would have expected a twenty-front
codegenerated types mutation, but it does not seem to be granular enough
to null/undefined coercion
# Introduction
From the moment replaced the FieldMetadataInterface definition to:
```ts
import { FieldMetadataType } from 'twenty-shared/types';
import { FieldMetadataEntity } from 'src/engine/metadata-modules/field-metadata/field-metadata.entity';
export type FieldMetadataInterface<
T extends FieldMetadataType = FieldMetadataType,
> = FieldMetadataEntity<T>;
```
After this PR merge will create a new one removing the type and
replacing it to `FieldMetadataEntity`.
Did not renamed it here to avoid conflicts on naming + type issues fixs
within the same PR
## Field metadata entity RELATION or MORPH
Relations fields cannot be null for those field metadata entity instance
anymore, but are never for the others see
`packages/twenty-server/src/engine/metadata-modules/field-metadata/types/field-metadata-entity-test.type.ts`
( introduced TypeScript tests )
## Concerns
- TS_VECTOR is the most at risk with the `generatedType` and
`asExpression` removal from interface
## What's next
- `FielMetadataInterface` removal and rename ( see introduction )
- Depcrecating `ObjectMetadataInterface`
- Refactor `FieldMetadataEntity` optional fiels to be nullable only
- TO DIG `never` occurences on settings, defaultValue etc
- Some interfaces will be replaced by the `FlatFieldMetadata` when
deprecating the current sync and comparators tools
## Context
Add an eventEmitter instance to twenty datasources so we can emit DB
events.
Add input and output formatting to twenty orm (formatData, formatResult)
Those 2 elements simplified existing logic when we interact with the
ORM, input will be formatted by the ORM so we can directly use
field-like structure instead of column-like. The output will be
formatted, for builder queries it will be in `result.generatedMaps`
where `result.raw` preserves the previous column-like structure.
Important change: We now have an authContext that we can pass when we
get a repository, this will be used for the different events emitted in
the ORM. We also removed the caching for repositories as it was not
scaling well and not necessary imho
Note: An upcoming PR should handle the onDelete: cascade behavior where
we send DESTROY events in cascade when there is an onDelete: CASCADE on
the FK.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
This PR does not produce any functional changes for our users. It
prepares the branches for workflows by:
- decommissioning `output` and `context` fields or `workflowRun` records
and use newly created `state` field from front-end and back-end
- use `stepStatus` computed by `back-end` in `front-end`
- add utils and types in `twenty-shared/workflow` (not completed, a
follow-up is scheduled
https://github.com/twentyhq/core-team-issues/issues/1211)
- add concurrency to `workflowQueue` message queue to avoid weird branch
execution when using forms in workflow branches
- add a WithLock decorator for better dev experience of
`CacheLockService.withLock` usage
Here is an example of such a workflow running (front branch display is
not yet done that's why it looks ugly) ->
https://discord.com/channels/1130383047699738754/1258024460238192691/1392897615171158098
Fixes https://github.com/twentyhq/twenty/issues/12867
Issue:
when you have a variable `toto` which is: `Record<string, MyType>` and
you do toto['xxx'], this will be typed as `MyType` instead of `MyType |
undefined`
Solutions:
- activate `noUncheckedIndexedAccess` check in tsconfig, this is the
preferred solution but will take time to get there (this raises 600+
errors)
- use a Map: cf https://github.com/twentyhq/twenty/pull/13125/files
- set the type to Partial<Record<string, MyType>>. Drawback is that when
you do Object.values(toto), you'll get `Array<MyType | undefined>`.
Hence why we have to filter these behind
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/d0a0bfed-c441-4e53-84c2-2da98ccbcf50"
/>
I rebuilt the advanced filters used in views and workflow search for a
specific filter step.
Components structure remains the same, using `stepFilterGroups` and
`stepFilters`. But those filters are directly sent to backend.
Also re-using the same kind of states we use for advanced filters to
share the current filters used. And a context to share what's coming
from workflow props (function to update step settings and readonly)
⚠️ this PR only focusses on the content of the step. There is still a
lot to do on the filter icon behavior in the workflow
https://github.com/user-attachments/assets/8a6a76f0-11fa-444a-82b9-71fc96b18af4
This PR is purely technical, it does produces any functional change to
the user
- add Lock mecanism to run steps concurrently
- update `workflow-executor.workspace-service.ts` to handle multi branch
workflow execution
- stop passing `context` through steps, it causes race condition issue
- refactor a little bit
- simplify `workflow-run.workspace-service.ts` to prepare `output` and
`context` removal
- move workflowRun status computing from `run-workflow.job.ts` to
`workflow-executor.workspace-service.ts`
## NOTA BENE
When a code step depends of 2 parents like in this config (see image
below)
If the form is submitted before the "Code - 2s" step succeed, the branch
merge "Form" step is launched twice.
- once because form is submission Succeed resumes the workflow in an
asynchronous job
- the second time is when the asynchronous job is launched when "Code -
2s" is succeeded
- the merge "Form" step makes the workflow waiting for response to
trigger the resume in another job
- during that time, the first resume job is launched, running the merge
"Form" step again
This issue only occurs with branch workflows. It will be solved by
checking if the currentStepToExecute is already in a SUCCESS state or
not
<img width="505" alt="image"
src="https://github.com/user-attachments/assets/b73839a1-16fe-45e1-a0d9-3efa26ab4f8b"
/>
Currently, when a server query or mutation from the front-end fails, the
error message defined server-side is displayed in a snackbar in the
front-end.
These error messages usually contain technical details that don't belong
to the user interface, such as "ObjectMetadataCollection not found" or
"invalid ENUM value for ...".
**BE**
In addition to the original error message that is still needed (for the
request response, debugging, sentry monitoring etc.), we add a
`displayedErrorMessage` that will be used in the snackbars. It's only
relevant to add it for the messages that will reach the FE (ie. not in
jobs or in rest api for instance) and if it can help the user sort out /
fix things (ie. we do add displayedErrorMessage for "Cannot create
multiple draft versions for the same workflow" or "Cannot delete
[field], please update the label identifier field first", but not
"Object metadata does not exist"), even if in practice in the FE users
should not be able to perform an action that will not work (ie should
not be able to save creation of multiple draft versions of the same
workflows).
**FE**
To ease the usage we replaced enqueueSnackBar with enqueueErrorSnackBar
and enqueueSuccessSnackBar with an api that only requires to pass on the
error.
If no displayedErrorMessage is specified then the default error message
is `An error occured.`
When we use a record field in a form, record relations are displayed as
available variables in following step.
But those are actually empty at execution.
When choosing the record in the form and submitting, we enrich the
record id with the full record before starting the workflow again. But
we were not adding the relations to that enrichment.
This PR does not produce any functional change
First step of the workflow branch feature
- add gather `workflowRun.output` and `workflowRun.context` into one
column `workflowRun.runContext`
- add a command to fill `runContext` from `output` and `context` in
existing records
- maintain `runContext` up to date during workflow runs
Previous logic was using the previous step output and filtering items
that were passing filters.
What we actually want is:
- send filters, right operand being always a step output key, left
operand being either a key, either a value
- resolve those filter variables
- apply the filters to decide whether the condition is passed or not
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
In this PR
1. fix workflow step creation by adding forgotten
`shouldBypassPermissionChecks` in WorkflowVersionStepWorkspaceService
2. clarify the rule for twentyORMGlobalManager: do not add unnecessary
`shouldBypassPermissionChecks` for system objects (there are no
object-records permission checks on system objects, they are dealt with
at resolver level)
