# Introduction
Extracting the legacy fields build and dispatch out of the object one to
follow the generic flat entity build, also update caches entries for
object
## Main tasks
- flat field map cache
- flat field builder
- refactored the dispatch matrix to return flat entity maps instead of
flat entity arrays
- orchestrator aggregator
- removing legacy code
- making universal identifier aka standardId of standard field for
custom object deterministically dynamic
- perfs debug logs for v2
## TODO
- [x] Refactor the generic entity builder to be dependency flat maps in
order to main foreign keys list in flat parent
- [x] Refactor the flat object metadata to contain the array of related
fields and avoid costy find object fields
- [ ] Improve the create field handler to handle multiple field at once
- [ ] Refactor the dispatch to embbed the comparison
- [ ] Improve perf by extracting from elements out of existing
- [ ] Fix the labelIdentifierId validators on object before field
creation ( integ tests are in failing mode )
## Debug logs snippet
```ts
[EntityBuilder fieldMetadata] matrix computation: 0.027ms
[EntityBuilder fieldMetadata] creation validation: 0.001ms
[EntityBuilder fieldMetadata] deletion validation: 0.293ms
[EntityBuilder fieldMetadata] update validation: 0.006ms
[EntityBuilder fieldMetadata] entity processing: 0.363ms
[EntityBuilder fieldMetadata] validateAndBuild: 0.455ms
[EntityBuilder index] matrix computation: 0.005ms
[EntityBuilder index] creation validation: 0.001ms
[EntityBuilder index] deletion validation: 0.146ms
[EntityBuilder index] update validation: 0.004ms
[EntityBuilder index] entity processing: 0.199ms
[EntityBuilder index] validateAndBuild: 0.228ms
[Runner] Initial cache retrieval: 0.549ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForWorkspaceSchema: 11.665ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForMetadata: 12.864ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 1.476ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 6.816ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 0.062ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 0.889ms
[Runner] Transaction execution: 23.434ms
[Runner] Cache invalidation: 316.662ms
[Runner] Total execution: 340.767ms
```
As you can see cache invalidation is way to long, we could replace the
cache by the optimistic in the end
## Context
-
packages/twenty-server/src/engine/metadata-modules/object-metadata/utils/build-default-relation-flat-field-metadatas-for-custom-object.util.ts
Fixing targetFlatFieldMetadata not being accurate for default relation
during object creation (was not used yet)
-
packages/twenty-server/src/engine/twenty-orm/workspace-schema-manager/services/workspace-schema-foreign-key-manager.service.ts
Simplifying API + Removing unused methods and the ones that were
querying pg schema as we want to avoid those as much as possible
-
packages/twenty-server/src/engine/workspace-manager/workspace-migration-v2/workspace-migration-runner-v2/action-handlers/field/services/create-field-action-handler.service.ts
Adding FK creation when Join column is created
# Introduction
Honestly this implem is a mess, discussing a potential side effect
handler with @weiko before the build and run that would handle each side
effect per entity and operation
Handling:
- [ ] unique index is generated when a field is updated with the
`isUnique`
- [x] an index is generated when a relation is created
- [x] search vector index creation on custom object creation
- [x] renaming a field metadata or an object should re-create all
related indexes which are composed by their namings
- [x] delete object should remove any related indexes
- [x] delete field should update related indexes ( if index ends up
empty it should be removed )
- [ ] on object renaming that contains morph fields -> triggers update
field -> trigger index recompute
- [x] on update name renaming should recompute all related indexes
## TODO
- [x] Integration testing
- [ ] Refactor the index maps cache to be storing a
`idsByObjectMetadataId`
- [x] Refactor deterministic name to use order sorting
- [x] Remove flat index from flat object
## What's next
Will handle morph indexes in a new dedicated PR for the moment will
stick to this
Same for the cache improvement and uniqueness
Enable unique constraint creation on name standard field.
Null values are handled in front with 'Untitled'.
Need to migrate fieldMetadata default value on all custom objects ?
Tested :
- Toggle on/off uniqueness on standard name field of standard/custom
object
- Create new custom text field and toggle on/off uniqueness
Fixes https://github.com/twentyhq/twenty/issues/13577.
When naming or renaming an object, we are already checking that the name
was available compared to other existing objects.
But we also need to check that the relation fields that will be created
or updated are available as well: we create relation fields on
Attachment, Note, Favorite, Task and TimelineActivites, that bear the
name of the object. Thus, it is not possible to create an object that
has the same name as one of the fields on Attachment, Note etc: name,
createdAt, ... are not valid names.
# Introduction
Migrating and improving performances of field enum integrations tests
success and failing tests cases to be using the new v2 api
## Discovered issue
When deleting an object in v1 it will leave related enums until the
object is re-created
Something not done anymore within the create in v2 but in the delete
operation
We should implem an upgrade command to remove such relicas
## Bugs
- Update/create default value multi select runner wrong sql query -> FIX
- Update default value multi select regression, we should allow option
without an id to be inserted -> FIX
- default value compare dynamic json stringify convertion or not in
compare tools for object and fields
## Context
- Adding ts-vector generatedType/asExpression as TS_VECTOR settings
- Using those settings to setup properly tsVector searchVector column
through the new migration runner
- Fix enum creation/suppression
Note: regarding the new tsVector, we should implement a command to
update existing fields
TODO:
- TS_VECTOR search vector column update (note: should be properly
updated whenever the object labelIdentifier is updated or a new TEXT
field is added to the object to follow the current logic)
- relation type fields and columns are not implemented yet
- index migrations
# Introduction
- Implementing the delete one for the new object metadata service v2.
- Handling relation fields and fields in the first place to finally
remove the object
- puting back updatedAt and createdAt in flat metadatas
- duplicate criteria addition for flat object
- removing datasource id from object metadata dto
# Introduction
Introducing v2 object metadata service create one handler v2, not
overkilling the flatObjectMetadata validation for the moment that will
require sequential validation for the import in order to handle
relations and morph relations
# Introduction
Following https://github.com/twentyhq/twenty/pull/13420
What has been done:
- `CreateFieldInput` transpilation to `FlatFieldMetadata`
- `FlatFieldMetadata` validator service
- A lof of transpilation utils from `input` to `flatObject` or
`flatField`
- Created dedicated v2 api metadata services
- Introducing `inferDeletionFromMissingObjectFieldIndex` in the builder,
to avoid diffing every object and field of the current workspace we
allow only generating create/update migration operations, usefull when
passing by the api metadata
## We still need to in another PR:
- Implement a strong unit test coverage and critical functions and
services
- Finalize flat field metadata validation exception for `options`
`defaultValue` `settings` and `relations`
- Finalize `flatObjectMetadata` validation and v2 service refactor
- Plug the new service when feature flag is enabled
# Introduction
In this PR we create basic transpilation methods and utils to handle
input to flat, entity to flat, object maps to flat. In order to
transpile everything into a common validation that will be implemented
in another PR
## FieldMetadataEntity typing
Added `never | null` to fields that should never be in order to ease
general abstracted method to pass null, as anw it's what is in the
database
## Todo
- ~~Create a feature flag~~
- Integration test for object creation through metadata api + pg col
introspection and snapshoting
Fixes https://github.com/twentyhq/twenty/issues/13398
This bug was introduced by https://github.com/twentyhq/twenty/pull/13215
The emails were empty in scenarios where the user wasn't authenticated
(reset passwords for instance) because in `hydrateGraphqlRequest` the
information about the locale was added only if the user was
authenticated `!this.isTokenPresent(request)`. So the locale was
undefined making ` i18n.activate(undefined) ` fail silently resulting in
an empty email.
# Why
If we have a Morph Relation, like :
Opportunity <-> Company & People
Let's say it's a MANY_TO_ONE on Opportunity side
Then we have two joinColumnNames looking like
- ownerPersonId
- ownerCompanyId
Let's say someone renames the obejct Person (assume we can even though
standard obejcts cannot be renames per say at the moment in the API)
We need to update the joinColumnName and create the associated
migrations
# Introduction
Following https://github.com/twentyhq/twenty/pull/13310
> After this PR merge will create a new one removing the type and
replacing it to ObjectMetadataEntity.
This is it !
In the BE we throw custom errors with precise error codes (e.g.
"LABEL_ALREADY_EXISTS") before catching them in filters and rethrowing
BaseGraphQLErrors (standard errors such as NotFoundError, UserInputError
etc.).
In the FE we were grouping sentries based on the error codes but we were
actually grouping by very broad codes such as "NOT_FOUND" or
"BAD_USER_INPUT", extracted from the BaseGraphQLErrors.
To fix that, we update the BaseGraphQLError constructor api to allow to
pass on the CustomError directly and retrieve from it the original code
and store it in existing property `subCode` that we will use in the FE
to send errors to sentry.
This new api also eases usage of `userFriendlyMessage` that is passed on
to the api response and therefore to the FE when CustomError is passed
on directly to the BaseGraphQLError constructor.
Currently, when a server query or mutation from the front-end fails, the
error message defined server-side is displayed in a snackbar in the
front-end.
These error messages usually contain technical details that don't belong
to the user interface, such as "ObjectMetadataCollection not found" or
"invalid ENUM value for ...".
**BE**
In addition to the original error message that is still needed (for the
request response, debugging, sentry monitoring etc.), we add a
`displayedErrorMessage` that will be used in the snackbars. It's only
relevant to add it for the messages that will reach the FE (ie. not in
jobs or in rest api for instance) and if it can help the user sort out /
fix things (ie. we do add displayedErrorMessage for "Cannot create
multiple draft versions for the same workflow" or "Cannot delete
[field], please update the label identifier field first", but not
"Object metadata does not exist"), even if in practice in the FE users
should not be able to perform an action that will not work (ie should
not be able to save creation of multiple draft versions of the same
workflows).
**FE**
To ease the usage we replaced enqueueSnackBar with enqueueErrorSnackBar
and enqueueSuccessSnackBar with an api that only requires to pass on the
error.
If no displayedErrorMessage is specified then the default error message
is `An error occured.`
Better catching label input
- there were absolutely no check on label when creating the target field
while doing a relation : we crearted these checks here.
- We keep the label quite open to special char as discussed with Felix.
so mostly checking length of label.
- We check that label does not already exists on the targetted object
- making sure the Target fieldinput label is checked before we create
it. The previous checks are not enough since the label goes through
anoteher merthod before going in the database
- validate-metadata-name-is-camel-case.utils.ts : making sure we can use
this error message for metadata name and for target label
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
Co-authored-by: prastoin <paul@twenty.com>
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
Update the default set of system fields for custom objects, to ensure
position is not nullabel and has a default value to 0
Steps to reproduce :
create a custom object,
send a POST request with body ```{position:null}```
the record should be created
After the change,
an error will be thrown
<img width="754" alt="Screenshot 2025-06-13 at 17 16 56"
src="https://github.com/user-attachments/assets/d40931f7-16cc-4b68-8dbb-deb0fa292be5"
/>
# What
Fully deprecate old relations because we have one bug tied to it and it
make the codebase complex
# How I've made this PR:
1. remove metadata datasource (we only keep 'core') => this was causing
extra complexity in the refactor + flaky reset
2. merge dev and demo datasets => as I needed to update the tests which
is very painful, I don't want to do it twice
3. remove all code tied to RELATION_METADATA /
relation-metadata.resolver, or anything tied to the old relation system
4. Remove ONE_TO_ONE and MANY_TO_MANY that are not supported
5. fix impacts on the different areas : see functional testing below
# Functional testing
## Functional testing from the front-end:
1. Database Reset ✅
2. Sign In ✅
3. Workspace sign-up ✅
5. Browsing table / kanban / show ✅
6. Assigning a record in a one to many / in a many to one ✅
7. Deleting a record involved in a relation ✅ => broken but not tied to
this PR
8. "Add new" from relation picker ✅ => broken but not tied to this PR
9. Creating a Task / Note, Updating a Task / Note relations, Deleting a
Task / Note (from table, show page, right drawer) ✅ => broken but not
tied to this PR
10. creating a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
11. updating a relation from settings should not be possible ✅
12. deleting a relation from settings (custom / standard x oneToMany /
manyToOne) ✅
13. Make sure timeline activity still work (relation were involved
there), espacially with Task / Note => to be double checked ✅ => Cannot
convert undefined or null to object
14. Workspace deletion / User deletion ✅
15. CSV Import should keep working ✅
16. Permissions: I have tested without permissions V2 as it's still hard
to test v2 work and it's not in prod yet ✅
17. Workflows global test ✅
## From the API:
1. Review open-api documentation (REST) ✅
2. Make sure REST Api are still able to fetch relations ==> won't do, we
have a coupling Get/Update/Create there, this requires refactoring
3. Make sure REST Api is still able to update / remove relation => won't
do same
## Automated tests
1. lint + typescript ✅
2. front unit tests: ✅
3. server unit tests 2 ✅
4. front stories: ✅
5. server integration: ✅
6. chromatic check : expected 0
7. e2e check : expected no more that current failures
## Remove // Todos
1. All are captured by functional tests above, nothing additional to do
## (Un)related regressions
1. Table loading state is not working anymore, we see the empty state
before table content
2. Filtering by Creator Tim Ap return empty results
3. Not possible to add Tasks / Notes / Files from show page
# Result
## New seeds that can be easily extended
<img width="1920" alt="image"
src="https://github.com/user-attachments/assets/d290d130-2a5f-44e6-b419-7e42a89eec4b"
/>
## -5k lines of code
## No more 'metadata' dataSource (we only have 'core)
## No more relationMetadata (I haven't drop the table yet it's not
referenced in the code anymore)
## We are ready to fix the 6 months lag between current API results and
our mocked tests
## No more bug on relation creation / deletion
---------
Co-authored-by: Weiko <corentin@twenty.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
This PR attemps at improving sentry grouping and filtering by
- Using the exceptionCode as the fingerprint when the error is a
customException. For this to work in this PR we are now throwing
customExceptions instead of internalServerError deprived of their code.
They will still be converted to Internal server errors when sent back as
response
- Filtering 4xx issues where it was missing (for emailVerification
because errors were not handled, for invalid captcha and billing errors
because they are httpErrors and not graphqlErrors)
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
In this PR:
- Remove deactivated objects from ActivityTargetInlineCell record picker
- Prevent users to deactivate createdAt, updatedAt, deletedAt fields on
any objects
Still left:
- write unit tests on the assert utils
- write integration tests on field metadata service
- prevent users to deactivate createdAt, updatedAt, deletedAt on FE
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
## Context
Field metadata service was reusing validators from
validate-**OBJECT**-metadata-input which were throwing ObjectMetadata
exceptions not handled in fieldMetadataGraphqlApiExceptionHandler and
were going to Sentry.
To solve the issue since this validator is associated with both fields
and objects I'm moving the util to the root utils folder of metadata
module and throwing a common metadata user input exception
# Introduction
This PR contains several SNAPSHOT files explaining big +
While refactoring the Object Model settings page in
https://github.com/twentyhq/twenty/pull/10653, encountered a critical
issue when submitting either one or both names with `""` empty string
hard corrupting a workspace.
This motivate this PR reviewing server side validation
I feel like we could share zod schema between front and back
## Refactored server validation
What to expect from Names:
- Plural and singular have to be different ( case insensitive and
trimmed check )
- Contains only a-z A-Z and 0-9
- Follows camelCase
- Is not empty => Is not too short ( 1 )
- Is not too long ( 63 )
- Is case insensitive( fooBar and fOoBar now rejected )
What to expect from Labels:
- Plural and singular have to be different ( case insensitive and
trimmed check )
- Is not empty => Is not too short ( 1 )
- Is not too long ( 63 )
- Is case insensitive ( fooBar and fOoBar now rejected )
close https://github.com/twentyhq/twenty/issues/10694
## Creation integrations tests
Created new integrations tests, following
[EachTesting](https://jestjs.io/docs/api#testeachtablename-fn-timeout)
pattern and uses snapshot to assert errors message. These tests cover
several failing use cases and started to implement ones for the happy
path but object metadata item deletion is currently broken unless I'm
mistaken @Weiko is on it
## Notes
- [ ] As we've added new validation rules towards names and labels we
should scan db in order to standardize existing values using either a
migration command or manual check
- [ ] Will review in an other PR the update path, adding integrations
tests and so on
## Context
Following the strategy where we want to block custom object creation
when the type is reserved by core objects. The issue happened again with
the recently introduced role table.
Fixes https://github.com/twentyhq/twenty/issues/8810
Fixes https://github.com/twentyhq/twenty/issues/5268
Fixes https://github.com/twentyhq/twenty/issues/8971
- Fixing Task/Note creation not sending position during creation
- Adding a command to backfill position being null, using existing
backfill command.
- Removed unused backfill job.
- Updated workspace entities to set position non-nullable and set a
default value to make it non-required on the API
- Updated position factory to set a default position for all objects
having a POSITION field instead of only company/people
- Moved the try/catch in each resolver factory calling
GraphqlQueryRunnerException handler, makes more sense to call it in the
actual graphql-query-runner and removing some duplicate codes
- Adding validations for input in QueryRunnerArgs factories
- Allow sync-metadata to override and sync defaultValues for certain
field types (that can't be updated by users)
- Removing health-check from sync-metadata command during force mode to
improve performances
Refactoring update of standard relations when a custom object is
renamed, after observing occasional issues.
---------
Co-authored-by: Weiko <corentin@twenty.com>
This PR adds "type" to the reserved keywords list in
validate-object-metadata-input.util.ts. This prevents users from
creating objects with "type" as a key, which has caused issues in the
past .
issue solved #8543
1.Updated the RESERVED_KEYWORDS array to include "type"
---------
Co-authored-by: Weiko <corentin@twenty.com>
fix#8204
I changed "API keys" to "API values".
Stopped inputting special characters in Select field option keys.
@lucasbordeau please check the changes and tell me if I need to do any
other changes. :)
---------
Co-authored-by: Félix Malfait <felix@twenty.com>
## Context
ObjectMetadataService became quite large and handles too many
responsibilities. I'm trying to refactor a bit this part in preparation
of a larger work that will combine object-metadata services and
sync-metadata logic
- Created a STANDARD_OBJECT_ICONS that can be reused in relation
creation to refer to a standard object icon.
- Created a STANDARD_OBJECT_FIELD_IDS that can be used with an object
name to get its standard field ids.
- Moved migration, record and relation creations to dedicated services,
refactored to improve performances and readability
- Refactored some validation logic
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
## Context
Name shouldn't be added to all tables, especially standard objects
because they already have their own labelIdentifierFieldMetadata
specified in the workspace-entity schema. This PR removes this column
from the "base" list of columns to add when creating a new object/table
and moves it to the object-metadata service that is, as of today, only
used for custom objects. Also had to modify the migration-runner to
handle column creation in a table creation migration (this was available
in the migration definition already but was not doing anything)
This also fixes an issue in standard objects that already have a "name"
field defined with a different field type, this is even more important
when the said field is a composite field. For example people already has
a FULL_NAME name field which clashes with the default TEXT name field
meaning it was only creating 1 field metadata for 'name' but 3 columns
were created: `name, nameFirstName, nameLastName`. This inconsistency
with metadata (which is our source of truth everywhere) brought some
issues (lately, converting back typeorm response to gql (including
composition) was broken).
In this PR:
1. Refactor guards to avoid duplicated queries: WorkspaceAuthGuard and
UserAuthGuard only check for existence of workspace and user in the
request without querying the database
At field creation we are checking the availability of the name by
comparing it to the other fields' names' on the object; but for
composite fields the fields' names' as indicated in the repository do
not exactly match the column names' on the tables (e.g "createdBy" field
is actually represented by columns createdByName, createdBySource etc.).
In this PR we prevent the conflict with the standard composite fields'
names.
There is still room for errors with the custom composite fields: for
example a custom composite field "address" of type address on a custom
object "listing" will introduce the columns addressAddressStreet1,
addressAddressStreet2 etc. while we won't prevent the user from later
creating a custom field named "addressAddressStreet1".
For now I decided not to tackle this as this seem extremely edgy + would
impact performance on creation of all fields while never actually useful
(I think).
In this PR, I'm:
- adding createdBy field (type ACTOR) on custom objects when created
- moving `name` and `position` default column to the set of columns
automatically creation on object creation
- fixing a bug on mutations (update / create), if the targetted object
has a 'data' custom field, it was conflicting with the payload ==> I
feel we need to refactor this part of the code but we can keep this for
a bit later as we plan to move out of pg_graphql
<img width="1198" alt="image"
src="https://github.com/user-attachments/assets/891c4a97-bab1-415c-8551-dabd5996a794">
When migrating the option values of a select type, if the field is non
nullable (for now, only available for opportunity's "stage" standard
field), we fallback to the (potentially updated) default value instead
of nullifying the value to avoid getting a database error.
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
