<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Implements last-modifier tracking and unified actor injection.
>
> - New `ActorFromAuthContextService` replaces createdBy-only logic;
pre-query hooks now inject both `createdBy` and `updatedBy` on create
and `updatedBy` on update
> - Add `updatedBy` standard field to core/custom objects (e.g.,
`person`, `company`, `task`, `note`, `attachment`, `dashboard`,
`workflow`, `workflowRun`) with IDs, metadata builders, and ORM entity
fields
> - Record CRUD: `create` now sets both `createdBy` and `updatedBy`;
`update` sets `updatedBy`; workflow actions use a shared workflow actor
builder; REST base handler uses the new actor service
> - Seeder data and snapshots updated to include `updatedBy`; GraphQL
result formatting adds defaults/handling for empty composite fields
(incl. actor)
> - Frontend `useUpdateOneRecord` upserts returned record into the local
store after mutation
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
1f283778e9. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
## Context
Deprecating the old objectMetadataMap type in favour of split flat
entities to match with our new caching.
In the long run, trying to achieve:
- Better performance through caching
- Consistent data access patterns across the codebase
- Reduced database queries
Now that everything is based on flat entities, which are cached, we can
finish the refactoring of workspace context cache which should already
improve performances.
Then the last step will be to consume that new cache in the new global
datasource to get rid of the many workspace datasources stored in the
server
@charlesBochet
In workflow codebase, NULL (instead of empty object) is expected on
workflow version object step field, when a new workflow is created for
example.
(packages/twenty-front/src/modules/workflow/workflow-diagram/utils/generateWorkflowDiagram.ts
- 42)
This case is not isolated and it creates many issues.
We decided to format NULL value to equivalent (empty string for text
field, empty object for raw_json) but it seems it complicates the dev x.
To unlock @Devessier I prefer revert the logic, the time we discuss how
to solve this cases.
closes https://github.com/twentyhq/core-team-issues/issues/1629
To do before requesting review :
- filter update
Migration to come in an other PR
Strat :
1/ Null transformation
- [x] Transform NULL equivalent value to NULL in field validation in
common api - pre-query - with feature flag
- [ ] Same logic in ORM (Not done, complex to handle feature flag here)
- [x] Transform NULL value to equivalent in data formatting in ORM -
post-query
2/ Migration (in other PR) for fieldMetadata not nullable with default
defaultValue (empty string, ...)
- [ ] Remove NOT NULL db constraint
- [ ] Update record value to NULL
- [ ] Update field metadata : isNullable:true
- [ ] Update uniqueIndex whereClause (also for standard uniqueIndex)
- [ ] Activate feature flag
3/ Update metadata creation
- [x] No more default default value
- [x] Update standard field nullability
- [x] Remove index default whereClause for standard field
4/ Update filter
- [x] When filtering on NULL or empty string, be sure all records are
returned (the one with NULL + the one with "")
5/ Test
- [ ] Strat. to do
This PR implements the necessary tools to have `react-datepicker`
calendar and our date picker components work reliably no matter the
timezone difference between the user execution environment and the user
application timezone.
Fixes https://github.com/twentyhq/core-team-issues/issues/1781
This PR won't cover everything needed to have Twenty handle timezone
properly, here is the follow-up issue :
https://github.com/twentyhq/core-team-issues/issues/1807
# Features in this PR
This PR brings a lot of features that have to be merged together.
- DATE field type is now handled as string only, because it shouldn't
involve timezone nor the JS Date object at all, since it is a day like a
birthday date, and not an absolute point in time.
- DATE_TIME field wasn't properly handled when the user settings
timezone was different from the system one
- A timezone abbreviation suffix has been added to most DATE_TIME
display component, only when the timezone is different from the system
one in the settings.
- A lot of bugs, small features and improvements have been made here :
https://github.com/twentyhq/core-team-issues/issues/1781
# Handling of timezones
## Essential concepts
This topic is so complex and easy to misunderstand that it is necessary
to define the precise terms and concepts first. It resembles character
encoding and should be treated with the same care.
- Wall-clock time : the time expressed in the timezone of a user, it is
distinct from the absolute point in time it points to, much like a
pointer being a different value than the value that it points to.
- Absolute time : a point in time, regardless of the timezone, it is an
objective point in time, of course it has to be expressed in a given
timezone, because we have to talk about when it is located in time
between humans, but it is in fact distinct from any wall clock time, it
exists in itself without any clock running on earth. However, by
convention the low-level way to store an absolute point in time is in
UTC, which is a timezone, because there is no way to store an absolute
point in time without a referential, much like a point in space cannot
be stored without a referential.
- DST : Daylight Save Time, makes the timezone shift in a specific
period every year in a given timezone, to make better use of longer days
for various reasons, not all timezones have DST. DST can be 1 hour or 30
min, 45 min, which makes computation difficult.
- UTC : It is NOT an “absolute timezone”, it is the wall-clock time at
0° longitude without DST, which is an arbitrary and shared human
convention. UTC is often used as the standard reference wall-clock time
for talking about absolute point in time without having to do timezone
and DST arithmetic. PostgreSQL stores everything in UTC by convention,
but outputs everything in the server’s SESSION TIMEZONE.
## How should an absolute point in time be stored ?
Since an absolute point in time is essentially distinct from its
timezone it could be stored in an absolute way, but in practice it is
impossible to store an absolute point in time without a referential. We
have to say that a rocket launched at X given time, in UTC, EST, CET,
etc. And of course, someone in China will say that it launched at 10:30,
while in San Francisco it will have launched at 19:30, but it is THE
SAME absolute point in time.
Let’s take a related example in computer science with character
encoding. If a text is stored without the associated encoding table, the
correct meaning associated to the bits stored in memory can be lost
forever. It can become impossible for a program to guess what encoding
table should be used for a given text stored as bits, thus the glitches
that appeared a lot back in the early days of internet and document
processing.
The same can happen with date time storing, if we don’t have the
timezone associated with the absolute point in time, the information of
when it absolutely happened is lost.
It is NOT necessary to store an absolute point in time in UTC, it is
more of a standard and practical wall-clock time to be associated with
an absolute point in time. But an absolute point in time MUST be store
with a timezone, with its time referential, otherwise the information of
when it absolutely happened is lost.
For example, it is easier to pass around a date as a string in UTC, like
`2024-01-02T00:00:00Z` because it allows front-end and back-end code to
“talk” in the same standard and DST-free wall-clock time, BUT it is not
necessary. Because we have date libraries that operate on the standard
ISO timezone tables, we can talk in different timezone and let the
libraries handle the conversion internally.
It is false to say that UTC is an absolute timezone or an absolute point
in time, it is just the standard, conventional time referential, because
one can perfectly store every absolute points in time in UTC+10 with a
complex DST table and have the exactly correct absolute points in time,
without any loss of information, without having any UTC+0 dates
involved.
Thus storing an absolute point in time without a timezone associated,
for example with `timestamp` PostgreSQL data type, is equivalent to
storing a wall-clock time and then throwing away voluntarily the
information that allows to know when it absolutely happened, which is a
voluntary data-loss if the code that stores and retrieves those
wall-clock points in time don’t store the associated timezone somewhere.
This is why we use `timestamptz` type in PostgreSQL, so that we make
sure that the correct absolute point in time is stored at the exact time
we send it to PostgreSQL server, no matter the front-end, back-end and
SQL server's timezone differences.
## The JavaScript Date object
The native JavaScript Date object is now officially considered legacy
([source](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date)),
the Date object stores an absolute point in time BUT it forces the
storage to use its execution environment timezone, and one CANNOT modify
this timezone, this is a legacy behavior.
To obtain the desired result and store an absolute point in time with an
arbitrary timezone there are several options :
- The new Temporal API that is the successor of the legacy Date object.
- Moment / Luxon / @date-fns/tz that expose objects that allow to use
any timezone to store an absolute point in time.
## How PostgreSQL stores absolute point in times
PostgreSQL stores absolute points in time internally in UTC
([source](https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME-INPUT-TIME-STAMPS)),
but the output date is expressed in the server’s session timezone
([source](https://www.postgresql.org/docs/current/sql-set.html)) which
can be different from UTC.
Example with the object companies in Twenty seed database, on a local
instance, with a new “datetime” custom column :
<img width="374" height="554" alt="image"
src="https://github.com/user-attachments/assets/4394cb43-d97e-4479-801d-ca068f800e39"
/>
<img width="516" height="524" alt="image"
src="https://github.com/user-attachments/assets/b652f36a-d2e2-47a4-8950-647ca688cbbd"
/>
## Why can’t I just use the JavaScript native Date object with some
manual logic ?
Because the JavaScript Date object does not allow to change its internal
timezone, the libraries that are based on it will behave on the
execution environment timezone, thus leading to bugs that appear only on
the computers of users in a timezone but not for other in another
timezone.
In our case the `react-datepicker` library forces to use the `Date`
object, thus forcing the calendar to behave in the execution environment
system timezone, which causes a lot of problems when we decide to
display the Twenty application DATE_TIME values in another timezone than
the user system one, the bugs that appear will be of the off-by-one date
class, for example clicking on 23 will select 24, thus creating an
unreliable feature for some system / application timezone combinations.
A solution could be to manually compute the difference of minutes
between the application user and the system timezones, but that’s not
reliable because of DST which makes this computation unreliable when DST
are applied at different period of the year for the two timezones.
## Why can’t I compute the timezone difference manually ?
Because of DST, the work to compute the timezone difference reliably,
not just for the usual happy path, is equivalent to developing the
internal mechanism of a date timezone library, which is equivalent to
use a library that handles timezones.
## Using `@date-fns/tz` to solve this problem
We could have used `luxon` but it has a heavy bundle size, so instead we
rely here on `@date-fns/tz` (~1kB) which gives us a `TZDate` object that
allows to use any given timezone to store an absolute point-in-time.
The solution here is to trick `react-datepicker` by shifting a Date
object by the difference of timezone between the user application
timezone and the system timezone.
Let’s take a concerte example.
System timezone : Midway, ⇒ UTC-11:00, has no DST.
User application timezone : Auckland, NZ ⇒ UTC+13:00, has a DST.
We’ll take the NZ daylight time, so that will make a timezone difference
of 24 hours !
Let’s take an error-prone date : `2025-01-01T00:00:00` . This date is
usually a good test-case because it can generate three classes of bugs :
off-by-one day bugs, off-by-one month bugs and off-by-one year bugs, at
the same time.
Here is the absolute point in time we take expressed in the different
wall-clock time points we manipulate
Case | In system timezone ⇒ UTC-11 | In UTC | In user application
timezone ⇒ UTC+13
-- | -- | -- | --
Original date | `2024-12-31T00:00:00-11:00` | `2024-12-31T11:00:00Z` |
`2025-01-01T00:00:00+13:00`
Date shifted for react-datepicker | `2025-01-01T00:00:00-11:00` |
`2025-01-01T11:00:00Z` | `2025-01-02T00:00:00+13:00`
We can see with this table that we have the number part of the date that
is the same (`2025-01-01T00:00:00`) but with a different timezone to
“trick” `react-datepicker` and have it display the correct day in its
calendar.
You can find the code in the hooks
`useTurnPointInTimeIntoReactDatePickerShiftedDate` and
`useTurnReactDatePickerShiftedDateBackIntoPointInTime` that contain the
logic that produces the above table internally.
## Miscellaneous
Removed FormDateFieldInput and FormDateTimeFieldInput stories as they do
not behave the same depending of the execution environment and it would
be easier to put them back after having refactored FormDateFieldInput
and FormDateTimeFieldInput
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
This PR aims to improve performances as we are making A LOT of queries
after having added events emission to the ORM layer. While investigating
issues, I've come across multiple problems I will describe below
## Add index on workspace.activationStatus as we are querying it a lot
As per title
## Add logs on core datasource destroy
It seems that we have postgres connection pool destruction in
production. I cannot reproduce locally but I suspect the Query Timeouts
to be the root cause. I'm fixing most of the Query Timeouts cause in
this PR but I'm adding the logs so we have more information in case in
keeps happening in production.
## GraphQL query runner createMany
It was using a for loop on each record. This is as issue has we will
emit an event separately for each record => we should always try to
batch events.
Replacing by a save on all records. Note that this is not perfect as we
should avoid using save (bad performances), and use insert + updateMany
instead. As I have follow up discussions regarding permissions, I
haven't replaced it by insert and updateMany yet. Using save instead of
a for loop makes the problem less worrying.
## Introduce updateMany in ORM @Weiko @ijreilly FYI
.save(manyRecords) is bad as it's querying the data for no good reason
(and doing a select for each record...).
We already have .insert(), i'm introducing .updateMany()
I think our ORM layer should be simplified a lot but I'm not starting
the refacto yet
## Fixing ORM @Weiko @ijreilly FYI
- Fixing events emission in delete function
- Fixing events emission in insert function
- make sure everything is batched
## Events performance @Weiko @ijreilly FYI
Do not emit timelineActivity db events as this does not seem useful and
is quite heavy
## Messaging and Calendar performance @bosiraphael FYI
Rework many functions to make sure they are batched. This does not touch
the driver layer and I have heavily tested it. (multiple account, with
multiple channels, common thread, common messages, etc...)
## Workflow Trigger relation Fetch performance @martmull FYI
Improved performance by batching
# Introduction
Following https://github.com/twentyhq/twenty/pull/13264
> After this PR merge will create a new one removing the type and
replacing it to FieldMetadataEntity.
This is it !
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
This is a first PR to remove old relation logic
Next steps:
- remove relationMetadata from cache
- remove relationMetadata table content and structure
- refactor relationDefinition to leverage field.settings instead
# Introduction
Followup of https://github.com/twentyhq/twenty/pull/11784
Again some propaganda for the
[noUncheckedIndexedAccess](https://www.typescriptlang.org/tsconfig/#noUncheckedIndexedAccess)
that involved this @charlesBochet 👀
That's very risky to have this setup to false especially in the backend
regarding the twenty-server metadata nature
Also suggested that we do some integration testing ( e2e nestsJs tests )
on a related endpoint that could related we always retrieve the same
form result output
We could also do some unit testing of the method but like the idea to
ship it through the api itself
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
## Context
All objects have '...duplicates' resolver but only companies and people
have duplicate criteria (hard coded constant).
Gql schema and resolver should be created only if duplicate criteria
exist.
## Solution
- Add a new @WorkspaceDuplicateCriteria decorator at object level,
defining duplicate criteria for given object.
- Add a new duplicate criteria field in ObjectMetadata table
- Update schema and resolver building logic
- Update front requests for duplicate check (only for object with
criteria defined)
closes https://github.com/twentyhq/twenty/issues/9828
This PR fixes a problem with how TypeORM handles date without time.
A date without time that is stored in PostgreSQL database as `date` type
gets returned as an ISO string date with a timezone that can shift its
date part in an unwanted way.
In short DB stores `2025-01-01`, TypeORM query builder returns
`2024-12-31T23:00:00Z` which gets parsed as `2024-12-31` on the front
end field.
We don't want to handle timezone here because we are manipulating a date
without its time part, so this PR adds a step that counteracts what
TypeORM does and returns `2025-01-01T00:00:00.000Z` so that the front
can parse it correctly.
@Weiko We might want to check other places of the backend where date
types are returned by TypeORM, we might have the same problem, this PR
only fixes it for updateOne resolver return.
- Fixed date persist on frontend which was shifting the date to a
different day due to timezone issue
- Fixed date returned by the backend update logic, which was shifting
the date by the timezone offset (so this PR adds back the offset so that
it stays at 00:00:00Z time)
Fixes https://github.com/twentyhq/twenty/issues/8300
## Context
API events were created too late and were already formatted as Gql
responses (including nesting with edges/node/type + formatting that
should not exist in an event payload). This PR moves the emit logic to
the resolver where we actually do the DB query
Note: Also added RESTORED events
Fixes: #8224
1. Summary
All multi-dropdowns across the application don't accept empty options
because emptiness is perceived as an invalid state.
2. Solution
The issue came down to the back-end and the specific string utility
function that converts a string separated by `,` to an array of options.
But the problem with it is that it returns `['']` for an empty string
representing an emptiness. And then `['']` fails on the parser because
simply `''` is not a valid option the user selected for the dropdown. So
I updated the string utility function to return an empty array for cases
like `'', '{}', '{ }'`, etc
3. Recording
https://github.com/user-attachments/assets/071fe5d2-2123-4deb-878c-67f62d9b3431
Fixes https://github.com/twentyhq/twenty/issues/6859
This PR adds all the remaining resolvers for
- updateOne/updateMany
- createOne/createMany
- deleteOne/deleteMany
- destroyOne
- restoreMany
Also
- refactored the graphql-query-runner to be able to add other resolvers
without too much boilerplate.
- add missing events that were not sent anymore as well as webhooks
- make resolver injectable so they can inject other services as well
- use objectMetadataMap from cache instead of computing it multiple time
- various fixes (mutation not correctly parsing JSON, relationHelper
fetching data with empty ids set, ...)
Next steps:
- Wrapping query builder to handle DB events properly
- Move webhook emitters to db event listener
- Add pagination where it's missing (findDuplicates, nested relations,
etc...)
