## Summary
Adds support for filtering records by fields on a related MANY_TO_ONE
object via the GraphQL API. Backend only — no frontend, no REST, no
view-filter persistence yet.
```graphql
{
people(filter: { company: { name: { like: "%Airbnb%" } } }) {
edges { node { id } }
}
}
```
### Where the work lands
- **Schema** — `relation-field-metadata-gql-type.generator.ts` now emits
`{relationName}: TargetFilterInput` alongside the existing
`{joinColumnName}: UUIDFilter` for MANY_TO_ONE relations. Mirrors the
order-by generator that already does this for sort. Lazy thunks in
`object-metadata-filter-gql-input-type.generator.ts` handle the cycle
between filter inputs.
- **Arg processor** — `FilterArgProcessorService` no longer hard-rejects
accessing a relation by its name. When the value is a nested object on a
MANY_TO_ONE field, it recurses into the target object's metadata so each
leaf still gets validated and coerced. Depth-capped at 1.
- **Query parser** — new `parseRelationSubFilter` branch in
`graphql-query-filter-field.parser.ts`. When triggered: looks up the
target object metadata, calls `ensureRelationJoin` against the outer
query builder, and recurses via a child
`GraphqlQueryFilterConditionParser` scoped to the target.
`and`/`or`/`not` inside the relation filter keep working because the
child dispatches through the same `parseKeyFilter`.
- **Shared join utility** — `ensureRelationJoin.util.ts` is a single
function that inspects `queryBuilder.expressionMap.joinAttributes` for
the alias before adding a `LEFT JOIN`. Rewired the existing inline
`qb.leftJoin` calls in the order parser and group-by service to use it,
so filter-driven joins no longer collide with sort-driven joins on the
same relation.
### Out of scope (explicit)
- ONE_TO_MANY reverse traversal (needs EXISTS subqueries)
- Aggregates (`company.people.count > 5` — needs HAVING)
- View-filter storage (no `relationPath` column on `ViewFilterEntity`)
- REST DSL changes
- Frontend filter-picker UX
- Nesting deeper than one hop (parser and arg-processor both reject)
### Open question for review
Permissions. The order-by-on-relation code path already lets users sort
People by Company.name without a Company read-permission check, and this
PR matches that behavior for filters — felt wrong to add a stricter gate
only on the filter side. If we want object-permission gating on the
relation target, it should be a follow-up that covers both paths
consistently. The only attack surface today is existence inference via
timing, identical to what sort already exposes.
## Test plan
- [x] `tsc --noEmit` — clean for changed files (5 unrelated pre-existing
errors on main untouched)
- [x] `oxlint --type-aware` + `prettier --check` — 0 errors on all 17
changed/new files
- [x] `jest filter-arg-processor.service.spec` — 229 tests pass (the new
optional `flatObjectMetadataMaps` arg is backwards-compatible)
- [x] Integration test (`filter-by-relation-field.integration-spec.ts`,
6 cases) — needs to be verified against a seeded test DB. Could not
exercise the happy path in my isolated worktree; depth-2 rejection
passed there.
- [ ] EXPLAIN ANALYZE on the integration test query to confirm the FK on
`person.companyId` is indexed for both standard and custom MANY_TO_ONE
relations.
### Integration test cases
1. Filter People by `company.name = "Airbnb"` (exact match)
2. Filter People by `company.name like "%irbnb%"`
3. Non-matching filter returns empty
4. Combined with a scalar filter at root via `and`
5. **Combined with `orderBy` on the same relation** — proves the
join-dedupe works (without `ensureRelationJoin`, TypeORM throws
"duplicate alias")
6. Depth-2 nesting (`company.accountOwner.name`) returns
`INVALID_ARGS_FILTER`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
## Summary
`joinColumnName` on relation field settings is always derivable from the
field name (and the target object name for morph relations). This PR
stops reading it from settings anywhere in production code; the stored
value is no longer used.
The settings field is **not** removed from data yet — a follow-up can
drop it once we are confident nothing depends on the stored value.
## Helpers
The helpers are split by layer because frontend and backend hold morph
relations differently: the frontend has a base name plus a
`morphRelations[]` array, the backend has one row per target with the
name already morph-resolved.
| Helper | Layer | When to use |
|---|---|---|
| `computeRelationGqlFieldJoinColumnName` | Shared / frontend
(`gqlField`) | Non-morph relation on the frontend. |
| `computeMorphRelationGqlFieldName` | Shared / frontend (`gqlField`) |
Need the per-target morph gqlField name (e.g. `targetCompany`). |
| `computeMorphRelationGqlFieldJoinColumnName` | Shared / frontend
(`gqlField`) | Per-target morph join column on the frontend. Prefer over
the non-morph helper for any morph field — it forces the per-target
inputs. |
| `computeMorphOrRelationFieldJoinColumnName` | Backend
(`FlatFieldMetadata.name`) | Any backend read or write — the flat name
is already morph-resolved, so one helper covers both cases. |
| `computeMorphRelationFlatFieldName` | Backend
(`FlatFieldMetadata.name`) | **Mutation paths only** (create / update /
object rename). Reads consume the stored `field.name` and never call
this. |
## Test plan
- [x] Typecheck and lint (front, server, shared)
- [x] Existing unit tests pass
- [ ] CI green
# Summary
(fixes#20044)
This PR implements two fixes for the REST API to enforce stricter
validation and provide better error messages.
Issue 1: Cursor parameter silently ignored
Problem: When users provided common cursor aliases (e.g., cursor, after,
before) instead of the correct parameter names (starting_after,
ending_before), the API silently ignored them and returned page 1 on
every request.
Solution: Added strict validation to detect common cursor aliases and
throw a clear error directing users to use the correct parameter names.
Files modified:
-
packages/twenty-server/src/engine/api/rest/input-request-parsers/rest-input-request-parser.exception.ts
-
packages/twenty-server/src/engine/api/rest/input-request-parsers/starting-after-parser-utils/parse-starting-after-rest-request.util.ts
-
packages/twenty-server/src/engine/api/rest/input-request-parsers/ending-before-parser-utils/parse-ending-before-rest-request.util.ts
- Test files for both parsers
Example error:
Invalid cursor parameter 'cursor'. Use 'starting_after' for pagination.
---
Issue 2: OpportunityStageEnum not validated on REST
Problem: When creating or updating opportunities via REST with an
invalid stage value, the API either silently dropped the value or
returned a generic error without listing valid options.
Solution: Updated the SELECT field validation to include valid options
in the error message.
Files modified:
-
packages/twenty-server/src/engine/api/common/common-args-processors/data-arg-processor/validator-utils/validate-rating-and-select-field-or-throw.util.ts
- Test file
Example error:
Invalid value "BAD_VALUE" for field "stage". Valid values are: NEW,
SCREENING, MEETING, PROPOSAL, CUSTOMER
---
### Testing
- Added 5 new test cases for `parse-starting-after-rest-request.util.ts`
- Added 6 new test cases for `parse-ending-before-rest-request.util.ts`
- Added 1 new test case for
`validate-rating-and-select-field-or-throw.util.ts`
- All 21 tests passing
---
Breaking Changes
None - correct usage is unaffected.
---------
Co-authored-by: Etienne <45695613+etiennejouan@users.noreply.github.com>
Bug fixes exposed by always-on direct execution
1. GraphQL spec compliance — data[field] = null on resolver error
direct-execution.service.ts — Changed from Promise.allSettled (which
lost the responseKey on rejection) to Promise.all with per-field
try/catch; errors now set data[responseKey] = null per spec
2. Empty object arguments skipped (extractArgumentsFromAst)
extract-arguments-from-ast.util.ts — Removed isEmptyObject check;
filter: {}, data: {} now correctly passed to resolvers instead of
silently dropped (which caused permissions to never be checked)
3. orderBy: {} factory default treated as "no ordering"
direct-execution.service.ts — Before calling the resolver, strips
orderBy: {} and orderByForRecords: {} (empty-object factory defaults
that mean "no ordering")
assert-find-many-args.util.ts / assert-group-by-args.util.ts — Accept {}
for orderBy without throwing
4. orderBy: { field: '...' } object auto-coerced to [{ field: '...' }]
array
direct-execution.service.ts — Applies GraphQL list coercion: a
non-array, non-empty orderBy object is wrapped in an array before
assertion and resolver call
5. totalCount and aggregate fields returned as strings from PostgreSQL
graphql-format-result-from-selected-fields.util.ts — Added
coerceAggregateValue that parses numeric strings to numbers for
totalCount, sum*, avg*, min*, max*, count*, percentageOf* fields
Test updates
nested-relation-queries.integration-spec.ts — Updated expected error
message from Yoga schema-validation message to direct execution resolver
message
~30 snapshot files — Updated to reflect direct execution's error
messages (different from Yoga schema-validation messages for input type
errors)
#### Direct Execution __typename & null backfill
##### __typename filling
The direct execution path now correctly derives __typename at every
level of the GraphQL response:
Connection types — CompanyConnection, CompanyEdge, Company, PageInfo
GroupBy types — TaskGroupByConnection (was incorrectly producing
TaskConnection)
Composite fields — Links, FullName, Currency, etc. handled by a
dedicated formatter (was inheriting the parent object's typename)
Previously, __typename was derived from the object's universal
identifier (a UUID), producing broken values like
20202020B3744779A56180086Cb2E17FConnection.
##### Null backfill
Selected fields missing from the resolver result are backfilled with
null, matching the standard Yoga schema behavior.
##### Integration test
A new test runs the same findMany query (with __typename at all
structural levels) through both paths — standard Yoga schema and direct
execution — and asserts identical output via toStrictEqual.
## Summary
- **Backend**: Add JSON validation for the `blocknote` subfield in rich
text API inputs — rejects values that aren't valid JSON or aren't arrays
(BlockNote content is always `PartialBlock[]`). This prevents corrupted
data from being persisted to the database.
- **Frontend**: Replace all 5 unprotected `JSON.parse` calls on
blocknote content with the safe `parseJson` utility from
`twenty-shared`. Invalid content now degrades gracefully (empty block /
empty string / unchanged passthrough) instead of crashing the app.
- **Tests**: Added integration tests for invalid blocknote JSON (both
GraphQL and REST), unit tests for the new validation, and updated
existing test constants to use valid BlockNote JSON.
## Context
A user reported a `SyntaxError: Expected ',' or ']' after array element`
crash caused by malformed blocknote JSON stored in the database. The
data had `"children":[]` nested inside the `content` array instead of as
a sibling property. The API accepted this invalid JSON because it only
validated that `blocknote` was a string, not that it contained valid
JSON. On the frontend, 5 call sites used bare `JSON.parse` with no error
handling, causing a white-screen crash.
## Test plan
- [x] Unit tests pass: `validate-rich-text-field-or-throw.util.spec.ts`
(10/10)
- [x] Integration tests pass: `rich-text-field-create-input-validation`
(8/8)
- [ ] Verify creating a note with valid rich text still works end-to-end
- [ ] Verify API returns clear error when blocknote contains invalid
JSON
- [ ] Verify frontend renders empty block instead of crashing when
encountering corrupted data
🤖 Generated with [Claude Code](https://claude.com/claude-code)
## Summary
- Renames the `FieldMetadataType` enum key from `RICH_TEXT_V2` to
`RICH_TEXT` across the entire codebase, while keeping the underlying
string value as `'RICH_TEXT_V2'` to maintain PostgreSQL database
compatibility
- Renames all related types, guards, hooks, components, and files from
`*RichTextV2*` / `*rich-text-v2*` to `*RichText*` / `*rich-text*` (e.g.
`FormRichTextV2FieldInput` → `FormRichTextFieldInput`,
`isFieldRichTextV2` → `isFieldRichText`)
- Updates generated files (GraphQL schema, SDK types) to use the new key
while preserving the `RICH_TEXT_V2` string value for DB/API layer
- Updates i18n locale files, test snapshots, and integration tests to
reflect the rename
## Context
The legacy `RICH_TEXT` (V1) field type was deprecated and migrated to
`TEXT` in a previous PR (#18623). With V1 gone, the `RICH_TEXT_V2`
naming is no longer necessary — `RICH_TEXT` is now the canonical name.
The DB enum value stays `'RICH_TEXT_V2'` to avoid confusion with the
just-deprecated V1 type and to prevent a database migration.
## Test plan
- [x] `twenty-server` typecheck passes
- [x] `twenty-front` typecheck passes (only pre-existing Apollo client
errors remain)
- [x] `twenty-server` lint passes
- [x] `twenty-front` lint passes
- [x] `twenty-shared` build passes
- [ ] CI passes
Made with [Cursor](https://cursor.com)
## Summary
- Removes the deprecated `RICH_TEXT` (V1) field metadata type from the
codebase entirely
- Adds a 1.20 upgrade command that migrates existing `RICH_TEXT` fields
to `TEXT` in `core.fieldMetadata`
- Cleans up ~70 files across `twenty-shared`, `twenty-server`,
`twenty-front`, `twenty-sdk`, and `twenty-zapier`
## Context
`RICH_TEXT` was a legacy field type that stored rich text as a single
`text` column. It was already **read-only** — writes threw errors
directing users to `RICH_TEXT_V2` instead. `RICH_TEXT_V2` is the current
approach: a composite type with `blocknote` (editor JSON) and `markdown`
subfields. Keeping the deprecated type added maintenance burden without
any value.
Since the underlying database column type for `RICH_TEXT` was already
`text` (same as `TEXT`), the migration only needs to update the metadata
— no data migration or column changes required.
## Changes
### Upgrade command (new)
- `1-20-migrate-rich-text-to-text.command.ts` — runs `UPDATE
core."fieldMetadata" SET "type" = 'TEXT' WHERE "type" = 'RICH_TEXT'` per
workspace, with cache invalidation
### Enum & shared types
- Removed `RICH_TEXT` from `FieldMetadataType` enum
- Removed from `FieldMetadataDefaultValueMapping`,
`isFieldMetadataTextKind`
### Server (~30 files)
- Removed from type mapper (scalar, filter, order-by), data processors,
input transformer, filter operators, zod schemas, column type mapping,
searchable fields, RLS matching, OpenAPI schema, fake value generators
- Removed from field creation flow and field metadata type validator
- Updated dev seeder Pet `bio` field to `TEXT`
- Cleaned up mocks, snapshots, integration tests
### Frontend (~25 files)
- Deleted: `RichTextFieldDisplay`, `isFieldRichText`,
`isFieldRichTextValue`, `useRichTextFieldDisplay`
- Removed from `FieldDisplay`, `usePersistField`, `isFieldValueEmpty`,
`isRecordMatchingFilter`, `generateEmptyFieldValue`,
`isFieldCellSupported`, spreadsheet import, workflow fake values
- Removed from settings types, field type configs, and field creation
exclusion list
- Updated tests, mocks, and stories
### SDK & Zapier
- Removed from generated GraphQL schema and TypeScript types
- Removed from Zapier `computeInputFields`
Closes https://github.com/twentyhq/core-team-issues/issues/1627
**FilterArgProcessor consolidation:**
Refactored to both validate AND transform filter values in a single pass
Coerced string inputs to native types (e.g., "1" → 1, "true" → true -
useful for Rest input)
Returns transformed filter instead of just validating
Removed overrideFilterByFieldMetadata calls from all computeArgs methods
**QueryRunnerArgsFactory cleanup**
**Testing:**
Add unit testing
uncomment integration tests
Fixes https://github.com/twentyhq/twenty/issues/17138
- Backend should have strict date/dateTime format validation
- FE in import csv is more permissive
---------
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: cubic-dev-ai[bot] <191113872+cubic-dev-ai[bot]@users.noreply.github.com>
- Migration command
- Check IS_FILES_FIELD_MIGRATED:false
- Check or create avatarFile field
- Fetch all people with avatarUrl
- Move (Copy/move) file in storage
- Create core.file record
- Update person record
- bonus : attachment migration : fullPath > file (same logic)
- BE logic
- Add avatarFile field on person
- FE logic
- Adapt logic to upload on/display avatarFile data
The whole imageIdentifier logic will be done later
## Fix resolver schema leaking between `/metadata` and `/graphql`
endpoints
### Summary
- Patch `@nestjs/graphql` to support a `resolverSchemaScope` option that
filters resolvers at both schema generation and runtime, preventing
cross-endpoint leaking
- Introduce `@CoreResolver()` and `@MetadataResolver()` decorators to
explicitly scope each resolver to its endpoint
- Move most resolvers (auth, billing, workspace, user, etc.) to the
metadata schema where the frontend expects them; only workflow and
timeline calendar/messaging resolvers remain on `/graphql`
- Fix frontend `SSEQuerySubscribeEffect` to use the default (metadata)
Apollo client instead of the core client
### Problem
NestJS GraphQL's module-based resolver discovery traverses transitive
imports, causing resolvers from `/metadata` modules to leak into the
`/graphql` schema and vice versa. This made the schemas unpredictable
and tightly coupled to module import order.
### Approach
- Added `resolverSchemaScope` to `GqlModuleOptions` via a patch on
`@nestjs/graphql`, filtering in both `filterResolvers()` (runtime
binding) and `getAllCtors()` (schema generation)
- Each resolver is explicitly decorated with `@CoreResolver()` or
`@MetadataResolver()`
- Organized decorator, constant, and type files under `graphql-config/`
following project conventions
Core GQL Schema: (see: no more fields!)
<img width="827" height="894" alt="image"
src="https://github.com/user-attachments/assets/668f3f0f-485e-43f0-92be-4345aeccacb6"
/>
Metadata GQL Schema (see no more getTimelineCalendarEventsFromCompany)
<img width="827" height="894" alt="image"
src="https://github.com/user-attachments/assets/443913db-e5fe-4161-b0e7-4a971cc80a71"
/>
- Add FILES field on attachment
- Adapt Attachment logic in front to use new resolver/controller
- Update files-field logic to infer applicationId from fieldMetadataId +
ask for fieldMetadataId in upload resolver
- Design update
To do in next PR :
- Adapt activity files logic
This PR fixes a bug that arises in `formatResult` following up the
recent refactor for DATE_TIME :
https://github.com/twentyhq/twenty/pull/17407
In the case of workflows, we pass a plain object to `formatResult` :
```ts
{
before: null,
after: '2026-01-27T10:59:15.525Z'
}
```
So a case has been added to handle this.
@Weiko are we ok with this more restrictive else-if part in this util ?
We could also just put a `continue` for unknown shapes.
# Introduction
In this PR we're introducing mainly two branded type signatures for both
`JsonbProperty` entities properties and `SerializedRelation` (jsonb
serialized property storing another entity id).
Allowing to dynamically map over them later in order to build universal
`jsonb` `serialized` relations.
## `JsonbProperty`
A branded wrapper type that marks entity properties stored as PostgreSQL
JSONB columns. It adds a phantom brand `__JsonbPropertyBrand__` to
object types while leaving primitives unchanged. The branded key is
optional and typed as never, also omitted when transpiled to
`UniversalFlat`
**Should be used at entities lvl only:**
```typescript
@Column({ type: 'jsonb', nullable: false })
gridPosition: JsonbProperty<GridPosition>;
@Column({ nullable: false, type: 'jsonb', default: [] })
publishedVersions: JsonbProperty<string[]>;
```
## `SerializedRelation`
A branded string type that marks foreign key IDs stored inside JSONB
objects. These are entity references serialized within a JSONB column
rather than being a regular database foreign key.
**Usage in jsonb property generic***
```ts
type FieldMetadataRelationSettings = {
relationType: RelationType;
onDelete?: RelationOnDeleteAction;
joinColumnName?: string | null;
junctionTargetFieldId?: SerializedRelation;
};
```
## `FormatJsonbSerializedRelation<T>`
A transformation type that processes JSONB properties for universal
entity mapping. It:
1. Detects properties with the `JsonbProperty` brand
2. Finds `SerializedRelation` properties
3. Renames them from `*Id` to `*UniversalIdentifier`
4. Removes the brand from the output type ( optional though )
```typescript
// Input: JsonbProperty<{ targetFieldMetadataId: SerializedRelation }>
// Output: { targetFieldMetadataUniversalIdentifier: SerializedRelation }
```
## Result
An example of the dynamic type mapping, through a type-test example
```ts
type SettingsTestCase = UniversalFlatFieldMetadata<
| FieldMetadataType.RELATION
| FieldMetadataType.NUMBER
| FieldMetadataType.TEXT
>['settings']
type SettingsExpectedResult =
| {
relationType: RelationType;
onDelete?: RelationOnDeleteAction | undefined;
joinColumnName?: string | null | undefined;
junctionTargetFieldUniversalIdentifier?: SerializedRelation | undefined;
}
| {
dataType?: NumberDataType | undefined;
decimals?: number | undefined;
type?: FieldNumberVariant | undefined;
}
| {
displayedMaxRows?: number | undefined;
}
| null;
type Assertions = [
Expect<Equal<SettingsTestCase, SettingsExpectedResult>>,
]
```
## Remarks
- Removed duplicated twenty-server and twenty-shared typed
- Removed class validator instances for default value that were not used
at runtime, we will refactor that to add validation across all entities
following a same pattern
This PR modifies our broadly used `formatResult` util to counter act
TypeORM transforming any date time to a `Date` object.
Instead we return the ISO string for any `DATE_TIME`, this way we're not
transporting Date object from one function to another in the backend.
We do this because there was problems working with events utils that
take string date time in parameters and received Date objects.
As this is a recurring problem and because it's an opinionated choice
from TypeORM, we chose to switch to string only in our codebase, from
TypeORM's output to frontend.
In this PR :
- New field type FieldMetadataType.FILES added (as jsonb in DB)
- Backend: GraphQL types, validation, REST API schema, data processor
handling
- Frontend: field configuration in settings
- Feature flag IS_FILES_FIELD_ENABLED to gate the feature
- Integration tests for create/filter validation
- Dev seed: Feature flag enabled + FILES field on survey result object
To do in next PRs :
- Backend :
-- new workspaceFile controller (for download) & new workspaceFile
upload resolver (for upload)
-- pre-hook/post-hook to ensure fileId existence + listener to ensure
cleaning
- Frontend : FILES field display/edit in table view, record, ...
Upgraded to Storybook 10. We still use `@storybook/test-runner` for
testing since it appears it'd require more work to move from Jest to
Vitest than I initially anticipated, but I completed this PR to fix
`storybook:serve:dev` - it takes time to load, but it works the way it
used to with Storybook 8.
https://github.com/user-attachments/assets/7afc32c6-4bcf-4b37-b83b-8d00d28dda15
# Introduction
Followup of
https://github.com/twentyhq/twenty/pull/17001#pullrequestreview-3638508738
close https://github.com/twentyhq/core-team-issues/issues/1910
We've completely decom the `sync-metadata` in production. We're now then
removing its implementation in favor of the v2.
## TODO:
- [x] Remove sync-metadata implem and commands
- [x] Remove workspace decorators
- [x] Type each deprecated field to deprecated on their workspaceEntity
- [x] Remove the `workspace-sync-metadata` folder entirely
- [x] remove workspace migration
- [x] workspace migration removal migration
- [x] remove the `v2` references from workspace manager file names
- [x] remove the `v2` references from workspace manager modules
- [ ] Double check impact on translation file path updates
## Note
- Removed the gate logic
- Remains some service v2 naming, serverless needs to be migrated on v2
fully
- Removed workspaceMigration service app health consumption, making it
always returning up ( no more down ) cc @FelixMalfait ( quite obsolete
health check now, will require complete refactor once we introduce inter
app dependency etc )
@charlesBochet
In workflow codebase, NULL (instead of empty object) is expected on
workflow version object step field, when a new workflow is created for
example.
(packages/twenty-front/src/modules/workflow/workflow-diagram/utils/generateWorkflowDiagram.ts
- 42)
This case is not isolated and it creates many issues.
We decided to format NULL value to equivalent (empty string for text
field, empty object for raw_json) but it seems it complicates the dev x.
To unlock @Devessier I prefer revert the logic, the time we discuss how
to solve this cases.
closes https://github.com/twentyhq/core-team-issues/issues/1629
To do before requesting review :
- filter update
Migration to come in an other PR
Strat :
1/ Null transformation
- [x] Transform NULL equivalent value to NULL in field validation in
common api - pre-query - with feature flag
- [ ] Same logic in ORM (Not done, complex to handle feature flag here)
- [x] Transform NULL value to equivalent in data formatting in ORM -
post-query
2/ Migration (in other PR) for fieldMetadata not nullable with default
defaultValue (empty string, ...)
- [ ] Remove NOT NULL db constraint
- [ ] Update record value to NULL
- [ ] Update field metadata : isNullable:true
- [ ] Update uniqueIndex whereClause (also for standard uniqueIndex)
- [ ] Activate feature flag
3/ Update metadata creation
- [x] No more default default value
- [x] Update standard field nullability
- [x] Remove index default whereClause for standard field
4/ Update filter
- [x] When filtering on NULL or empty string, be sure all records are
returned (the one with NULL + the one with "")
5/ Test
- [ ] Strat. to do
This PR implements the necessary tools to have `react-datepicker`
calendar and our date picker components work reliably no matter the
timezone difference between the user execution environment and the user
application timezone.
Fixes https://github.com/twentyhq/core-team-issues/issues/1781
This PR won't cover everything needed to have Twenty handle timezone
properly, here is the follow-up issue :
https://github.com/twentyhq/core-team-issues/issues/1807
# Features in this PR
This PR brings a lot of features that have to be merged together.
- DATE field type is now handled as string only, because it shouldn't
involve timezone nor the JS Date object at all, since it is a day like a
birthday date, and not an absolute point in time.
- DATE_TIME field wasn't properly handled when the user settings
timezone was different from the system one
- A timezone abbreviation suffix has been added to most DATE_TIME
display component, only when the timezone is different from the system
one in the settings.
- A lot of bugs, small features and improvements have been made here :
https://github.com/twentyhq/core-team-issues/issues/1781
# Handling of timezones
## Essential concepts
This topic is so complex and easy to misunderstand that it is necessary
to define the precise terms and concepts first. It resembles character
encoding and should be treated with the same care.
- Wall-clock time : the time expressed in the timezone of a user, it is
distinct from the absolute point in time it points to, much like a
pointer being a different value than the value that it points to.
- Absolute time : a point in time, regardless of the timezone, it is an
objective point in time, of course it has to be expressed in a given
timezone, because we have to talk about when it is located in time
between humans, but it is in fact distinct from any wall clock time, it
exists in itself without any clock running on earth. However, by
convention the low-level way to store an absolute point in time is in
UTC, which is a timezone, because there is no way to store an absolute
point in time without a referential, much like a point in space cannot
be stored without a referential.
- DST : Daylight Save Time, makes the timezone shift in a specific
period every year in a given timezone, to make better use of longer days
for various reasons, not all timezones have DST. DST can be 1 hour or 30
min, 45 min, which makes computation difficult.
- UTC : It is NOT an “absolute timezone”, it is the wall-clock time at
0° longitude without DST, which is an arbitrary and shared human
convention. UTC is often used as the standard reference wall-clock time
for talking about absolute point in time without having to do timezone
and DST arithmetic. PostgreSQL stores everything in UTC by convention,
but outputs everything in the server’s SESSION TIMEZONE.
## How should an absolute point in time be stored ?
Since an absolute point in time is essentially distinct from its
timezone it could be stored in an absolute way, but in practice it is
impossible to store an absolute point in time without a referential. We
have to say that a rocket launched at X given time, in UTC, EST, CET,
etc. And of course, someone in China will say that it launched at 10:30,
while in San Francisco it will have launched at 19:30, but it is THE
SAME absolute point in time.
Let’s take a related example in computer science with character
encoding. If a text is stored without the associated encoding table, the
correct meaning associated to the bits stored in memory can be lost
forever. It can become impossible for a program to guess what encoding
table should be used for a given text stored as bits, thus the glitches
that appeared a lot back in the early days of internet and document
processing.
The same can happen with date time storing, if we don’t have the
timezone associated with the absolute point in time, the information of
when it absolutely happened is lost.
It is NOT necessary to store an absolute point in time in UTC, it is
more of a standard and practical wall-clock time to be associated with
an absolute point in time. But an absolute point in time MUST be store
with a timezone, with its time referential, otherwise the information of
when it absolutely happened is lost.
For example, it is easier to pass around a date as a string in UTC, like
`2024-01-02T00:00:00Z` because it allows front-end and back-end code to
“talk” in the same standard and DST-free wall-clock time, BUT it is not
necessary. Because we have date libraries that operate on the standard
ISO timezone tables, we can talk in different timezone and let the
libraries handle the conversion internally.
It is false to say that UTC is an absolute timezone or an absolute point
in time, it is just the standard, conventional time referential, because
one can perfectly store every absolute points in time in UTC+10 with a
complex DST table and have the exactly correct absolute points in time,
without any loss of information, without having any UTC+0 dates
involved.
Thus storing an absolute point in time without a timezone associated,
for example with `timestamp` PostgreSQL data type, is equivalent to
storing a wall-clock time and then throwing away voluntarily the
information that allows to know when it absolutely happened, which is a
voluntary data-loss if the code that stores and retrieves those
wall-clock points in time don’t store the associated timezone somewhere.
This is why we use `timestamptz` type in PostgreSQL, so that we make
sure that the correct absolute point in time is stored at the exact time
we send it to PostgreSQL server, no matter the front-end, back-end and
SQL server's timezone differences.
## The JavaScript Date object
The native JavaScript Date object is now officially considered legacy
([source](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date)),
the Date object stores an absolute point in time BUT it forces the
storage to use its execution environment timezone, and one CANNOT modify
this timezone, this is a legacy behavior.
To obtain the desired result and store an absolute point in time with an
arbitrary timezone there are several options :
- The new Temporal API that is the successor of the legacy Date object.
- Moment / Luxon / @date-fns/tz that expose objects that allow to use
any timezone to store an absolute point in time.
## How PostgreSQL stores absolute point in times
PostgreSQL stores absolute points in time internally in UTC
([source](https://www.postgresql.org/docs/current/datatype-datetime.html#DATATYPE-DATETIME-INPUT-TIME-STAMPS)),
but the output date is expressed in the server’s session timezone
([source](https://www.postgresql.org/docs/current/sql-set.html)) which
can be different from UTC.
Example with the object companies in Twenty seed database, on a local
instance, with a new “datetime” custom column :
<img width="374" height="554" alt="image"
src="https://github.com/user-attachments/assets/4394cb43-d97e-4479-801d-ca068f800e39"
/>
<img width="516" height="524" alt="image"
src="https://github.com/user-attachments/assets/b652f36a-d2e2-47a4-8950-647ca688cbbd"
/>
## Why can’t I just use the JavaScript native Date object with some
manual logic ?
Because the JavaScript Date object does not allow to change its internal
timezone, the libraries that are based on it will behave on the
execution environment timezone, thus leading to bugs that appear only on
the computers of users in a timezone but not for other in another
timezone.
In our case the `react-datepicker` library forces to use the `Date`
object, thus forcing the calendar to behave in the execution environment
system timezone, which causes a lot of problems when we decide to
display the Twenty application DATE_TIME values in another timezone than
the user system one, the bugs that appear will be of the off-by-one date
class, for example clicking on 23 will select 24, thus creating an
unreliable feature for some system / application timezone combinations.
A solution could be to manually compute the difference of minutes
between the application user and the system timezones, but that’s not
reliable because of DST which makes this computation unreliable when DST
are applied at different period of the year for the two timezones.
## Why can’t I compute the timezone difference manually ?
Because of DST, the work to compute the timezone difference reliably,
not just for the usual happy path, is equivalent to developing the
internal mechanism of a date timezone library, which is equivalent to
use a library that handles timezones.
## Using `@date-fns/tz` to solve this problem
We could have used `luxon` but it has a heavy bundle size, so instead we
rely here on `@date-fns/tz` (~1kB) which gives us a `TZDate` object that
allows to use any given timezone to store an absolute point-in-time.
The solution here is to trick `react-datepicker` by shifting a Date
object by the difference of timezone between the user application
timezone and the system timezone.
Let’s take a concerte example.
System timezone : Midway, ⇒ UTC-11:00, has no DST.
User application timezone : Auckland, NZ ⇒ UTC+13:00, has a DST.
We’ll take the NZ daylight time, so that will make a timezone difference
of 24 hours !
Let’s take an error-prone date : `2025-01-01T00:00:00` . This date is
usually a good test-case because it can generate three classes of bugs :
off-by-one day bugs, off-by-one month bugs and off-by-one year bugs, at
the same time.
Here is the absolute point in time we take expressed in the different
wall-clock time points we manipulate
Case | In system timezone ⇒ UTC-11 | In UTC | In user application
timezone ⇒ UTC+13
-- | -- | -- | --
Original date | `2024-12-31T00:00:00-11:00` | `2024-12-31T11:00:00Z` |
`2025-01-01T00:00:00+13:00`
Date shifted for react-datepicker | `2025-01-01T00:00:00-11:00` |
`2025-01-01T11:00:00Z` | `2025-01-02T00:00:00+13:00`
We can see with this table that we have the number part of the date that
is the same (`2025-01-01T00:00:00`) but with a different timezone to
“trick” `react-datepicker` and have it display the correct day in its
calendar.
You can find the code in the hooks
`useTurnPointInTimeIntoReactDatePickerShiftedDate` and
`useTurnReactDatePickerShiftedDateBackIntoPointInTime` that contain the
logic that produces the above table internally.
## Miscellaneous
Removed FormDateFieldInput and FormDateTimeFieldInput stories as they do
not behave the same depending of the execution environment and it would
be easier to put them back after having refactored FormDateFieldInput
and FormDateTimeFieldInput
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
To be done in an other PR, move graphql-query-runner handlers in
common-api-query-runner folder (+ renaming + typing improvments)
Fixes :
- totalCount on findMany (Rest)
- getAllSelectedFields did not handle composite field (Rest)
- issue with endCursor (Rest)
- args processing for updateOne and createOne (Rest + Gql)
- fix findDuplicates (Gql)
