# Introduction
In preparation of the workspace agnostic builder, we're migrating
`FlatEntityMaps` to be universal identifier oriented and based
As in the builder context there're won't be any ids at all
Please also note that the FlatEntity is a UniversalFlatEntity superset
From
```ts
import { type SyncableFlatEntity } from 'src/engine/metadata-modules/flat-entity/types/flat-entity-from.type';
export type FlatEntityMaps<T extends SyncableFlatEntity> = {
byId: Partial<Record<string, T>>;
idByUniversalIdentifier: Partial<Record<string, string>>;
universalIdentifiersByApplicationId: Partial<Record<string, string[]>>;
};
```
To
```ts
export type FlatEntityMaps<
T extends SyncableFlatEntity | UniversalSyncableFlatEntity,
> = {
byUniversalIdentifier: Partial<Record<string, T>>;
universalIdentifierById: Partial<Record<string, string>>;
universalIdentifiersByApplicationId: Partial<Record<string, string[]>>; // this might make more sense to be migrated to universalIdentifiersByApplicationUniversalIdentifier but it's the main topic of this PR
};
```
## Low level maps tools
Had to refactor find | create | delete | replace | find-many | get-sub
tools ( through mutations and or throw equivalent )
## Context
This PR adds the core structure for RLS implementation:
- RLS data model
- RLS service layer
- RLS WorkspaceMigration and Syncable Entity + cache + Validations
- RLS resolver layer
- ORM layer with RLS Predicate to ORM WHERE clause conversion with
workspaceMember record transposition
Tests are missing though
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Establishes core row-level permissions infrastructure and enforcement
across the stack.
>
> - Backend: new `rowLevelPermissionPredicate` and
`rowLevelPermissionPredicateGroup` entities, TypeORM migration, feature
flag `IS_ROW_LEVEL_PERMISSION_PREDICATES_ENABLED`, flat-entity
maps/cache wiring, services and GraphQL resolvers for CRUD, and
inclusion of `workspaceMember` in auth context
> - ORM: applies row-level permission predicates to SELECT, DELETE, and
SOFT DELETE query builders; propagates context through
GlobalWorkspaceOrmManager/EntityManager
> - GraphQL: generated schema/types/queries/mutations for
creating/updating/deleting/fetching predicates and groups
> - Frontend: settings page adds a gated "Record-level" section
(placeholder) and metadata error handler labels for new entities
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
fe955cc458. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
## Context
This PR refactors how authentication context is handled in the GraphQL
resolver layer. Previously, the auth context was baked into the schema
builder context at schema creation time. Now, the auth context is
extracted from the request at query execution time, enabling better
schema caching.
## Implementation
- Removed user-specific data from schema cache key: The schema cache key
no longer includes userId and apiKeyId, allowing the same schema to be
shared across all users in a workspace
- Cache key simplified from
${workspaceId}-${userId}-${apiKeyId}-${url}-${cacheVersion} to
${workspaceId}-${url}-${cacheVersion}
- Removed authContext from WorkspaceSchemaBuilderContext: The schema
builder context is now purely about object/field metadata, not about who
is accessing it
- Created createQueryRunnerContext utility: A new helper that combines
the schema builder context with the auth context from the actual request
- Updated all resolver factories: All 15 resolver factories now use
createQueryRunnerContext to build the full context needed by query
runners at execution time
## Benefits
- Improved cache efficiency: GraphQL schemas are now cached per
workspace rather than per user, significantly reducing memory usage and
schema regeneration
- Cleaner separation of concerns: Schema building (metadata-driven) is
now separate from query execution (auth-driven)
- Fresh auth context: Auth context is always retrieved from the current
request, ensuring it reflects the latest state
- Reduced complexity: Simplified the schema creation path by removing
unnecessary auth checks
## Next steps
- Introduce a hash in redis and expose it in the req object so the yoga
patch can access the hash and use it during its own cache key
computation. This way we will be able to invalidate the local cache in
yoga graphql schema generation without restarting pods.
This PR improves the general UX and DX of boards, by modifying the query
effect to only use paged group by queries.
In this PR we implement two more things in the backend for group by
queries :
- Fixed ORDER BY in the PARTITION BY sub-query (this wasn't working
because it was applied in the main query, so it sorted randomly picked
records, which was a correct sort on an incorrect dataset returned by
the sub-query)
- Added offset paging in PARTITION BY
Miscellaneous, various bug fixes and improvements along the way :
- Throttled loading of cards to avoid React freeze
- Handling of drag & drop
- Handling of create / delete / update
- Reworked skeleton (the library slows down a lot with hundreds of
skeleton for a spinning effect that is hardly noticed)
- Fixed refetch of aggregate queries (I included the new group by
aggregates query we use in the existing refetch mechanism)
- Re-trigger queries on filters and sorts changes
- Unselect all record ids when deleting / restoring / detroying
- Fetch only groups that still have records to lighten the group by
query.
# What remains to be done
This is still a naïve fetch more implementation that will work for a few
fetch more rounds, but if you scroll and load say 200 cards per column
on a board, React will re-render all 200 cards of each column each time.
We would probably need to virtualize the board with paged queries as we
did for the table, this could be done after this PR but seems less
urgent.
What's nice is that this new query pattern is well designed for
virtualization also, drawing from our experience with table
virtualization, and adapted to a multi-column request pattern, like a
2:2 matrix of records, for our boards.
So the remaining work would be to design a UI solution for virtualizing
this matrix of records, which could be quite different from our table
virtualization mechanism.
## Context
Deprecating the old objectMetadataMap type in favour of split flat
entities to match with our new caching.
In the long run, trying to achieve:
- Better performance through caching
- Consistent data access patterns across the codebase
- Reduced database queries
Now that everything is based on flat entities, which are cached, we can
finish the refactoring of workspace context cache which should already
improve performances.
Then the last step will be to consume that new cache in the new global
datasource to get rid of the many workspace datasources stored in the
server
Closes https://github.com/twentyhq/core-team-issues/issues/1891
Create empty buckets according to the date granularity
Video QA:
https://github.com/user-attachments/assets/86c0f817-35b3-4bab-b093-d11491684b82
Note: We would also need to create empty buckets for cyclic
granularities (DAY_OF_THE_WEEK, MONTH_OF_THE_YEAR, QUARTER_OF_THE_YEAR).
TODO:
- Always order the cyclic granularities Monday -> Sunday (take
firstDayOfTheWeek into account), January -> December, Q1 -> Q4. For now
they are returned by the backend in alphabetical order, which doesn't
make much sense
- Remove the translation into the user's locale of these granularities
from the backend because otherwise we can't reconstruct the missing days
or month in the frontend since they will be translated
---------
Co-authored-by: Lucas Bordeau <bordeau.lucas@gmail.com>
To be done in an other PR, move graphql-query-runner handlers in
common-api-query-runner folder (+ renaming + typing improvments)
Fixes :
- totalCount on findMany (Rest)
- getAllSelectedFields did not handle composite field (Rest)
- issue with endCursor (Rest)
- args processing for updateOne and createOne (Rest + Gql)
- fix findDuplicates (Gql)
Done ⬇️
Gql : move delete and destroy logic to common
Rest :
- rename 'delete' handler to 'destroy'
- create soft delete handlers (named 'delete') : one and many
- create destroy many handler
- update doc
--> Rest api gains NEW soft delete one/many + destroy many capabilities
closes : https://github.com/twentyhq/core-team-issues/issues/1579
Closes https://github.com/twentyhq/core-team-issues/issues/1563.
We now have two inter-group orderBy criteria: one on the aggregate
values, the other on the dimension values.
Ex: I am grouping companies by city and querying the average number of
employees for each group. I can either order the groups by the city
name, or by the average number of employees, or by one then the other.
I could actually also order groups by an aggregated value that I did not
ask for (ex: average ARR), but I cannot order groups by a field value I
did not group records by (ex: country).
[See discord
discussion](https://discord.com/channels/1130383047699738754/1425438213555753050/1425438223097921659)
An example of query variables:
```
{
"groupBy": [
{
"createdAt": {
"granularity": "QUARTER_OF_THE_YEAR"
}
},
{"city": true}
],
"orderBy": [
{
"aggregate": {
"avgEmployees": "DescNullsLast"
}
},
{
"aggregate": {
"percentyEmptyEmployees": "DescNullsLast"
}
},
{
"city": "AscNullsLast"
},
{
"createdAt": {
"orderBy": "AscNullsLast",
"granularity": "QUARTER_OF_THE_YEAR",
}
}
]
}
```
The aggregate orderBy criteria had already been implemented, but I
updated the implementation to add an "aggregate" key to prefix them, in
order to avoid confusion between the two + for the schema generation not
to break (otherwise we would have an issue when generating
OrderByWithGroupByInput if a user has created a field that has the same
name as an aggregate field, such as avgEmployees).
This big PR implements table virtualization with an offset paging,
allowing a way more fluid UX.
It is a v1 that should be improved in the future with partial data
loading and optimization of the browser display performance of a row.
But with this PR we have the solid enough technical foundation, both
frontend and backend, to get to a smooth table UX.
Fixes and improvements after first successful round of development
(needed to have main clean) :
- [x] Delete should refresh virtualized portion only and reset all table
- [x] Fix add new : top and bottom
- [x] Table empty shouldn’t show when first loading
- [x] Fix d&d
- [x] Fix sorts
- [x] Fix drag when scrolling after a full virtual page (it throws an
error)
- [x] Si update mais qu’on a un sort ou filter, alors il faut trigger le
refresh
- [x] Reset scroll position between tables
- [x] Reset scroll shadows between tables
- [x] Setup d&n for virtual list :
https://github.com/hello-pangea/dnd/blob/main/docs/patterns/virtual-lists.md
- [x] Full table re-render when entering edit mode
- [x] Clean code and prepare for merge
Fixes https://github.com/twentyhq/core-team-issues/issues/1613 that
contains other bugs to be fixed before merge
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
closes https://github.com/twentyhq/core-team-issues/issues/1418
Tested :
- findOne on Rest and Gql
### Vision
#### Common
- Common is kind of renamed Gql base resolver
- Common handles args (filter, values, ...) validation
- Common accepts depth or raw gql selected fields to compute
selectedFields
- Common is directly called by each CommonQueries (findOne, ...)
service, which extend CommonBaseQuery service
- Common sequence :
| - Parse & Validate args (args-handlers, to create)
| - Build query (query-parsers : currently in gql-query-parsers, to
move)
| - Execute query
| - Fetch relation + format
#### Rest
- Simple parsing (without metadata validation)
- Calling Common API
- Simple rest response formatting
#### Gql
- Calling Common API
Tested :
- workspace needs to flush cache after activating feature flag
- workspace does not need to flush cache if ff false and coming from
previous version (before gql schema gen updates)
## Problem
After migrating webhooks and API keys from workspace to core level, REST
API endpoints were still creating entities in workspace schema
(`workspace_*`) instead of core schema, causing webhooks to not fire.
## Solution
- Added dedicated REST controllers for webhooks (`/rest/webhooks`) and
API keys (`/rest/apiKeys`)
- Updated dynamic controller to block workspace-gated entities from
being processed
- Fixed OpenAPI documentation to exclude these endpoints from playground
- Ensured return formats match GraphQL resolvers exactly
## Testing
✅ All endpoints tested with provided auth token - webhooks and API keys
now correctly stored in `core` schema
# Introduction
Following https://github.com/twentyhq/twenty/pull/13310
> After this PR merge will create a new one removing the type and
replacing it to ObjectMetadataEntity.
This is it !
Fixes https://github.com/twentyhq/twenty/issues/12867
Issue:
when you have a variable `toto` which is: `Record<string, MyType>` and
you do toto['xxx'], this will be typed as `MyType` instead of `MyType |
undefined`
Solutions:
- activate `noUncheckedIndexedAccess` check in tsconfig, this is the
preferred solution but will take time to get there (this raises 600+
errors)
- use a Map: cf https://github.com/twentyhq/twenty/pull/13125/files
- set the type to Partial<Record<string, MyType>>. Drawback is that when
you do Object.values(toto), you'll get `Array<MyType | undefined>`.
Hence why we have to filter these behind
<img width="1512" alt="image"
src="https://github.com/user-attachments/assets/d0a0bfed-c441-4e53-84c2-2da98ccbcf50"
/>
In this PR:
## Improve recompute metadata cache performance. We are aiming for
~100ms
Deleting relationMetadata table and FKs pointing on it
Fetching indexMetadata and indexFieldMetadata in a separate query as
typeorm is suboptimizing
## Remove caching lock
As recomputing the metadata cache is lighter, we try to stop preventing
multiple concurrent computations. This also simplifies interfaces
## Introduce self recovery mecanisms to recompute cache automatically if
corrupted
Aka getFreshObjectMetadataMaps
## custom object resolver performance improvement: 1sec to 200ms
Double check queries and indexes used while creating a custom object
Remove the queries to db to use the cached objectMetadataMap
## reduce objectMetadataMaps to 500kb
<img width="222" alt="image"
src="https://github.com/user-attachments/assets/2370dc80-49b6-4b63-8d5e-30c5ebdaa062"
/>
We used to stored 3 fieldMetadataMaps (byId, byName, byJoinColumnName).
While this is great for devXP, this is not great for performances.
Using the same mecanisme as for objectMetadataMap: we only keep byIdMap
and introduce two otherMaps to idByName, idByJoinColumnName to make the
bridge
## Add dataloader on IndexMetadata (aka indexMetadataList in the API)
## Improve field resolver performances too
## Deprecate ClientConfig
# Introduction
Added a no-explicit-any rule to the twenty-server, not applicable to
tests and integration tests folder
Related to https://github.com/twentyhq/core-team-issues/issues/975
Discussed with Charles
## In case of conflicts
Until this is approved I won't rebased and handle conflict, just need to
drop two latest commits and re run the scripts etc
## Legacy
We decided not to handle the existing lint error occurrences and
programmatically ignored them through a disable next line rule comment
## Open question
We might wanna activate the
[no-explicit-any](https://typescript-eslint.io/rules/no-explicit-any/)
`ignoreRestArgs` for our use case ?
```
ignoreRestArgs?: boolean;
```
---------
Co-authored-by: etiennejouan <jouan.etienne@gmail.com>
related to https://github.com/twentyhq/core-team-issues/issues/601
## Done
- add a `onDbEvent` `Subscription` graphql endpoint to listen to
database_event using what we have done with webhooks:
- you can subscribe to any `action` (created, updated, ...) for any
`objectNameSingular` or a specific `recordId`. Parameters are nullable
and treated as wildcards when null.
- returns events with following shape
```typescript
@Field(() => String)
eventId: string;
@Field()
emittedAt: string;
@Field(() => DatabaseEventAction)
action: DatabaseEventAction;
@Field(() => String)
objectNameSingular: string;
@Field(() => GraphQLJSON)
record: ObjectRecord;
@Field(() => [String], { nullable: true })
updatedFields?: string[];
```
- front provide a componentEffect `<ListenRecordUpdatesEffect />` that
listen for an `objectNameSingular`, a `recordId` and a list of
`listenedFields`. It subscribes to record updates and updates its apollo
cached value for specified `listenedFields`
- subscription is protected with credentials
## Result
Here is an application with `workflowRun`
https://github.com/user-attachments/assets/c964d857-3b54-495f-bf14-587ba26c5a8c
---------
Co-authored-by: prastoin <paul@twenty.com>
# Introduction
In this PR we've migrated `twenty-shared` from a `vite` app
[libary-mode](https://vite.dev/guide/build#library-mode) to a
[preconstruct](https://preconstruct.tools/) "atomic" application ( in
the future would like to introduce preconstruct to handle of all our
atomic dependencies such as `twenty-emails` `twenty-ui` etc it will be
integrated at the monorepo's root directly, would be to invasive in the
first, starting incremental via `twenty-shared`)
For more information regarding the motivations please refer to nor:
- https://github.com/twentyhq/core-team-issues/issues/587
-
https://github.com/twentyhq/core-team-issues/issues/281#issuecomment-2630949682
close https://github.com/twentyhq/core-team-issues/issues/589
close https://github.com/twentyhq/core-team-issues/issues/590
## How to test
In order to ease the review this PR will ship all the codegen at the
very end, the actual meaning full diff is `+2,411 −114`
In order to migrate existing dependent packages to `twenty-shared` multi
barrel new arch you need to run in local:
```sh
yarn tsx packages/twenty-shared/scripts/migrateFromSingleToMultiBarrelImport.ts && \
npx nx run-many -t lint --fix -p twenty-front twenty-ui twenty-server twenty-emails twenty-shared twenty-zapier
```
Note that `migrateFromSingleToMultiBarrelImport` is idempotent, it's atm
included in the PR but should not be merged. ( such as codegen will be
added before merging this script will be removed )
## Misc
- related opened issue preconstruct
https://github.com/preconstruct/preconstruct/issues/617
## Closed related PR
- https://github.com/twentyhq/twenty/pull/11028
- https://github.com/twentyhq/twenty/pull/10993
- https://github.com/twentyhq/twenty/pull/10960
## Upcoming enhancement: ( in others dedicated PRs )
- 1/ refactor generate barrel to export atomic module instead of `*`
- 2/ generate barrel own package with several files and tests
- 3/ Migration twenty-ui the same way
- 4/ Use `preconstruct` at monorepo global level
## Conclusion
As always any suggestions are welcomed !
Closes https://github.com/twentyhq/core-team-issues/issues/393
- enforcing object-records permission checks in resolvers for now. we
will move the logic to a lower level asap
- add integration tests that will still be useful when we have moved the
logic
- introduce guest seeded role to test limited permissions on
object-records
## Context
All objects have '...duplicates' resolver but only companies and people
have duplicate criteria (hard coded constant).
Gql schema and resolver should be created only if duplicate criteria
exist.
## Solution
- Add a new @WorkspaceDuplicateCriteria decorator at object level,
defining duplicate criteria for given object.
- Add a new duplicate criteria field in ObjectMetadata table
- Update schema and resolver building logic
- Update front requests for duplicate check (only for object with
criteria defined)
closes https://github.com/twentyhq/twenty/issues/9828
Fixes https://github.com/twentyhq/twenty/issues/8810
Fixes https://github.com/twentyhq/twenty/issues/5268
Fixes https://github.com/twentyhq/twenty/issues/8971
- Fixing Task/Note creation not sending position during creation
- Adding a command to backfill position being null, using existing
backfill command.
- Removed unused backfill job.
- Updated workspace entities to set position non-nullable and set a
default value to make it non-required on the API
- Updated position factory to set a default position for all objects
having a POSITION field instead of only company/people
- Moved the try/catch in each resolver factory calling
GraphqlQueryRunnerException handler, makes more sense to call it in the
actual graphql-query-runner and removing some duplicate codes
- Adding validations for input in QueryRunnerArgs factories
- Allow sync-metadata to override and sync defaultValues for certain
field types (that can't be updated by users)
- Removing health-check from sync-metadata command during force mode to
improve performances
Fixes https://github.com/twentyhq/twenty/issues/8300
## Context
API events were created too late and were already formatted as Gql
responses (including nesting with edges/node/type + formatting that
should not exist in an event payload). This PR moves the emit logic to
the resolver where we actually do the DB query
Note: Also added RESTORED events
First step of #https://github.com/twentyhq/twenty/issues/3298.
Here we update the search endpoint to allow for a filter argument, which
we currently use in the relation pickers to restrict or exclude ids from
search.
In a future PR we will try to simplify the search logic in the FE
