# Introduction
Adding view-group to core engine v2
Following https://github.com/twentyhq/twenty/pull/15010 ( same pattern )
## What's done
- Created flat-view-group
- flat view group runner
- flat view group builder
- create view group service v2 and input transpilers
- refactor the existing view group resolver to fix standard (
BREAKING_CHANGE on graphql api update especially ) REST stays the same
- refactored the front to consume the mutations autogenerated
close https://github.com/twentyhq/core-team-issues/issues/1665
# Introduction
Migrating `viewFilter` to v2 in order to migrate later the field update
side effect on view to v2 too
## What's done
- Created flat-view-filter
- flat view filter runner
- flat view filter builder
- create view filter service v2 and input transpilers
- refactor the existing view filter resolver to fix standard (
BREAKING_CHANGE on graphql api update especially ) REST stays the same
- refactored the front to consume the mutations autogenerated
## New generic tools
### Compare two flat entity
Introducing a new util to compare two flat entity, it's strictly typed
and will be added to the generic builder in a following PR
This will ease flat entity addition as won't required to create a
specific abstraction for comparison
Generic builder will expect specific constant: properties to compare and
properties to stringify
### Transform flat entity for comparison
Forked and refactor the initial existing method for flat entity business
scope and type safety
## Coverage
Migrated existing integration tests to fit new contract API
This PR does not add strong coverage on validation exceptions
Deadlines are too short
close https://github.com/twentyhq/core-team-issues/issues/1666
# Introduction
Preparing view-filter and view-group introduction in v2 core engine
Moving view from `core-modules` to `metadata-modules`
## What happened
### Created dedicated modules for each view entity:
- ViewFieldModule
- ViewFilterModule
- ViewFilterGroupModule
- ViewGroupModule
- ViewSortModule
### Each module is now completely independent with its own:
- Controller
- Resolver
- Service
- Entity
### Created dedicated abstraction metadata module folder for:
- flat-view-field
- flat-view
### Dependencies
- Eleminated circular dep on ViewModule to all others ones
- Granular import not importing the whole viewModule anymore everywhere
close https://github.com/twentyhq/core-team-issues/issues/1703
# Introduction
Extracting the legacy fields build and dispatch out of the object one to
follow the generic flat entity build, also update caches entries for
object
## Main tasks
- flat field map cache
- flat field builder
- refactored the dispatch matrix to return flat entity maps instead of
flat entity arrays
- orchestrator aggregator
- removing legacy code
- making universal identifier aka standardId of standard field for
custom object deterministically dynamic
- perfs debug logs for v2
## TODO
- [x] Refactor the generic entity builder to be dependency flat maps in
order to main foreign keys list in flat parent
- [x] Refactor the flat object metadata to contain the array of related
fields and avoid costy find object fields
- [ ] Improve the create field handler to handle multiple field at once
- [ ] Refactor the dispatch to embbed the comparison
- [ ] Improve perf by extracting from elements out of existing
- [ ] Fix the labelIdentifierId validators on object before field
creation ( integ tests are in failing mode )
## Debug logs snippet
```ts
[EntityBuilder fieldMetadata] matrix computation: 0.027ms
[EntityBuilder fieldMetadata] creation validation: 0.001ms
[EntityBuilder fieldMetadata] deletion validation: 0.293ms
[EntityBuilder fieldMetadata] update validation: 0.006ms
[EntityBuilder fieldMetadata] entity processing: 0.363ms
[EntityBuilder fieldMetadata] validateAndBuild: 0.455ms
[EntityBuilder index] matrix computation: 0.005ms
[EntityBuilder index] creation validation: 0.001ms
[EntityBuilder index] deletion validation: 0.146ms
[EntityBuilder index] update validation: 0.004ms
[EntityBuilder index] entity processing: 0.199ms
[EntityBuilder index] validateAndBuild: 0.228ms
[Runner] Initial cache retrieval: 0.549ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForWorkspaceSchema: 11.665ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_index executeForMetadata: 12.864ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 1.476ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 6.816ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForWorkspaceSchema: 0.062ms
[BaseWorkspaceMigrationRunnerActionHandlerService] delete_field executeForMetadata: 0.889ms
[Runner] Transaction execution: 23.434ms
[Runner] Cache invalidation: 316.662ms
[Runner] Total execution: 340.767ms
```
As you can see cache invalidation is way to long, we could replace the
cache by the optimistic in the end
# Introduction
Honestly this implem is a mess, discussing a potential side effect
handler with @weiko before the build and run that would handle each side
effect per entity and operation
Handling:
- [ ] unique index is generated when a field is updated with the
`isUnique`
- [x] an index is generated when a relation is created
- [x] search vector index creation on custom object creation
- [x] renaming a field metadata or an object should re-create all
related indexes which are composed by their namings
- [x] delete object should remove any related indexes
- [x] delete field should update related indexes ( if index ends up
empty it should be removed )
- [ ] on object renaming that contains morph fields -> triggers update
field -> trigger index recompute
- [x] on update name renaming should recompute all related indexes
## TODO
- [x] Integration testing
- [ ] Refactor the index maps cache to be storing a
`idsByObjectMetadataId`
- [x] Refactor deterministic name to use order sorting
- [x] Remove flat index from flat object
## What's next
Will handle morph indexes in a new dedicated PR for the moment will
stick to this
Same for the cache improvement and uniqueness
## Context
Adding a new service that provides an abstract caching system for
FlatEntityMaps.
This takes care of cache invalidation and storing local and remote cache
for the map with retrieval after a comparison with map hash between
local and remote (redis).
## Implementation
Remote (redis) keys
- Flat map data:
`engine:workspace:flat-maps:{flatMapKey}:{workspaceId}:flat-map`
- Content hash:
`engine:workspace:flat-maps:{flatMapKey}:{workspaceId}:hash`
**Local Cache Hit**: If local hash matches Redis hash, return local data
**Remote Cache Hit**: If Redis has data with different hash, update
local cache
**Remote Cache Miss**: Recompute from database, store in Remote and
locally
**Invalidation**: Remove from Remote, triggering recomputation on next
access
## Usage
```typescript
@WorkspaceFlatMapCache('view') // redis key
export class WorkspaceFlatViewMapCacheService extends WorkspaceFlatMapCacheService<FlatViewMaps> {
constructor(
@InjectCacheStorage(CacheStorageNamespace.EngineWorkspace)
cacheStorageService: CacheStorageService,
@InjectRepository(ViewEntity)
private readonly viewRepository: Repository<ViewEntity>,
) {
super(cacheStorageService);
}
// only method to implement
public async computeFlatMap(workspaceId: string): Promise<FlatViewMaps> {
const views = await this.viewRepository.find({
where: { workspaceId },
relations: ['viewFields'],
select: { viewFields: { id: true } },
});
return generateFlatViewMaps(views);
}
}
```
```typescript
// 2 public methods, getExistingOrRecomputeFlatMaps to fetch the map and invalidateCache after a mutation
await this.workspaceFlatViewMapCacheService.invalidateCache(
viewData.workspaceId,
);
const flatViewMaps =
await this.workspaceFlatViewMapCacheService.getExistingOrRecomputeFlatMaps(
workspaceId,
);
```
## Multi-Pod Synchronization
- Each pod maintains local cache for performance
- SHA256 hash of flatMap content used for version comparison
- `invalidateCache()` reset Redis data, forcing other pods to refresh
when calling getExistingOrRecomputeFlatMaps
---
<img width="1072" height="325" alt="Screenshot 2025-09-11 at 15 04 48"
src="https://github.com/user-attachments/assets/ed6ce82c-db35-4a1b-8a4e-247b694e2ddf"
/>
<img width="1030" height="328" alt="Screenshot 2025-09-11 at 15 04 40"
src="https://github.com/user-attachments/assets/43a15789-7f27-4104-a9bb-bef19908a5cd"
/>
Next step: Implement a locking mechanism by reusing existing WithLock
decorator
