Now that all existing and new workspaces have the following syncStage:
- CALENDAR_EVENT_LIST_FETCH_PENDING
- MESSAGE_LIST_FETCH_PENDING
We can fully deprecate the old FULL_CALENDAR_EVENT_LIST_FETCH_PENDING
and PARTIAL_CALENDAR_EVENT_LIST_FETCH_PENDING (full vs partial is now
directly inferred from the presence of a cursor)
## What
In this PR, we are fixing two issues:
- while deleting an object, the views are not properly refreshed leading
to FE bug. This is due to the fact that we were refetching **views
associated to the object** which has been deleted. As views are properly
deleted in the BE, the FE gets no views from this call and cannot
optimistically react. In this case, I'm triggering a complete view
refetch
- messaging error handling had a hole
## Improvements
- Add logs to all gql operations and rest calls to help debug CPU issues
on the backend. These are temporary and should be removed
- Remove nested relations from workflowVersions load (used to add manual
triggers in the side bar). On some workspaces this call result in a
response of 4MB which is heavy on CPU
- Investigated Redis Usage ==> made a few improvements, we are should
still migrate to the new cache service once available
- investigated db calls in messaging / calendar fetch list + workflow
enqueue run cron jobs. Everything seems to be properly batched
- Removed buffer to string conversion on message and we instead pass the
direct Buffer
- Removed redundant flattening of results
- Cached `JSDOM` and `DOMPurify` Instance
- MailParser added `skipTextToHtml` along with some other options, which
reduced parsing time from 50ms to 35ms
---------
Co-authored-by: Félix Malfait <felix.malfait@gmail.com>
Co-authored-by: Félix Malfait <felix@twenty.com>
## Context
While refactoring messaging, I forgot that workspaces that have been
created before july do not have 'CALENDAR_EVENT_LIST_FETCH_PENDING' and
'MESSAGE_LIST_FETCH_PENDING' enum values (as we don't sync-metadata the
enum values, we would need a migrate command that has not been written
yet)
In the current implementation CALENDAR_EVENT_LIST_FETCH_PENDING =
FULL_CALENDAR_EVENT_LIST_FETCH_PENDING, so I'm putting it back as it was
before
## Context
1. We are debugging the experience on messaging and calendar sync. I'm
adding two scripts to relaunch messaging and calendar channels
2. We are seeing errors in Sentry regarding microsoft drivers errors. I
was about to fix them but I'm refactoring / simplifying a bit the logic
first. It will be a betters starting point
In this PR, I'm solving several issues:
1) We were not checking if the currentWorkspaceMember was owning the
message in the thread. It kind of worked before because the case of
shared threads (with shared threads visibility restriction) was not
happening that often. It seems that the bug has always been there
2) Re-implement orphan messages and threads deletion on messageChannel
deletion. We used to brutally look for all orphans, we disabled it last
week because it was too heavy on db. I've re-implemented it more
carefully and "surgically"
3) Gmail sync was not handling folder synced correctly. It was
leveraging labelIds which it shouldn't do (this is a AND AND parameter)
in full sync
4) Added a command to clean orphan message threads manually if needed.
Usually this is done when you remove a messageChannel, or change
blocklist rules but it can be useful to have it to debug
## Context
To simplify the way we inject our default datasource, I've recently
removed the token injection that was confusion since we only had once
configured on the module level. Now I'm removing TypeORM service which
allows us to instantiate a new Datasource with the same parameters as
the default one, it was redundant and confusing.
- Removed `IMAPMessageLocator` service which relied on `Message-ID`
header lookups.
We now fetch messages directly by UID, which should eliminate most of
the issues we were hitting.
This had a challenge, `messageIdsToFetch` were flattened and UIDs are
only unique within their own folder.
To preserve context without modifying the core message module, I landed
on using composite keys for IMAP message IDs.
Each UID is now prefixed with its folder name, and we parse it back when
needed to recover the folder.
This feels like a reasonable trade-off since it resolves a lot of the
failures we saw with `Message-ID` lookups.
And Syncs are faster now since there's no double scanning.
- Added `QRESYNC` support for servers that provide it, giving us faster
incremental sync.
- Refactored code to better split concerns.
### Compatibility Test
| Server | Status |
|---------------------|------------|
| Gmail | ✅ Tested |
| Dovecot | ✅ Tested |
| Stalwart | ✅ Tested |
| Titan Email | ✅ Tested |
| FastMail | ✅ Tested |
<img width="804" height="418" alt="image"
src="https://github.com/user-attachments/assets/b414a484-2798-4839-b395-9dd5c153effe"
/>
- Added `ImapFindSentMailboxService` to replace utility function it
fixes an edge case where special flag `Sent` folder may have zero
messages, we fall back to regex based approach to find other candidates
- Fixed edge case where some servers may not return text content for
body it parses the HTML in that case
- Some other enhancements with capability based detection
Tested with Stalwart server and Titan email
/closes #13884
In this PR:
- refactor timelineActivity computation to batch it
- make sure to pass the authContext to insert-query-builder in order to
pass user information to timelineActivity
- refactor message-participant /calendar-participant creation to batch
more
- fix favorite on view race condition (FE)
- deprecate PARTIAL_CALENDAR_EVENT_FETCH_LIST syncStage as we will
deprecate partial vs full notion (we will just leverage cursor emptyness
or not)
- introduce calendar / messging SCHEDULED syncStage that will allow
better performance granularity later
- activate quick message import after message list fetch to speed
performance on small message lists
- Removed caching of IMAP client as it was problematic and didn't really
matter because of our pipeline, also added retry logic to client.
- Some code refactoring and cleanup
- Breaking changes:
- Removed messageChannel.syncCursor
- Uses messageFolder.syncCursor instead
- Removed Date based sync cursor in favor of message UID which is much
more reliable and precise
---------
Co-authored-by: Charles Bochet <charles@twenty.com>
