Improve account sync performance, handle concurrent market data syncing (#2236)

* PlaidConnectable concern

* Remove bad abstraction

* Put sync implementations in own concerns

* Sync strategies

* Move sync orchestration to Sync class

* Clean up sync class, add state machine

* Basic market data sync cron

* Fix price sync

* Improve sync window column names, add timestamps

* 30 day syncs by default

* Clean up market data methods

* Report high duplicate sync counts to Sentry

* Add sync states throughout app

* account tab session

* Persistent account tab selections

* Remove manual sleep

* Add migration to clear stale syncs on self hosted apps

* Tweak sync states

* Sync completion event broadcasts

* Fix timezones in tests

* Cleanup

* More cleanup

* Plaid item UI broadcasts for sync

* Fix account ID namespace conflict

* Sync broadcasters

* Smoother account sync refreshes

* Remove test sync delay

config/initializers/mini_profiler.rb

@@ -1,3 +1,4 @@
 Rails.application.configure do
-  Rack::MiniProfiler.config.skip_paths = [ "/design-system" ]
+  Rack::MiniProfiler.config.skip_paths = [ "/design-system", "/assets", "/cable", "/manifest", "/favicon.ico", "/hotwire-livereload", "/logo-pwa.png" ]
+  Rack::MiniProfiler.config.max_traces_to_show = 50
 end

config/initializers/sidekiq.rb

@@ -7,3 +7,8 @@ Sidekiq::Web.use(Rack::Auth::Basic) do |username, password|
   ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(username), configured_username) &&
     ActiveSupport::SecurityUtils.secure_compare(::Digest::SHA256.hexdigest(password), configured_password)
 end
+
+Sidekiq::Cron.configure do |config|
+  # 10 min "catch-up" window in case worker process is re-deploying when cron tick occurs
+  config.reschedule_grace_period = 600
+end

config/routes.rb

@@ -1,4 +1,5 @@
 require "sidekiq/web"
+require "sidekiq/cron/web"
 
 Rails.application.routes.draw do
   # MFA routes
@@ -25,6 +26,8 @@ Rails.application.routes.draw do
   get "changelog", to: "pages#changelog"
   get "feedback", to: "pages#feedback"
 
+  resource :current_session, only: %i[update]
+
   resource :registration, only: %i[new create]
   resources :sessions, only: %i[new create destroy]
   resource :password_reset, only: %i[new create edit update]
@@ -104,10 +107,6 @@ Rails.application.routes.draw do
   end
 
   resources :accounts, only: %i[index new], shallow: true do
-    collection do
-      post :sync_all
-    end
-
     member do
       post :sync
       get :chart

config/schedule.yml

@@ -0,0 +1,5 @@
+sync_market_data:
+  cron: "0 17 * * 1-5" # 5:00 PM EST (1 hour after market close)
+  class: "SyncMarketDataJob"
+  queue: "scheduled"
+  description: "Syncs market data daily at 5:00 PM EST (1 hour after market close)"

config/sidekiq.yml

@@ -1,6 +1,7 @@
 concurrency: <%= ENV.fetch("RAILS_MAX_THREADS") { 3 } %>
 queues:
-  - [high_priority, 6]
+  - [scheduled, 10] # For cron-like jobs (e.g. "daily market data sync")
+  - [high_priority, 4]
   - [medium_priority, 2]
   - [low_priority, 1]
   - [default, 1]