Transaction rules engine V1 (#1900)

* Domain model sketch

* Scaffold out rules domain

* Migrations

* Remove existing data enrichment for clean slate

* Sketch out business logic and basic tests

* Simplify rule scope building and action executions

* Get generator working again

* Basic implementation + tests

* Remove manual merchant management (rules will replace)

* Revert "Remove manual merchant management (rules will replace)"

This reverts commit 83dcbd9ff0.

* Family and Provider merchants model

* Fix brakeman warnings

* Fix notification loader

* Update notification position

* Add Rule action and condition registries

* Rule form with compound conditions and tests

* Split out notification types, add CTA type

* Rules form builder and Stimulus controller

* Clean up rule registry domain

* Clean up rules stimulus controller

* CTA message for rule when user changes transaction category

* Fix tests

* Lint updates

* Centralize notifications in Notifiable concern

* Implement category rule prompts with auto backoff and option to disable

* Fix layout bug caused by merge conflict

* Initialize rule with correct action for category CTA

* Add rule deletions, get rules working

* Complete dynamic rule form, split Stimulus controllers by resource

* Fix failing tests

* Change test password to avoid chromium conflicts

* Update integration tests

* Centralize all test password references

* Add re-apply rule action

* Rule confirm modal

* Run migrations

* Trigger rule notification after inline category updates

* Clean up rule styles

* Basic attribute locking for rules

* Apply attribute locks on user edits

* Log data enrichments, only apply rules to unlocked attributes

* Fix merge errors

* Additional merge conflict fixes

* Form UI improvements, ignore attribute locks on manual rule application

* Batch AI auto-categorization of transactions

* Auto merchant detection, ai enrichment in batches

* Fix Plaid merchant assignments

* Plaid category matching

* Cleanup 1

* Test cleanup

* Remove stale route

* Fix desktop chat UI issues

* Fix mobile nav styling issues

app/models/rule/action.rb

@@ -0,0 +1,17 @@
+class Rule::Action < ApplicationRecord
+  belongs_to :rule
+
+  validates :action_type, presence: true
+
+  def apply(resource_scope, ignore_attribute_locks: false)
+    executor.execute(resource_scope, value: value, ignore_attribute_locks: ignore_attribute_locks)
+  end
+
+  def options
+    executor.options
+  end
+
+  def executor
+    rule.registry.get_executor!(action_type)
+  end
+end

app/models/rule/action_executor.rb

@@ -0,0 +1,43 @@
+class Rule::ActionExecutor
+  TYPES = [ "select", "function" ]
+
+  def initialize(rule)
+    @rule = rule
+  end
+
+  def key
+    self.class.name.demodulize.underscore
+  end
+
+  def label
+    key.humanize
+  end
+
+  def type
+    "function"
+  end
+
+  def options
+    nil
+  end
+
+  def execute(scope, value: nil, ignore_attribute_locks: false)
+    raise NotImplementedError, "Action executor #{self.class.name} must implement #execute"
+  end
+
+  def as_json
+    {
+      type: type,
+      key: key,
+      label: label,
+      options: options
+    }
+  end
+
+  private
+    attr_reader :rule
+
+    def family
+      rule.family
+    end
+end

app/models/rule/action_executor/auto_categorize.rb

@@ -0,0 +1,23 @@
+class Rule::ActionExecutor::AutoCategorize < Rule::ActionExecutor
+  def label
+    if rule.family.self_hoster?
+      "Auto-categorize transactions with AI ($$)"
+    else
+      "Auto-categorize transactions"
+    end
+  end
+
+  def execute(transaction_scope, value: nil, ignore_attribute_locks: false)
+    enrichable_transactions = transaction_scope.enrichable(:category_id)
+
+    if enrichable_transactions.empty?
+      Rails.logger.info("No transactions to auto-categorize for #{rule.title} #{rule.id}")
+      return
+    end
+
+    enrichable_transactions.in_batches(of: 20).each_with_index do |transactions, idx|
+      Rails.logger.info("Scheduling auto-categorization for batch #{idx + 1} of #{enrichable_transactions.count}")
+      rule.family.auto_categorize_transactions_later(transactions)
+    end
+  end
+end

app/models/rule/action_executor/auto_detect_merchants.rb

@@ -0,0 +1,23 @@
+class Rule::ActionExecutor::AutoDetectMerchants < Rule::ActionExecutor
+  def label
+    if rule.family.self_hoster?
+      "Auto-detect merchants with AI ($$)"
+    else
+      "Auto-detect merchants"
+    end
+  end
+
+  def execute(transaction_scope, value: nil, ignore_attribute_locks: false)
+    enrichable_transactions = transaction_scope.enrichable(:merchant_id)
+
+    if enrichable_transactions.empty?
+      Rails.logger.info("No transactions to auto-detect merchants for #{rule.title} #{rule.id}")
+      return
+    end
+
+    enrichable_transactions.in_batches(of: 20).each_with_index do |transactions, idx|
+      Rails.logger.info("Scheduling auto-merchant-enrichment for batch #{idx + 1} of #{enrichable_transactions.count}")
+      rule.family.auto_detect_transaction_merchants_later(transactions)
+    end
+  end
+end

app/models/rule/action_executor/set_transaction_category.rb

@@ -0,0 +1,31 @@
+class Rule::ActionExecutor::SetTransactionCategory < Rule::ActionExecutor
+  def type
+    "select"
+  end
+
+  def options
+    family.categories.pluck(:name, :id)
+  end
+
+  def execute(transaction_scope, value: nil, ignore_attribute_locks: false)
+    category = family.categories.find_by_id(value)
+
+    scope = transaction_scope
+
+    unless ignore_attribute_locks
+      scope = scope.enrichable(:category_id)
+    end
+
+    scope.each do |txn|
+      Rule.transaction do
+        txn.log_enrichment!(
+          attribute_name: "category_id",
+          attribute_value: category.id,
+          source: "rule"
+        )
+
+        txn.update!(category: category)
+      end
+    end
+  end
+end

app/models/rule/action_executor/set_transaction_tags.rb

@@ -0,0 +1,31 @@
+class Rule::ActionExecutor::SetTransactionTags < Rule::ActionExecutor
+  def type
+    "select"
+  end
+
+  def options
+    family.tags.pluck(:name, :id)
+  end
+
+  def execute(transaction_scope, value: nil, ignore_attribute_locks: false)
+    tag = family.tags.find_by_id(value)
+
+    scope = transaction_scope
+
+    unless ignore_attribute_locks
+      scope = scope.enrichable(:tag_ids)
+    end
+
+    rows = scope.each do |txn|
+      Rule.transaction do
+        txn.log_enrichment!(
+          attribute_name: "tag_ids",
+          attribute_value: [ tag.id ],
+          source: "rule"
+        )
+
+        txn.update!(tag_ids: [ tag.id ])
+      end
+    end
+  end
+end

app/models/rule/condition.rb

@@ -0,0 +1,62 @@
+class Rule::Condition < ApplicationRecord
+  belongs_to :rule, optional: -> { where.not(parent_id: nil) }
+  belongs_to :parent, class_name: "Rule::Condition", optional: true, inverse_of: :sub_conditions
+
+  has_many :sub_conditions, class_name: "Rule::Condition", foreign_key: :parent_id, dependent: :destroy, inverse_of: :parent
+
+  validates :condition_type, presence: true
+  validates :operator, presence: true
+  validates :value, presence: true, unless: -> { compound? }
+
+  accepts_nested_attributes_for :sub_conditions, allow_destroy: true
+
+  # We don't store rule_id on sub_conditions, so "walk up" to the parent rule
+  def rule
+    parent&.rule || super
+  end
+
+  def compound?
+    condition_type == "compound"
+  end
+
+  def apply(scope)
+    if compound?
+      build_compound_scope(scope)
+    else
+      filter.apply(scope, operator, value)
+    end
+  end
+
+  def prepare(scope)
+    if compound?
+      sub_conditions.reduce(scope) { |s, sub| sub.prepare(s) }
+    else
+      filter.prepare(scope)
+    end
+  end
+
+  def options
+    filter.options
+  end
+
+  def operators
+    filter.operators
+  end
+
+  def filter
+    rule.registry.get_filter!(condition_type)
+  end
+
+  private
+    def build_compound_scope(scope)
+      if operator == "or"
+        combined_scope = sub_conditions
+          .map { |sub| sub.apply(scope) }
+          .reduce { |acc, s| acc.or(s) }
+
+        combined_scope || scope
+      else
+        sub_conditions.reduce(scope) { |s, sub| sub.apply(s) }
+      end
+    end
+end

app/models/rule/condition_filter.rb

@@ -0,0 +1,87 @@
+class Rule::ConditionFilter
+  UnsupportedOperatorError = Class.new(StandardError)
+
+  TYPES = [ "text", "number", "select" ]
+
+  OPERATORS_MAP = {
+    "text" => [ [ "Contains", "like" ], [ "Equal to", "=" ] ],
+    "number" => [ [ "Greater than", ">" ], [ "Greater or equal to", ">=" ], [ "Less than", "<" ], [ "Less than or equal to", "<=" ], [ "Is equal to", "=" ] ],
+    "select" => [ [ "Equal to", "=" ] ]
+  }
+
+  def initialize(rule)
+    @rule = rule
+  end
+
+  def type
+    "text"
+  end
+
+  def number_step
+    family_currency = Money::Currency.new(family.currency)
+    family_currency.step
+  end
+
+  def key
+    self.class.name.demodulize.underscore
+  end
+
+  def label
+    key.humanize
+  end
+
+  def options
+    nil
+  end
+
+  def operators
+    OPERATORS_MAP.dig(type)
+  end
+
+  # Matchers can prepare the scope with joins by implementing this method
+  def prepare(scope)
+    scope
+  end
+
+  # Applies the condition to the scope
+  def apply(scope, operator, value)
+    raise NotImplementedError, "Condition #{self.class.name} must implement #apply"
+  end
+
+  def as_json
+    {
+      type: type,
+      key: key,
+      label: label,
+      operators: operators,
+      options: options,
+      number_step: number_step
+    }
+  end
+
+  private
+    attr_reader :rule
+
+    def family
+      rule.family
+    end
+
+    def build_sanitized_where_condition(field, operator, value)
+      sanitized_value = operator == "like" ? "%#{ActiveRecord::Base.sanitize_sql_like(value)}%" : value
+
+      ActiveRecord::Base.sanitize_sql_for_conditions([
+        "#{field} #{sanitize_operator(operator)} ?",
+        sanitized_value
+      ])
+    end
+
+    def sanitize_operator(operator)
+      raise UnsupportedOperatorError, "Unsupported operator: #{operator} for type: #{type}" unless operators.map(&:last).include?(operator)
+
+      if operator == "like"
+        "ILIKE"
+      else
+        operator
+      end
+    end
+end

app/models/rule/condition_filter/transaction_amount.rb

@@ -0,0 +1,14 @@
+class Rule::ConditionFilter::TransactionAmount < Rule::ConditionFilter
+  def type
+    "number"
+  end
+
+  def prepare(scope)
+    scope.with_entry
+  end
+
+  def apply(scope, operator, value)
+    expression = build_sanitized_where_condition("ABS(entries.amount)", operator, value.to_d)
+    scope.where(expression)
+  end
+end

app/models/rule/condition_filter/transaction_merchant.rb

@@ -0,0 +1,18 @@
+class Rule::ConditionFilter::TransactionMerchant < Rule::ConditionFilter
+  def type
+    "select"
+  end
+
+  def options
+    family.assigned_merchants.pluck(:name, :id)
+  end
+
+  def prepare(scope)
+    scope.left_joins(:merchant)
+  end
+
+  def apply(scope, operator, value)
+    expression = build_sanitized_where_condition("merchants.id", operator, value)
+    scope.where(expression)
+  end
+end

app/models/rule/condition_filter/transaction_name.rb

@@ -0,0 +1,10 @@
+class Rule::ConditionFilter::TransactionName < Rule::ConditionFilter
+  def prepare(scope)
+    scope.with_entry
+  end
+
+  def apply(scope, operator, value)
+    expression = build_sanitized_where_condition("entries.name", operator, value)
+    scope.where(expression)
+  end
+end

app/models/rule/registry.rb

@@ -0,0 +1,46 @@
+class Rule::Registry
+  UnsupportedActionError = Class.new(StandardError)
+  UnsupportedConditionError = Class.new(StandardError)
+
+  def initialize(rule)
+    @rule = rule
+  end
+
+  def resource_scope
+    raise NotImplementedError, "#{self.class.name} must implement #resource_scope"
+  end
+
+  def condition_filters
+    []
+  end
+
+  def action_executors
+    []
+  end
+
+  def get_filter!(key)
+    filter = condition_filters.find { |filter| filter.key == key }
+    raise UnsupportedConditionError, "Unsupported condition type: #{key}" unless filter
+    filter
+  end
+
+  def get_executor!(key)
+    executor = action_executors.find { |executor| executor.key == key }
+    raise UnsupportedActionError, "Unsupported action type: #{key}" unless executor
+    executor
+  end
+
+  def as_json
+    {
+      filters: condition_filters.map(&:as_json),
+      executors: action_executors.map(&:as_json)
+    }
+  end
+
+  private
+    attr_reader :rule
+
+    def family
+      rule.family
+    end
+end

app/models/rule/registry/transaction_resource.rb

@@ -0,0 +1,32 @@
+class Rule::Registry::TransactionResource < Rule::Registry
+  def resource_scope
+    family.transactions.active.with_entry.where(entry: { date: rule.effective_date.. })
+  end
+
+  def condition_filters
+    [
+      Rule::ConditionFilter::TransactionName.new(rule),
+      Rule::ConditionFilter::TransactionAmount.new(rule),
+      Rule::ConditionFilter::TransactionMerchant.new(rule)
+    ]
+  end
+
+  def action_executors
+    enabled_executors = [
+      Rule::ActionExecutor::SetTransactionCategory.new(rule),
+      Rule::ActionExecutor::SetTransactionTags.new(rule)
+    ]
+
+    if ai_enabled?
+      enabled_executors << Rule::ActionExecutor::AutoCategorize.new(rule)
+      enabled_executors << Rule::ActionExecutor::AutoDetectMerchants.new(rule)
+    end
+
+    enabled_executors
+  end
+
+  private
+    def ai_enabled?
+      Provider::Registry.get_provider(:openai).present?
+    end
+end