Aggregating batched results

[JonathanFrias]

So I don't see a dedicated api, but here's a use case that I think may help people:

class MyParallelWorkerJob
    include GoodJob::ActiveJobExtensions::Batches

   def perform
      # create some resource somewhere in parallel
      id = http.post(...)

     # concurrently safe aggregation
     append_id(%Q[
      UPDATE
        good_job_batches
      SET serialized_properties = jsonb_insert(serialized_properties, '{"ids", 0}', '"#{id}"')
      WHERE id = '#{self.batch.id}'
     ])
   end
end

class OnFinishJob
  def perform
     self.batch[:properties][:ids].each do |id|
        # You can handle your aggregated resources! 
     end
  end
end

GoodJob::Batch.enqueue(on_finish: OnFinishJob, ids: [], stage: 1) do
   1000.times do 
      MyParallelWorkerJob.perform_later
   end
end

Hopefully this inspires a more dedicated API to achieve this :)

[bensheldon]

@JonathanFrias could you explain the purpose? Sorry, I'm not quite following if you're doing more than:

batch.properties[:ids] = [...]
batch.save

...oh, unless the goal is to safely insert the property in parallel?

[bensheldon]

There are no locks. No reason one way or the other, it just isn't something that's been implemented ("sharp knives", etc.). If you need a lock, you'll need to add one yourself or suggest an interface to be implemented in GoodJob.

[gap777]

@bensheldon I'm sorry to be dense... can the same batch property (batch.properties[:ids]) be updated from batched-jobs concurrently?

[bensheldon]

Safely concurrently? No.

The properties column is updated atomically so any previously loaded values will overwrite newer updates ones on save.

[JonathanFrias]

@gap777 I didn't adopt that style, I kept the sql append. Although if I were to redo this, I might try something like

batch.properties[:ids] = Concurrent::Array.new # from concurrent-ruby gem
batch.properties[:ids] << compute_id_somehow() # in the various jobs

Not sure if that's safe code. Try it and let me know

I don't think that would trigger any locks for appends, I haven't tried it for this use case. Both should use the same principle, just using the database or not. The only lock you might need is for creating the array.

If you need inspiration look at how ActiveRecord implements increment_counter for thread safe updates

[JeanSebTr]

@JonathanFrias Concurrent::Array won't preserve its concurrency guaranties once deserialized in another job (it will probably be deserialized as a normal array)

As ben mentioned, the whole properties column gets updated at once, so no matter the access/write pattern of a single property by a job, the job will persist its own stale view of the properties.