Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Remote cancel job can fail sometimes #711

Open
atruskie opened this issue Feb 3, 2025 · 0 comments
Open

Remote cancel job can fail sometimes #711

atruskie opened this issue Feb 3, 2025 · 0 comments
Labels
bug feature: analysis jobs

Comments

@atruskie
Copy link
Member

atruskie commented Feb 3, 2025 

2025-01-30T08:34:09.591 W [414:5480 status.rb:258] [BawWorkers::Jobs::Analysis::RemoteCancelJob] [RemoteCancelJob:job=5:t=20250130T061053-641155Z] Rails -- [Status] job failed -- {:error=>"4294967296 is out of range for ActiveModel::Type::Integer with limit 4 bytes"}
2025-01-30T08:34:09.592 E [414:5480 application_job.rb:69] [BawWorkers::Jobs::Analysis::RemoteCancelJob] [RemoteCancelJob:job=5:t=20250130T061053-641155Z] Rails -- Unhandled job error -- Exception: ActiveModel::RangeError: 4294967296 is out of range for ActiveModel::Type::Integer with limit 4 bytes
/usr/local/bundle/gems/activemodel-7.2.1.1/lib/active_model/type/integer.rb:95:in `ensure_in_range'
/usr/local/bundle/gems/activemodel-7.2.1.1/lib/active_model/type/integer.rb:71:in `serialize_cast_value'
/usr/local/bundle/gems/activemodel-7.2.1.1/lib/active_model/type/serialize_cast_value.rb:31:in `serialize'
/usr/local/bundle/gems/activemodel-7.2.1.1/lib/active_model/attribute.rb:208:in `_value_for_database'
/usr/local/bundle/gems/activemodel-7.2.1.1/lib/active_model/attribute.rb:57:in `value_for_database'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/quoting.rb:227:in `block in type_casted_binds'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/quoting.rb:225:in `map'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/quoting.rb:225:in `type_casted_binds'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:891:in `exec_no_cache'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/postgresql_adapter.rb:872:in `execute_and_clear'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:79:in `exec_delete'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:208:in `update'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/query_cache.rb:27:in `update'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/persistence.rb:280:in `block in _update_record'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:389:in `with_connection'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_handling.rb:296:in `with_connection'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/persistence.rb:279:in `_update_record'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/persistence.rb:887:in `_update_row'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/locking/optimistic.rb:93:in `_update_row'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/persistence.rb:909:in `_update_record'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/attribute_methods/dirty.rb:234:in `_update_record'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/callbacks.rb:449:in `block (2 levels) in _update_record'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/timestamp.rb:140:in `record_update_timestamps'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/callbacks.rb:449:in `block in _update_record'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:110:in `run_callbacks'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:913:in `_run_update_callbacks'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/callbacks.rb:449:in `_update_record'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/timestamp.rb:122:in `_update_record'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/persistence.rb:896:in `create_or_update'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/callbacks.rb:441:in `block in create_or_update'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/autosave_association.rb:362:in `around_save_collection_association'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:141:in `run_callbacks'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:913:in `_run_save_callbacks'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/callbacks.rb:441:in `create_or_update'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/timestamp.rb:127:in `create_or_update'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/persistence.rb:393:in `save'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/validations.rb:48:in `save'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/transactions.rb:362:in `block in save'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/transactions.rb:418:in `block (2 levels) in with_transaction_returning_status'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:359:in `transaction'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/transactions.rb:414:in `block in with_transaction_returning_status'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:389:in `with_connection'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_handling.rb:296:in `with_connection'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/transactions.rb:410:in `with_transaction_returning_status'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/transactions.rb:362:in `save'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/suppressor.rb:52:in `save'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/persistence/active_record_persistence.rb:87:in `aasm_save'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/persistence/orm.rb:24:in `aasm_write_state'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/instance_base.rb:129:in `set_current_state_with_persistence'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/aasm.rb:157:in `aasm_fired'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/aasm.rb:107:in `aasm_fire_event'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/persistence/orm.rb:132:in `block in aasm_fire_event'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/persistence/active_record_persistence.rb:105:in `block in aasm_transaction'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/transaction.rb:616:in `block in within_new_transaction'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/concurrency/null_lock.rb:9:in `synchronize'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/transaction.rb:613:in `within_new_transaction'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/database_statements.rb:361:in `transaction'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/transactions.rb:234:in `block in transaction'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:395:in `with_connection'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/connection_handling.rb:296:in `with_connection'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/transactions.rb:233:in `transaction'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/persistence/active_record_persistence.rb:103:in `aasm_transaction'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/persistence/orm.rb:131:in `aasm_fire_event'
/usr/local/bundle/gems/aasm-5.5.0/lib/aasm/base.rb:126:in `block in event'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/analysis/remote_cancel_job.rb:89:in `cancel_item'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/analysis/remote_cancel_job.rb:76:in `block (2 levels) in perform'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/analysis/remote_cancel_job.rb:75:in `each'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/analysis/remote_cancel_job.rb:75:in `each_with_index'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/analysis/remote_cancel_job.rb:75:in `block in perform'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/analysis/remote_cancel_job.rb:59:in `loop'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/jobs/analysis/remote_cancel_job.rb:59:in `perform'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/execution.rb:68:in `block in _perform_job'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:121:in `block in run_callbacks'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/active_job/concurrency.rb:158:in `around_perform_check_concurrency'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/active_job/status.rb:234:in `block in safe_perform!'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/active_job/status.rb:233:in `catch'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/active_job/status.rb:233:in `safe_perform!'
/home/baw_web/baw-server/lib/gems/baw-workers/lib/baw_workers/active_job/status.rb:225:in `around_perform_with_status'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/usr/local/bundle/gems/i18n-1.14.6/lib/i18n.rb:348:in `with_locale'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/translation.rb:9:in `block (2 levels) in <module:Translation>'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/core_ext/time/zones.rb:65:in `use_zone'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/timezones.rb:9:in `block (2 levels) in <module:Timezones>'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `instance_exec'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:130:in `block in run_callbacks'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/callbacks.rb:141:in `run_callbacks'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/execution.rb:67:in `_perform_job'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:32:in `_perform_job'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/execution.rb:51:in `perform_now'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:26:in `block in perform_now'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/railties/job_runtime.rb:13:in `block in instrument'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:40:in `block in instrument'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/notifications.rb:210:in `block in instrument'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/notifications/instrumenter.rb:58:in `instrument'
/usr/local/bundle/gems/activesupport-7.2.1.1/lib/active_support/notifications.rb:210:in `instrument'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:39:in `instrument'
/usr/local/bundle/gems/activerecord-7.2.1.1/lib/active_record/railties/job_runtime.rb:11:in `instrument'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/instrumentation.rb:26:in `perform_now'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/logging.rb:32:in `block in perform_now'
/usr/local/bundle/gems/semantic_logger-4.16.1/lib/semantic_logger/base.rb:190:in `block in tagged'
/usr/local/bundle/gems/semantic_logger-4.16.1/lib/semantic_logger/semantic_logger.rb:356:in `tagged'
/usr/local/bundle/gems/semantic_logger-4.16.1/lib/semantic_logger/base.rb:202:in `tagged'
/usr/local/bundle/gems/rails_semantic_logger-4.17.0/lib/rails_semantic_logger/extensions/active_job/logging.rb:13:in `tag_logger'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/logging.rb:32:in `perform_now'
/usr/local/bundle/gems/activejob-7.2.1.1/lib/active_job/execution.rb:23:in `perform_now'
(baw):2:in `<main>'
/usr/local/bundle/gems/irb-1.14.1/lib/irb/workspace.rb:121:in `eval'
/usr/local/bundle/gems/irb-1.14.1/lib/irb/workspace.rb:121:in `evaluate'
/usr/local/bundle/gems/irb-1.14.1/lib/irb/context.rb:622:in `evaluate_expression'
/usr/local/bundle/gems/irb-1.14.1/lib/irb/context.rb:590:in `evaluate'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1051:in `block (2 levels) in eval_input'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1363:in `signal_status'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1043:in `block in eval_input'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1122:in `block in each_top_level_statement'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1119:in `loop'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1119:in `each_top_level_statement'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1042:in `eval_input'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1023:in `block in run'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1022:in `catch'
/usr/local/bundle/gems/irb-1.14.1/lib/irb.rb:1022:in `run'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/commands/console/irb_console.rb:119:in `start'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/commands/console/console_command.rb:59:in `start'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/commands/console/console_command.rb:8:in `start'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/commands/console/console_command.rb:87:in `perform'
/usr/local/bundle/gems/thor-1.3.2/lib/thor/command.rb:28:in `run'
/usr/local/bundle/gems/thor-1.3.2/lib/thor/invocation.rb:127:in `invoke_command'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/command/base.rb:178:in `invoke_command'
/usr/local/bundle/gems/thor-1.3.2/lib/thor.rb:538:in `dispatch'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/command/base.rb:73:in `perform'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/command.rb:71:in `block in invoke'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/command.rb:149:in `with_argv'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/command.rb:69:in `invoke'
/usr/local/bundle/gems/railties-7.2.1.1/lib/rails/commands.rb:18:in `<main>'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
<internal:/usr/local/lib/ruby/3.2.0/rubygems/core_ext/kernel_require.rb>:37:in `require'
/usr/local/bundle/gems/bootsnap-1.18.4/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
./bin/rails:4:in `<main>'
@atruskie atruskie mentioned this issue Feb 3, 2025
Open
atruskie added a commit that referenced this issue Feb 4, 2025
@atruskie
Change remote cancel job
c1d2896 
Adds better progress messages as it iterates through batches.

Also patches ActiveModel::Attribute so we know which attribute causes a range overflow. Related to #711
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug feature: analysis jobs
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@atruskie