Nf test tune #166
Merged
Nf test tune #166
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e, weigths are now set in a reproducible and deterministic manner
…he correct tune_run path
…de of the process dir
mathysgrapotte
approved these changes
Jul 24, 2024
| elif user_tune_config["tune"]["scheduler"]["name"] == "FIFOScheduler": | ||
| user_tune_config["tune"]["run_params"]["stop"]["training_iteration"] = 1 | ||
|
|
||
| # TODO future schedulers specific info will go here as well. maybe find a cleaner way. |
There was a problem hiding this comment.
Most likely this needs to become a stimulus class in the same way experiment is
| results.save_best_model(output) | ||
| results.save_best_config(best_config_path) | ||
| results.save_best_metrics_dataframe(best_metrics_path) | ||
| results.save_best_optimizer(best_optimizer_path) | ||
|
|
||
| # debug section. predict the validation data using the best model. |
There was a problem hiding this comment.
Why does the behavior of debug load the best model and tests on the validation data, shouldn't this be reserved to the analysis module? How is this helping us with debug tuning?
| @@ -179,12 +210,34 @@ def setup(self, config: dict, training: object, validation: object) -> None: | |||
| self.training = DataLoader(training, batch_size=self.batch_size, shuffle=True) # TODO need to check the reproducibility of this shuffling | |||
| self.validation = DataLoader(validation, batch_size=self.batch_size, shuffle=True) | |||
|
|
|||
| # debug section, first create a dedicated directory for each worker inside Ray_results/<tune_model_run_specific_dir> location | |||
There was a problem hiding this comment.
In the future, I believe this should be done regardless of debug or not (saving seed and initial model), this would be a "robustness mode" toggled "on" by default!
There was a problem hiding this comment.
Or a "reproducibility" mode, as I believe the formula "model + initial state + seed + training code + training data" is our "deep learning" container
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes aported:
TODO for the future is decide wich flavor of the nf-test tune should be put as guthub action to test reproducibility.