Improved Deployment Process

[SnazzyPanda]

This issue is largely meant to start a discussion and get ideas on the matter. Hopefully it can also encourage someone to implement something like this solution, or one that can achieve a similar improvement.

Issue

I may be wrong, but, as far as I can tell, the deploying of mods results in going through and re-extracting ALL (or nearly all) enabled mods regardless of what mods have been changed/added. Ideally, I think we should try to minimize the work to reach the mod-state that the user wants.

For example, if you add a single mod to the end of the mod order, it takes quite a bit of time to go through the entire deploy process. In this case, we could ideally detect that just the one mod was added, and only extract and deploy the one mod to reach the desired state.

Potential Solution (overview)

Without having fully examined the current process, I thought about how this problem could be solved, and I feel like I came to a workable solution. I don't know if any of this matches what already exists, or if this would mean a complete rewrite of the mod handling for deployment.

Here are my thoughts on how this could be solved:

• For each mod archive, store its filename + file hash (using xml for this seems like it would make sense).
• Under each mod's archive info (filename/hash), store the info for each file contained in that archive, including the game-relative path for each file and the file's hash.
• I think this information only needs to be generated during the deployment phase for enabled mods. The information should be kept until the mod is deleted (if the mod is disabled, we may still need the info, and there would not be a need to re-generate it).
• With that information, we can essentially simulate (in memory) the end-state for the deployment
• We can then check the files we expect to deploy against the files that already exist under the game directory. 

This should allow us to minimize work by deploying only the mods (or even individual mod files) that we actually need to deploy to reach the final, desired mod state.

Deployment

Here is a bit more depth on how I see this working:

1. When the user hits deploy, we grab the enabled mods in their load order.
2. We loop over each mod and check if we already have the "deploy info" for the zip/rar/7z/archive, and that it matches the hash we have stored.
   1. If a mod archive does match, we pull in the info for the individual files, overwriting any conflicts (in memory/whatever we are using internally), as we go through the mods in the load order.
   2. If it does not match, or does not exist in the internal "deploy info", we generate and store (to a file tracking the info)/add the "deploy info" for the mod by:
      1. Getting the archive's hash to and name to store things under
      2. Extracting its contents to a temporary directory
      3. Grabbing each file's path and hash to store. (I think it would be most useful to store the file's relative deployed path for comparing with other files)
      4. We can then move the files to a staged deploy directory, as we will likely need these files for the final deployment (and if any further conflicts come up, these files will be overwritten with the correct file as we go down the list)
3. Once we have generated any missing info and resolved file conflicts in memory, we can then compare our internal files for deploy with what is currently already deployed.
   1. As we go through, if we are missing any files that need to be deployed (and are not already deployed), we will need to extract the related mod archive and get the file copied over to the pre-deploy directory. Ideally this is optimized to extract what we need from each mod archive once (and not extract each time there is a missing file or something inefficient like that)
4. At this point, our pre-deploy directory should contain any missing files and any files that we need to overwrite, so we should be able to finish the deployment by moving the files over to the final deploy directory

With something along these lines, we should be able to effectively minimize the work we do on subsequent deployments (at the cost of a bit of overhead for the initial deployment compared to what is done now).

Disabling Mods/File Cleanup

The main thing that I think may need some more consideration here is how to handle removing mod files, for example when a mod is disabled. My initial thought is to handle removing relevant files as mods are disabled:

1. When a mod is disabled, you go to the stored "deploy info" for the mod, and compare its files to what is still deployed.
2. If the file hashes match, you would remove the file.
3. If the hashes don't match, assume that the file is handled by a different mod/externally from Lampray

There are concerns with doing this:

1. First, the user may not want files to be removed immediately when disabled, as that is not how Lampray currently functions.
2. Second, this adds work that needs to be done during disabling of mods, which could take some time depending on the mod. We may need to provide some feedback/info to the user so that they know something is happening (possibly locking the interface as well?).
3. Third is that we make the assumption that file hashes that match belong to the mod. There is a chance that this is not always the case, and that we could end up removing a vital file for other mods. This may be mitigated some if the other mod is managed and active in Lampray (as we should just need to deploy again), but this may need to be considered more for other such issues.

Additional Thoughts/Notes

With this setup, I think we would need (and maybe even want) to remove files immediately when an enabled/deployed mod is disabled. This would help prevent files from being left over if a mod is then deleted before deploy. The alternative is that we need to handle this cleanup in the deploy phase (which means we would only delete a mod's "deploy info" during deployment (presumptively during pre-deploy).

It looks like right now, some amount of this info may be getting generated and dumped into the lampTrack.mdf file (specifically file hashes?), so perhaps we could make use of that. To help contain things, I think it may be a good idea to have the info split out and managed separately for each game (instead of having everything for all games in lampTrack.mdf), but it likely would not matter functionally.

These are just my thoughts at this time. I don't know that this is a good solution, I just think it might be a workable solution. In the long run, it would probably be good to find a way to have separate deployment types that you let the user choose from (ie, you could have a "direct deploy" that copies files like Lampray current does, "soft-link deploy" using something like symlinks to files, "virtual deploy" for I believe vfs or uvfs or ofs or something like that, etc). If the deployment is separated in this way, I think it could still be valuable to keep the original deployment around as an option (something like "full direct deploy"?).

I am not a c++ developer and have not familiarized myself with how the deployment currently functions, so actually implementing this is quite a ways outside of my current skillset. Hopefully my description and ideas above makes some sense, and can help improve the deployment process.

[CHollingworth]

After a quick read:

You are right this is how lamp currently deals with deploying mods regardless of state.

Your concept for deployment seems almost flawless the only changes id make is with disabling mods/cleanup i think that it would be done during deployment time.

On the 3rd concern of Clean-up, Lamp currently takes a hash (and a copy) of any file that it replaces during deployment time so that the reset it can move files back into place if we employ the same strategy most issues regarding it should be avoided if not it id assume user error removing a mod needed for another. As much as id like to remove a mod the moment its disabled (like MO2 or Vortex) i personally think its a waste of time if you will be doing a cpu heavy task (Deployment) later anyway.

Lamp track is used at the moment to restore the game back to its default state and with this id probably start anew as lamptrack data wont be too useful here.

On different deployment methods, in my own testing proton games do not seem to like being run with symlinked files and something along the lines of vfs is heavily system specific although i am looking at potentially implementing it using a cross-platform library or including a new dependency.

[SnazzyPanda]

Your concept for deployment seems almost flawless the only changes id make is with disabling mods/cleanup i think that it would be done during deployment time.

As much as id like to remove a mod the moment its disabled (like MO2 or Vortex) i personally think its a waste of time if you will be doing a cpu heavy task (Deployment) later anyway.

Your confidence in the concept is nice to hear. It sounds like you have a good way to resolve the deactivation issues I had. I would just make a note here that, if we do cleanup during deploy, we would probably want to detect when someone enables/disables a mod and display a notice reminding them to deploy their changes (possibly a banner at the top/bottom?). Obviously you would then clear out that banner at the end of the deployment and reset things to go back to detecting if deployment is needed again.

On different deployment methods, in my own testing proton games do not seem to like being run with symlinked files and something along the lines of vfs is heavily system specific although i am looking at potentially implementing it using a cross-platform library or including a new dependency.

I get that. I was throwing those out there as a possible consideration for the future (in that we might want to support multiple deployment methods and enable the users to choose). For the time being, I was thinking that there may still be a use-case for the "fully deploy everything anyway" deploy, even if we have this method set up and working. The main point being that, in the process of potentially implementing this improvement, it may be worth exploring modifying things internally to make supporting alternate deployment methods easier in the future.

If this mod manager gets to be more general-purpose (which I hope it does), the alternate methods could be valuable options for users to apply as needed/desired. If other deployment methods are added, it may be good to have a spot somewhere in the application that maybe notes the possible issues with each deployment method, though that is a separate issue for the future.