Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use memfd for file data #835

Closed
Xyene opened this issue Feb 13, 2021 · 2 comments · Fixed by #990
Closed

Use memfd for file data #835

Xyene opened this issue Feb 13, 2021 · 2 comments · Fixed by #990

Comments

@Xyene
Copy link
Member

Xyene commented Feb 13, 2021

We can create a memfd pointing to decompressed problem data, and pass that off to the submissions.

Pros:

  • no more piping necessary in judge; fewer context switches as a result
  • submissions can seek the fd
  • still no disk thrashing

Cons(?):

  • 3.7+ kernel necessary
  • extra memcpy
@quantum5
Copy link
Member

Support for creation of memfd-backed files is implemented in #836, with unlinked temporary files as fallback on FreeBSD.

This should now be a matter of writing decompressed data into an MemoryIO instance and passing the fd to TracedPopen.

@Xyene
Copy link
Member Author

Xyene commented Dec 30, 2024

Some stats from prod (numbers are the total number of context switches, summed across all test cases):

miracle-base-1: 1402
miracle-base-2: 1201
miracle-tip-1: 47
miracle-tip-2: 57
rmt-base-1: 65887
rmt-base-2: 65318
rmt-base-3: 61411
rmt-tip-1: 25833
rmt-tip-2: 25849

miracle refers to https://dmoj.ca/problem/miraclesort, rmt to https://dmoj.ca/problem/ccc17s5.

I expect we could decrease the number of RMT context switches by implementing writing submission output to memfds instead.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants