Skip to content

Commit

Permalink
[SPARK-40114][R] Arrow 9.0.0 support with SparkR
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR switches the deprecated API `write_arrow(..., raw())` to use `write_to_raw`. This is exactly the same behaviour/replacement.

See also:
- https://arrow.apache.org/docs/r/news/index.html#reading-and-writing-9-0-0 for `write_arrow`
- https://arrow.apache.org/docs/r/news/index.html#feather-v2-0-17-0 for `write_to_raw`.
- https://github.com/apache/arrow/blob/apache-arrow-8.0.0/r/R/deprecated.R#L31-L40 for deprecated implementation of `write_arrow`.

### Why are the changes needed?

`write_arrow` is removed out from Arrow 9.0.0 in R Array library.

Note that `write_to_raw` has existed from Arrow 0.17.0 so I expect this is a safe change.

### Does this PR introduce _any_ user-facing change?

Yes, users can use R Arrow 9.0.0.

### How was this patch tested?

Ci in this PR should test it out.

Closes apache#37553 from HyukjinKwon/SPARK-40114.

Authored-by: Hyukjin Kwon <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
  • Loading branch information
HyukjinKwon authored and dongjoon-hyun committed Aug 17, 2022
1 parent 103768b commit eef7596
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 3 deletions.
2 changes: 1 addition & 1 deletion R/pkg/R/serialize.R
Original file line number Diff line number Diff line change
Expand Up @@ -231,7 +231,7 @@ writeSerializeInArrow <- function(conn, df) {
# There looks no way to send each batch in streaming format via socket
# connection. See ARROW-4512.
# So, it writes the whole Arrow streaming-formatted binary at once for now.
writeRaw(conn, arrow::write_arrow(df, raw()))
writeRaw(conn, arrow::write_to_raw(df))
} else {
stop("'arrow' package should be installed.")
}
Expand Down
3 changes: 1 addition & 2 deletions appveyor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,7 @@ install:
# Install SBT and dependencies
- ps: .\dev\appveyor-install-dependencies.ps1
# Required package for R unit tests. xml2 is required to use jUnit reporter in testthat.
# TODO(SPARK-40114): Add 'arrow' back with supporting Arrow 9.0.0
- cmd: Rscript -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival', 'xml2'), repos='https://cloud.r-project.org/')"
- cmd: Rscript -e "install.packages(c('knitr', 'rmarkdown', 'testthat', 'e1071', 'survival', 'arrow', 'xml2'), repos='https://cloud.r-project.org/')"
- cmd: Rscript -e "pkg_list <- as.data.frame(installed.packages()[,c(1, 3:4)]); pkg_list[is.na(pkg_list$Priority), 1:2, drop = FALSE]"

build_script:
Expand Down

0 comments on commit eef7596

Please sign in to comment.