Sparklyr

sparklyr 1.3: Higher-order Functions, Avro and Custom Serializers

Sparklyr 1.3 is now available, featuring integration of Spark higher-order functions, and data import/export in Avro and in user-defined serialization formats

sparklyr 1.2: Foreach, Spark 3.0 and Databricks Connect

sparklyr 1.2: foreach parallel backend, Databricks Connect support, and Spark 3.0 compatibility

sparklyr 1.1: Foundations, Books, Lakes and Barriers

sparklyr 1.1: Delta Lake support, Spark 3.0 preview, and barrier execution for deep learning

Edgar Ruiz | Databases using R The latest | RStudio (2019)

Learn about the latest packages and techniques that can help you access and analyze data found inside databases using R. Many of the techniques we will cover are based on our personal and the community’s experiences of implementing concepts introduced last year, such as offloading most of the data wrangling to the database using dplyr, and using the RStudio IDE to preview the database’s layout and data. Also, learn more about the most recent improvements to the RStudio products that are geared to aid developers in using R with databases effectively. VIEW MATERIALS https://github.com/edgararuiz/databases-w-r About the Author Edgar Ruiz Edgar is the author and administrator of the https://db.rstudio.com web site, and current administrator of the [sparklyr] web site: https://spark.rstudio.com. Author of the Data Science in Spark with sparklyr cheatsheet. Co-author of the dbplyr package and creator of the dbplot package

Edgar Ruiz

Javier Luraschi | Scaling R with Spark | RStudio (2019)

This talk introduces new features in sparklyr that enable real-time data processing, brand new modeling extensions and significant performance improvements. The sparklyr package provides an interface to Apache Spark to enable data analysis and modeling in large datsets through familiar packages like dplyr and broom. VIEW MATERIALS https://github.com/rstudio/rstudio-conf/tree/master/2019/Scaling%20R%20with%20Spark%20-%20Javier%20Luraschi About the Author Javier Luraschi Javier is a Software Engineer with experience in technologies ranging from desktop, web, mobile and backend; to augmented reality and deep learning applications. He previously worked for Microsoft Research and SAP and holds a double degree in Mathematics and Software Engineering

sparklyr 1.0: Apache Arrow, XGBoost, Broom and TFRecords

sparklyr 1.0: Apache Arrow for faster data transfers, XGBoost models, broom integration, and TFRecords

sparklyr 0.9: Streams and Kubernetes

sparklyr 0.9: Spark structured streams for real-time data processing and Kubernetes cluster support

sparklyr 0.8: Production pipelines and graphs

sparklyr 0.8: production ML pipelines with mleap export and graph analysis with graphframes

sparklyr 0.7: Spark Pipelines and Machine Learning

sparklyr 0.7: ML Pipelines API for building, tuning, and deploying machine learning workflows at scale

Understanding Sparklyr Deployment Modes | RStudio Webinar - 2017

This is a recording of an RStudio webinar. You can subscribe to receive invitations to future webinars at https://www.rstudio.com/resources/webinars/. We try to host a couple each month with the goal of furthering the R community’s understanding of R and RStudio’s capabilities. We are always interested in receiving feedback, so please don’t hesitate to comment or reach out with a personal message

Sparklyr: Using Spark with RMarkdown | RStudio Webinar - 2016

This is a recording of an RStudio webinar. You can subscribe to receive invitations to future webinars at https://www.rstudio.com/resources/web …. We try to host a couple each month with the goal of furthering the R community’s understanding of R and RStudio’s capabilities. We are always interested in receiving feedback, so please don’t hesitate to comment or reach out with a personal message

Extending Spark Using Sparklyr | RStudio Webinar - 2017

This is a recording of an RStudio webinar. You can subscribe to receive invitations to future webinars at https://www.rstudio.com/resources/webinars/. We try to host a couple each month with the goal of furthering the R community’s understanding of R and RStudio’s capabilities. We are always interested in receiving feedback, so please don’t hesitate to comment or reach out with a personal message

Advanced Features of Sparkyr | RStudio Webinar - 2017

This is a recording of an RStudio webinar. You can subscribe to receive invitations to future webinars at https://www.rstudio.com/resources/webinars/. We try to host a couple each month with the goal of furthering the R community’s understanding of R and RStudio’s capabilities. We are always interested in receiving feedback, so please don’t hesitate to comment or reach out with a personal message

SparklyR

Tareef Kawaf | Posit useR! 2017 Presentation, Brussels

What has Posit been up to?

sparklyr 0.6: Distributed R and external sources

sparklyr 0.6: distributed R with spark_apply() and external data source connections

sparklyr 0.5: Livy and dplyr improvements

sparklyr 0.5 extends dplyr with do() and n_distinct(), adds experimental Livy support for remote Spark connections