People

External Contributor

Yitao Li

Posts and resources by Yitao Li#

blog post

sparklyr.sedona: A sparklyr extension for analyzing geospatial data

sparklyr.sedona: A sparklyr extension for analyzing geospatial data

We are excited to announce the availability of sparklyr.sedona, a sparklyr extension making geospatial functionalities of the Apache Sedona library easily accessible from R

Yitao Li

Jul 7, 2021

MLOps and Admin AI Packages Releases Distributed Computing Spatial Data

blog post

sparklyr 1.7: New data sources and spark_apply() capabilities, better interfaces for sparklyr extensions, and more!

sparklyr 1.7: New data sources and spark_apply() capabilities, better interfaces for sparklyr extensions, and more!

Sparklyr 1.7 delivers much-anticipated improvements, including R interfaces for image and binary data sources, several new spark_apply() capabilities, and better integration with sparklyr extensions

Yitao Li

Jul 6, 2021

MLOps and Admin AI Packages Releases Distributed Computing

blog post

sparklyr 1.6: weighted quantile summaries, power iteration clustering, spark_write_rds(), and more

The sparklyr 1.6 release introduces weighted quantile summaries, an R interface to power iteration clustering, spark_write_rds(), as well as a number of dplyr-related improvements

Yitao Li

Mar 25, 2021

MLOps and Admin AI Packages Releases Distributed Computing

blog post

A hollywood star with the text 'sparklyr 1.5'

sparklyr 1.5: better dplyr interface, more sdf_* functions, and RDS-based serialization routines

Unlike all three previous sparklyr releases, the recent release of sparklyr 1.5 placed much more emphasis on enhancing existing sparklyr features rather than creating new ones. As a result, many valuable suggestions from sparklyr users were taken into account and were successfully addressed in a long list of bug fixes and improvements

Yitao Li

Dec 14, 2020

MLOps and Admin AI Packages Releases Distributed Computing

blog post

sparklyr.flint 0.2: ASOF Joins, OLS Regression, and additional summarizers

sparklyr.flint 0.2: ASOF Joins, OLS Regression, and additional summarizers

We are excited to announce a number of powerful, new functionalities and improvements which are now part of sparklyr.flint 0.2!

Yitao Li

Oct 12, 2020

Machine Learning AI Packages Releases Time Series

blog post

sparklyr 1.4: Weighted Sampling, Tidyr Verbs, Robust Scaler, RAPIDS, and more

sparklyr 1.4: Weighted Sampling, Tidyr Verbs, Robust Scaler, RAPIDS, and more

Sparklyr 1.4 is now available! This release comes with delightful new features such as weighted sampling and tidyr verbs support for Spark dataframes, robust scaler for standardizing data based on median and interquartile range, spark_connect interface for RAPIDS GPU acceleration plugin, as well as a number of dplyr-related improvements

Yitao Li

Sep 30, 2020

MLOps and Admin AI Packages Releases Distributed Computing

blog post

Introducing sparklyr.flint: A time-series extension for sparklyr

Introducing sparklyr.flint: A time-series extension for sparklyr

We are pleased to announce that sparklyr.flint, a sparklyr extension for analyzing time series at scale with Flint, is now available on CRAN. Flint is an open-source library for working with time-series in Apache Spark which supports aggregates and joins on time-series datasets

Yitao Li

Sep 7, 2020

Machine Learning AI Time Series

blog post

Parallelized sampling using exponential variates

Parallelized sampling using exponential variates

How can the seemingly iterative process of weighted sampling without replacement be transformed into something highly parallelizable? Turns out a well-known technique based on exponential variates accomplishes exactly that

Yitao Li

Jul 29, 2020

Best Practices MLOps and Admin AI Concepts Distributed Computing

blog post

sparklyr 1.3: Higher-order Functions, Avro and Custom Serializers

Sparklyr 1.3 is now available, featuring integration of Spark higher-order functions, and data import/export in Avro and in user-defined serialization formats

Yitao Li

Jul 16, 2020

Data Wrangling Big Data Sparklyr Spark Packages Rstudio

blog post

sparklyr 1.3: Higher-order Functions, Avro and Custom Serializers

sparklyr 1.3: Higher-order Functions, Avro and Custom Serializers

Sparklyr 1.3 is now available, featuring exciting new functionalities such as integration of Spark higher-order functions and data import/export in Avro and in user-defined serialization formats

Yitao Li

Jul 16, 2020

MLOps and Admin AI Packages Releases Distributed Computing

blog post

sparklyr 1.2: Foreach, Spark 3.0 and Databricks Connect

sparklyr 1.2: Foreach, Spark 3.0 and Databricks Connect

sparklyr 1.2: foreach parallel backend, Databricks Connect support, and Spark 3.0 compatibility

Yitao Li

May 6, 2020

Data Wrangling MLOps and Admin Big Data Sparklyr Spark Packages Rstudio

blog post

sparklyr 1.2: Foreach, Spark 3.0 and Databricks Connect

A new sparklyr release is now available. This sparklyr 1.2 release features new functionalities such as support for Databricks Connect, a Spark backend for the ‘foreach’ package, inter-op improvements for working with Spark 3.0 preview, as well as a number of bug fixes and improvements addressing user-visible pain points

Yitao Li

Apr 21, 2020

MLOps and Admin AI Packages Releases Distributed Computing