Other packages > Find by keyword >

sparklyr  

R Interface to Apache Spark
View on CRAN: Click here


Download and install sparklyr package within the R console
Install from CRAN:
install.packages("sparklyr")

Install from Github:
library("remotes")
install_github("cran/sparklyr")

Install by package version:
library("remotes")
install_version("sparklyr", "1.8.6")



Attach the package and use:
library("sparklyr")
Maintained by
Edgar Ruiz
[Scholar Profile | Author Map]
All associated links for this package
First Published: 2016-09-24
Latest Update: 2023-09-02
Description:
R interface to Apache Spark, a fast and general engine for big data processing, see <https://spark.apache.org/>. This package supports connecting to local and remote Apache Spark clusters, provides a 'dplyr' compatible back-end, and provides an interface to Spark's built-in machine learning algorithms.
How to cite:
Edgar Ruiz (2016). sparklyr: R Interface to Apache Spark. R package version 1.8.6, https://cran.r-project.org/web/packages/sparklyr. Accessed 22 Dec. 2024.
Previous versions and publish date:
0.4 (2016-09-24 22:40), 0.5.1 (2016-12-19 15:56), 0.5.2 (2017-02-16 22:40), 0.5.3 (2017-03-09 18:09), 0.5.4 (2017-04-25 09:24), 0.5.5 (2017-05-26 08:19), 0.5.6 (2017-06-10 23:42), 0.5 (2016-12-18 11:23), 0.6.0 (2017-07-29 07:22), 0.6.1 (2017-08-06 18:35), 0.6.2 (2017-08-13 07:40), 0.6.3 (2017-09-19 18:08), 0.6.4 (2017-11-02 01:57), 0.7.0 (2018-01-23 09:49), 0.8.0 (2018-05-01 05:45), 0.8.1 (2018-05-02 09:59), 0.8.2 (2018-05-06 08:23), 0.8.3 (2018-05-12 07:53), 0.8.4 (2018-05-25 23:39), 0.9.1 (2018-09-27 07:00), 0.9.2 (2018-10-17 07:20), 0.9.3 (2018-11-29 07:00), 0.9.4 (2019-01-09 07:30), 1.0.0 (2019-02-25 09:30), 1.0.1 (2019-05-17 23:30), 1.0.2 (2019-07-04 08:33), 1.0.3 (2019-09-15 07:10), 1.0.4 (2019-10-04 23:40), 1.0.5 (2019-11-14 21:00), 1.1.0 (2020-01-11 01:50), 1.2.0 (2020-04-20 21:30), 1.3.0 (2020-06-27 18:50), 1.3.1 (2020-07-09 22:00), 1.4.0 (2020-09-16 15:30), 1.5.0 (2020-11-26 04:00), 1.5.1 (2020-12-01 06:50), 1.5.2 (2020-12-12 00:00), 1.6.0 (2021-03-09 22:20), 1.6.1 (2021-03-23 17:50), 1.6.2 (2021-03-30 15:20), 1.6.3 (2021-06-01 15:40), 1.7.0 (2021-06-10 10:50), 1.7.1 (2021-06-17 13:50), 1.7.2 (2021-09-16 17:00), 1.7.3 (2021-11-30 13:30), 1.7.4 (2022-01-08 12:20), 1.7.5 (2022-02-02 15:30), 1.7.6 (2022-05-26 17:20), 1.7.7 (2022-06-07 22:50), 1.7.8 (2022-08-16 22:30), 1.7.9 (2022-12-08 18:20), 1.8.0 (2023-03-07 01:40), 1.8.1 (2023-03-22 14:40), 1.8.2 (2023-07-01 06:20), 1.8.3 (2023-09-02 07:10), 1.8.4 (2023-10-30 09:20), 1.8.5 (2024-03-25 22:40)
Other packages that cited sparklyr R package
View sparklyr citation profile
Other R packages that sparklyr depends, imports, suggests or enhances
Complete documentation for sparklyr
Functions, R codes and Examples using the sparklyr R package
Some associated functions: DBISparkResult-class . arrow_enabled_object . checkpoint_directory . collect . collect_from_rds . compile_package_jars . connection_config . connection_is_open . connection_spark_shinyapp . copy_to . copy_to.spark_connection . distinct . download_scalac . dplyr_hof . ensure . fill . filter . find_scalac . ft_binarizer . ft_bucketizer . ft_chisq_selector . ft_count_vectorizer . ft_dct . ft_elementwise_product . ft_feature_hasher . ft_hashing_tf . ft_idf . ft_imputer . ft_index_to_string . ft_interaction . ft_lsh . ft_lsh_utils . ft_max_abs_scaler . ft_min_max_scaler . ft_ngram . ft_normalizer . ft_one_hot_encoder . ft_one_hot_encoder_estimator . ft_pca . ft_polynomial_expansion . ft_quantile_discretizer . ft_r_formula . ft_regex_tokenizer . ft_robust_scaler . ft_standard_scaler . ft_stop_words_remover . ft_string_indexer . ft_tokenizer . ft_vector_assembler . ft_vector_indexer . ft_vector_slicer . ft_word2vec . full_join . generic_call_interface . get_spark_sql_catalog_implementation . grapes-greater-than-grapes . hive_context_config . hof_aggregate . hof_array_sort . hof_exists . hof_filter . hof_forall . hof_map_filter . hof_map_zip_with . hof_transform . hof_transform_keys . hof_transform_values . hof_zip_with . inner_join . invoke . invoke_method . j_invoke . j_invoke_method . jarray . jfloat . jfloat_array . jobj_class . jobj_set_param . join.tbl_spark . left_join . list_sparklyr_jars . livy_config . livy_install . livy_service . ml-constructors . ml-model-constructors . ml-params . ml-persistence . ml-transform-methods . ml-tuning . ml_add_stage . ml_aft_survival_regression . ml_als . ml_als_tidiers . ml_bisecting_kmeans . ml_call_constructor . ml_chisquare_test . ml_clustering_evaluator . ml_corr . ml_decision_tree . ml_default_stop_words . ml_evaluate . ml_evaluator . ml_feature_importances . ml_fpgrowth . ml_gaussian_mixture . ml_generalized_linear_regression . ml_glm_tidiers . ml_gradient_boosted_trees . ml_isotonic_regression . ml_isotonic_regression_tidiers . ml_kmeans . ml_kmeans_cluster_eval . ml_lda . ml_lda_tidiers . ml_linear_regression . ml_linear_svc . ml_linear_svc_tidiers . ml_logistic_regression . ml_logistic_regression_tidiers . ml_metrics_binary . ml_metrics_multiclass . ml_metrics_regression . ml_model_data . ml_multilayer_perceptron_classifier . ml_multilayer_perceptron_tidiers . ml_naive_bayes . ml_naive_bayes_tidiers . ml_one_vs_rest . ml_pca_tidiers . ml_pipeline . ml_power_iteration . ml_prefixspan . ml_random_forest . ml_stage . ml_standardize_formula . ml_summary . ml_survival_regression_tidiers . ml_tree_tidiers . ml_uid . ml_unsupervised_tidiers . mutate . na.replace . nest . pipe . pivot_longer . pivot_wider . print_jobj . quote_sql_name . random_string . reactiveSpark . reexports . registerDoSpark . register_extension . replace_na . right_join . sdf-saveload . sdf-transform-methods . sdf_along . sdf_bind . sdf_broadcast . sdf_checkpoint . sdf_coalesce . sdf_collect . sdf_copy_to . sdf_crosstab . sdf_debug_string . sdf_describe . sdf_dim . sdf_distinct . sdf_drop_duplicates . sdf_expand_grid . sdf_fast_bind_cols . sdf_from_avro . sdf_is_streaming . sdf_last_index . sdf_len . sdf_num_partitions . sdf_partition_sizes . sdf_persist . sdf_pivot . sdf_project . sdf_quantile . sdf_random_split . sdf_rbeta . sdf_rbinom . sdf_rcauchy . sdf_rchisq . sdf_read_column . sdf_register . sdf_repartition . sdf_residuals . sdf_rexp . sdf_rgamma . sdf_rgeom . sdf_rhyper . sdf_rlnorm . sdf_rnorm . sdf_rpois . sdf_rt . sdf_runif . sdf_rweibull . sdf_sample . sdf_schema . sdf_separate_column . sdf_seq . sdf_sort . sdf_sql . sdf_to_avro . sdf_unnest_longer . sdf_unnest_wider . sdf_weighted_sample . sdf_with_sequential_id . sdf_with_unique_id . select . separate . spark-api . spark-connections . spark_adaptive_query_execution . spark_advisory_shuffle_partition_size . spark_apply . spark_apply_bundle . spark_apply_log . spark_auto_broadcast_join_threshold . spark_coalesce_initial_num_partitions . spark_coalesce_min_num_partitions . spark_coalesce_shuffle_partitions . spark_compilation_spec . spark_compile . spark_config . spark_config_exists . spark_config_kubernetes . spark_config_packages . spark_config_settings . spark_config_value . spark_configuration . spark_connect_method . spark_connection-class . spark_connection . spark_connection_find . spark_context_config . spark_dataframe . spark_default_compilation_spec . spark_default_version . spark_dependency . spark_dependency_fallback . spark_extension . spark_get_java . spark_home_dir . spark_home_set . spark_ide_connection_open . spark_insert_table . spark_install . spark_install_find . spark_install_sync . spark_integ_test_skip . spark_jobj-class . spark_jobj . spark_last_error . spark_load_table . spark_log . spark_pipeline_stage . spark_read . spark_read_avro . spark_read_binary . spark_read_csv . spark_read_delta . spark_read_image . spark_read_jdbc . spark_read_json . spark_read_libsvm . spark_read_orc . spark_read_parquet . spark_read_source . spark_read_table . spark_read_text . spark_save_table . spark_statistical_routines . spark_table_name . spark_version . spark_version_from_home . spark_versions . spark_web . spark_write . spark_write_avro . spark_write_csv . spark_write_delta . spark_write_jdbc . spark_write_json . spark_write_orc . spark_write_parquet . spark_write_rds . spark_write_source . spark_write_table . spark_write_text . sparklyr_get_backend_port . sql-transformer . src_databases . stream_find . stream_generate_test . stream_id . stream_lag . stream_name . stream_read_csv . stream_read_delta . stream_read_json . stream_read_kafka . stream_read_orc . stream_read_parquet . stream_read_socket . stream_read_text . stream_render . stream_stats . stream_stop . stream_trigger_continuous . stream_trigger_interval . stream_view . stream_watermark . stream_write_console . stream_write_csv . stream_write_delta . stream_write_json . stream_write_kafka . stream_write_memory . stream_write_orc . stream_write_parquet . stream_write_text . sub-.tbl_spark . tbl_cache . tbl_change_db . tbl_uncache . transform_sdf . unite . unnest . worker_spark_apply_unbundle . 
Some associated R codes: arrow_data.R . avro_utils.R . browse_url.R . config_settings.R . config_spark.R . connection_instances.R . connection_progress.R . connection_shinyapp.R . connection_spark.R . core_arrow.R . core_config.R . core_connection.R . core_deserialize.R . core_gateway.R . core_invoke.R . core_jobj.R . core_serialize.R . core_utils.R . core_worker_config.R . data_copy.R . data_csv.R . data_interface.R . databricks_connection.R . dbi_spark_connection.R . dbi_spark_result.R . dbi_spark_table.R . do_spark.R . dplyr_do.R . dplyr_hof.R . dplyr_join.R . dplyr_spark.R . dplyr_spark_data.R . dplyr_spark_table.R . dplyr_sql.R . dplyr_sql_translation.R . dplyr_verbs.R . imports.R . install_spark.R . install_spark_versions.R . install_spark_windows.R . install_tools.R . java.R . jobs_api.R . kubernetes_config.R . livy_connection.R . livy_install.R . livy_invoke.R . livy_service.R . ml_classification_decision_tree_classifier.R . ml_classification_gbt_classifier.R . ml_classification_linear_svc.R . ml_classification_logistic_regression.R . ml_classification_multilayer_perceptron_classifier.R . ml_classification_naive_bayes.R . ml_classification_one_vs_rest.R . ml_classification_random_forest_classifier.R . ml_clustering.R . ml_clustering_bisecting_kmeans.R . ml_clustering_gaussian_mixture.R . ml_clustering_kmeans.R . ml_clustering_lda.R . ml_clustering_power_iteration.R . ml_constructor_utils.R . ml_evaluate.R . ml_evaluation_clustering.R . ml_evaluation_prediction.R . ml_evaluator.R . ml_feature_binarizer.R . ml_feature_bucketed_random_projection_lsh.R . ml_feature_bucketizer.R . ml_feature_chisq_selector.R . ml_feature_count_vectorizer.R . ml_feature_dct.R . ml_feature_dplyr_transformer.R . ml_feature_elementwise_product.R . ml_feature_feature_hasher.R . ml_feature_hashing_tf.R . ml_feature_idf.R . ml_feature_imputer.R . ml_feature_index_to_string.R . ml_feature_interaction.R . ml_feature_lsh_utils.R . ml_feature_max_abs_scaler.R . ml_feature_min_max_scaler.R . ml_feature_minhash_lsh.R . ml_feature_ngram.R . ml_feature_normalizer.R . ml_feature_one_hot_encoder.R . ml_feature_one_hot_encoder_estimator.R . ml_feature_pca.R . ml_feature_polynomial_expansion.R . ml_feature_quantile_discretizer.R . ml_feature_r_formula.R . ml_feature_regex_tokenizer.R . ml_feature_robust_scaler.R . ml_feature_sql_transformer.R . ml_feature_standard_scaler.R . ml_feature_stop_words_remover.R . ml_feature_string_indexer.R . ml_feature_string_indexer_model.R . ml_feature_tokenizer.R . ml_feature_vector_assembler.R . ml_feature_vector_indexer.R . ml_feature_vector_slicer.R . ml_feature_word2vec.R . ml_fpm_fpgrowth.R . ml_fpm_prefixspan.R . ml_helpers.R . ml_mapping_tables.R . ml_metrics.R . ml_model_als.R . ml_model_bisecting_kmeans.R . ml_model_constructors.R . ml_model_decision_tree.R . ml_model_gaussian_mixture.R . ml_model_generalized_linear_regression.R . ml_model_gradient_boosted_trees.R . ml_model_helpers.R . ml_model_isotonic_regression.R . ml_model_kmeans.R . ml_model_lda.R . ml_model_linear_regression.R . ml_model_linear_svc.R . ml_model_logistic_regression.R . ml_model_naive_bayes.R . ml_model_one_vs_rest.R . ml_model_random_forest.R . ml_model_utils.R . ml_param_utils.R . ml_persistence.R . ml_pipeline.R . ml_pipeline_utils.R . ml_print_utils.R . ml_recommendation_als.R . ml_regression_aft_survival_regression.R . ml_regression_decision_tree_regressor.R . ml_regression_gbt_regressor.R . ml_regression_generalized_linear_regression.R . ml_regression_isotonic_regression.R . ml_regression_linear_regression.R . ml_regression_random_forest_regressor.R . ml_stat.R . ml_summary.R . ml_transformation_methods.R . ml_transformer_and_estimator.R . ml_tuning.R . ml_tuning_cross_validator.R . ml_tuning_train_validation_split.R . ml_utils.R . ml_validator_utils.R . mutation.R . na_actions.R . new_model_multilayer_perceptron.R . params_validator.R . precondition.R . project_template.R . qubole_connection.R . reexports.R . sdf_dim.R . sdf_distinct.R . sdf_interface.R . sdf_ml.R . sdf_saveload.R . sdf_sequence.R . sdf_sql.R . sdf_stat.R . sdf_streaming.R . sdf_unnest_longer.R . sdf_unnest_wider.R . sdf_utils.R . sdf_wrapper.R . shell_connection.R . spark_apply.R . spark_apply_bundle.R . spark_compile.R . spark_connection.R . spark_context_config.R . spark_data_build_types.R . spark_dataframe.R . spark_extensions.R . spark_gateway.R . spark_gen_embedded_sources.R . spark_globals.R . spark_hive.R . spark_home.R . spark_ide.R . spark_invoke.R . spark_schema_from_rdd.R . spark_sql.R . spark_submit.R . spark_update_embedded_sources.R . spark_utils.R . spark_verify_embedded_sources.R . spark_version.R . sql_utils.R . stratified_sample.R . stream_data.R . stream_job.R . stream_operations.R . stream_shiny.R . stream_view.R . synapse_connection.R . tables_spark.R . tbl_spark.R . test_connection.R . tidiers_ml_aft_survival_regression.R . tidiers_ml_als.R . tidiers_ml_isotonic_regression.R . tidiers_ml_lda.R . tidiers_ml_linear_models.R . tidiers_ml_logistic_regression.R . tidiers_ml_multilayer_perceptron.R . tidiers_ml_naive_bayes.R . tidiers_ml_svc_models.R . tidiers_ml_tree_models.R . tidiers_ml_unsupervised_models.R . tidiers_pca.R . tidiers_utils.R . tidyr_fill.R . tidyr_nest.R . tidyr_pivot_longer.R . tidyr_pivot_utils.R . tidyr_pivot_wider.R . tidyr_separate.R . tidyr_unite.R . tidyr_unnest.R . tidyr_utils.R . utils.R . worker_apply.R . worker_connect.R . worker_connection.R . worker_invoke.R . worker_log.R . worker_main.R . yarn_cluster.R . yarn_config.R . yarn_ui.R . zzz.R .  Full sparklyr package functions and examples
Downloads during the last 30 days
Get rewarded with contribution points by helping add
Reviews / comments / questions /suggestions ↴↴↴

Today's Hot Picks in Authors and Packages

elect  
Estimation of Life Expectancies Using Multi-State Models
Functions to compute state-specific and marginal life expectancies. The computation is based on a fi ...
Download / Learn more Package Citations See dependency  
Rfast2  
A Collection of Efficient and Extremely Fast R Functions II
A collection of fast statistical and utility functions for data analysis. Functions for regression, ...
Download / Learn more Package Citations See dependency  
quickcode  
Quick and Essential 'R' Tricks for Better Scripts
The NOT functions, 'R' tricks and a compilation of some simple quick plus often used 'R' codes to im ...
Download / Learn more Package Citations See dependency  
composits  
Compositional, Multivariate and Univariate Time Series Outlier Ensemble
A compositional multivariate and univariate time series outlier ensemble.It uses the four R packages ...
Download / Learn more Package Citations See dependency  
dmlalg  
Double Machine Learning Algorithms
Implementation of double machine learning (DML) algorithms in R, based on Emmenegger and Buehlmann ...
Download / Learn more Package Citations See dependency  
tropAlgebra  
Tropical Algebraic Functions
It includes functions like tropical addition, tropical multiplication for vectors and matrices. In t ...
Download / Learn more Package Citations See dependency  

23,394

R Packages

201,798

Dependencies

63,416

Author Associations

23,395

Publication Badges

© Copyright 2022 - present. All right reserved, rpkg.net.  Based in Cambridge, Massachusetts, USA