SemanticScuttle - klotz.me

Tags: spark*

Spark is an open-source, distributed computing framework for large-scale data processing, originally developed by the UC Berkeley AmpLab It is designed to be fast and general enough to handle a wide variety of workloads, including ETL, machine learning, streaming, and graph processing. It is built on top of Hadoop, Yarn, or other substrates and provides a programming interface for programming with an ecosystem of libraries for machine learning, graph processing, and streaming. Spark is used in cloud engineering and machine learning science for its ability to process large amounts of data quickly and efficiently. It is written in Scala, and can be used with Python, Java, and R for production-level applications. It integrates with Kubernetes and cloud providers for scalability and management.

0 bookmark(s) - Sort by: Date ↓ / Title /

11. Running Timeseries Anomaly Detection at Scale on SQL Data | by Sachin Bansal | Aug, 2021 | Towards Data Science

    2021-08-28 Tags: time series, anomaly detection, facebook prophet, spark, python by klotz
12. apache spark - Adding a group count column to a PySpark dataframe - Stack Overflow

    2021-04-21 Tags: spark, pyspark, groupby, count, window, .spark sql by klotz
13. sql - percentage count per group and pivot with pyspark - Stack Overflow

    2021-04-21 Tags: spark, pyspark, pivot by klotz
14. PySpark Pivot and Unpivot DataFrame — SparkByExamples

    2021-04-21 Tags: spark, pyspark, pivot by klotz
15. Analyzing Apache access logs with Spark and Scala (a tutorial) | alvinalexander.com

    Apache logfile parser with Spark

    2021-04-01 Tags: apache, logfile, parser, spark, alvin alexander by klotz
16. ScalaApacheAccessLogParser/AccessLogParser.scala at master · alvinj/ScalaApacheAccessLogParser · GitHub

    2021-04-01 Tags: clf, apache, log, parse, spark, scala, alvin alexander by klotz
17. Spark RDDs Vs DataFrames vs SparkSQL – Part 3 : Web Server Log Analysis | DataScience+

    Extract the 11 elements from each log
    
    def map_log(line):
    match = re.search('^(S+) (S+) (S+) (S+) [- » (d{4})] "(S+)s*(S+)s*(S+)s*(+)?s*"* (d{3}) (S+)',line)
    if match is None:
    match = re.search('^(S+) (S+) (S+) (S+) [- » (d{4})] "(S+)s*(+)>*( w/s. » +)s*(S+)s*(d{3})s*(S+)',line)
    return(match.groups())
    parsed_rdd = rdd.map(lambda line: parse_log2(line)).filter(lambda line: line 1 » == 1).map(lambda line : line 0 » )
    parsed_rdd2 = parsed_rdd.map(lambda line: map_log(line))

    2021-04-01 Tags: spark, apache, rdd, cdf, logs, pyspark by klotz
18. How to analyze log data with Python and Apache Spark | Opensource.com

    2021-04-01 Tags: spark, log, apache, pyspark, data analytics by klotz
19. How to Write Spark UDFs (User Defined Functions) in Python – BMC Software | Blogs

    2021-03-18 Tags: pyspark, spark, udf, python by klotz
20. Apache Spark — Multi-part Series: Spark Architecture | by Luke Thorp | Mar, 2021 | Towards Data Science

    2021-03-17 Tags: spark, architecture by klotz

Top of the page

First / Previous / Next / Last / Page 2 of 0