SemanticScuttle - klotz.me

Tags: spark* + scala*

0 bookmark(s) - Sort by: Date ↓ / Title /

11. scala - Spark: Add column to dataframe conditionally - Stack Overflow

    val newDf = df.withColumn("D", when($"B".isNull or $"B" === "", 0).otherwise(1))

    2018-04-16 Tags: spark, scala by klotz
12. sparkmagic/Spark Kernel.ipynb at master · jupyter-incubator/sparkmagic

    "executorMemory": "4G", "executorCores":4}

    2017-07-28 Tags: spark, jupyter, hadoop, notebook, sparkmagic, scala, python, sql by klotz
13. How to Extract Nested JSON Data in Spark - Big Datums

    2017-07-06 Tags: json, spark, scala by klotz
14. How to execute external system commands in Scala | alvinalexander.com

    import sys.process._
    "ls -al".!

    2017-06-14 Tags: spark, scala, shell, exec, import by klotz
15. There's a number of Scala libraries using NLP tha...

    2017-05-14 Tags: scala, spark, nlp, reddit by klotz
16. Spark Word2Vec example using text8 file - Stack Overflow

    Unfortunately text8 has had periods stripped out so you can't just split on them, but you can find the raw version here as well as the perl script used to process it, and it isn't hard to edit the script to not remove periods.

    2017-04-21 Tags: spark, scala, word2vec by klotz
17. twitter/chill: Scala extensions for the Kryo serialization library

    2017-04-04 Tags: scala, hadoop, spark, kryo, serialization, java, performance by klotz
18. bloom filter - Spark and BloomFilter sharing - Stack Overflow

    Example Bloom Filter use in Spark 2.0

    2017-04-04 Tags: spark, bloom filter, scala by klotz
19. Explode (transpose?) multiple columns in Spark SQL table - Stack Overflow

    import org.apache.spark.sql.functions.{udf, explode}
    
    val zip = udf((xs: Seq Long » , ys: Seq Long » ) => xs.zip(ys))
    
    df.withColumn("vars", explode(zip_udf($"varA", $"varB"))).select(
    $"userId", $"someString",
    $"vars._1".alias("varA"), $"vars._2".alias("varB")).show

    2017-03-09 Tags: spark, scala, zip, udf by klotz
20. scala - Spark dataframes: Extract a column based on the value of another column - Stack Overflow

    column name indirection

    2017-02-27 Tags: scala, spark, sql, tricks by klotz

Top of the page

First / Previous / Next / Last / Page 2 of 0