Create new dataset from specific columns of other 2 datasets in Scala

zubug55

I have the following datasets with 2 different schemas.

case class schema1(a: Double, b: Double) -> dataset1
case class schema2(c: Double, d: Double, e: Double, f: Double) -> dataset2

I want to create another dataset with the following schema:

case class schema3(c: Double,  b: Double) -> dataset3

That is, the Mode 3 dataset contains the first column c in the Mode 2 dataset and the second column b in the Mode 1 dataset.

How to create a third dataset based on schema3 using the data in columns c and b of dataset2 and dataset1.

Or more simply, I have to create a third dataset by taking one column from the first dataset and another column from the second dataset and mapping it to the third schema defined above.

please help.

Srinivas

Use monotonically_increasing_id& row_numerto add unique id values in both datasets and join the two datasets using idcolumn and the desired columns in both datasets and finally remove the id from the resulting dataset.

Please check the code below.

scala> case class schema1(a: Double, b: Double)
defined class schema1

scala> case class schema2(c: Double, d: Double, e: Double, f: Double)
defined class schema2

scala> import org.apache.spark.sql.expressions._
import org.apache.spark.sql.expressions._

scala> val sa = Seq(schema1(11,12),schema1(22,23)).toDF.withColumn("id",monotonically_increasing_id).withColumn("id",row_number().over(Window.orderBy("id")))
sa: org.apache.spark.sql.DataFrame = [a: double, b: double ... 1 more field]

scala> val sb = Seq(schema2(22,23,24,25),schema2(32,33,34,35),schema2(132,133,134,135)).toDF.withColumn("id",monotonically_increasing_id).withColumn("id",row_number().over(Window.orderBy("id")))
sb: org.apache.spark.sql.DataFrame = [c: double, d: double ... 3 more fields]

scala> sa.show(false)
+----+----+---+
|a   |b   |id |
+----+----+---+
|11.0|12.0|0  |
|22.0|23.0|1  |
+----+----+---+


scala> sb.show(false)
+-----+-----+-----+-----+---+
|c    |d    |e    |f    |id |
+-----+-----+-----+-----+---+
|22.0 |23.0 |24.0 |25.0 |0  |
|32.0 |33.0 |34.0 |35.0 |1  |
|132.0|133.0|134.0|135.0|2  |
+-----+-----+-----+-----+---+

scala> sb.select("c","id").join(sa.select("b","id"),Seq("id"),"full").drop("id").show(false)
+-----+----+
|c    |b   |
+-----+----+
|22.0 |12.0|
|32.0 |23.0|
|132.0|null|
+-----+----+

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Lynette I have two datasets: df1 and df2, I want the columns from df1 to be merged together if the datetime is within 20 seconds of df2 df1 Connect Ended 4/6/2020 1:15:21 PM 4/6/2020 2:05:18 PM 3/31/2020 11:00:08 AM 3/31/2020 11:0

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Create variables from other values in other datasets

es_dutch I want to create a variable that counts each party seat for each municipality, so I get the following: df <- data.frame( stringsAsFactors = FALSE, municipality= c("Aa en Hunze","Aa en Hunze", "Aa en Hunze","Aalburg",

Create a new column at a specific location by summing other columns by row

Yes I want to transform this file by adding two columns to accumulate the values into another column. I want this new column to be next to the corresponding summed column. A B C 2000 33000 2 2000 153000 1 2000 178000 1 2000 225000

Subset large dataset with strings in two columns from other datasets

Shaxi Liver In this case, our large dataset would look like this: structure(list(Car = c("Mazda RX4", "Maserati Bora", "Leticia", "Hornet 4 Drive", "Hornet Sportabout", "Alex", "Duster 360", "Merc 240D", "Merc 23

Create a new specific value column from other columns

new 123 I have county data, but I want to create columns that only list states. Basically, I have this: County County 1, NY County 2, NY County 3, NY County 4, TX County 5, TX County 6, IL County 7, IL But I want this: County State Coun

I have two datasets and need to merge strings from one dataset column with other dataset columns in R

user 3061575 I have two datasets and need to merge strings from one dataset column with other dataset columns in R: Below are the details. case can be ignored Can anyone help me with this problem. First dataset: <table><tbody><tr><th>instancename</th><th>hostn

Create a dataset from two different datasets

Sanghamitra Lahiri I have two different datasets: 1. state VDM MDM OM AP 1 2 5 GOA 1 2 1 GU 1 2 4 KA 1 5 1 2. Attribute:Value Support Item VDM:1 4 1 VDM:2

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Create a new column at a specific location by summing other columns by row

Create variables from other values in other datasets

Create a new specific value column from other columns

R - compare two columns from different datasets and get new dataset

Mica I have two datasets. For each dataset, every two columns is a pair, the *afirst column and *bthe second column. For example df1, for A1, df1$A1a= first column & df1$A1b= second column. Likewise, df2, for B1, df2$B1a= first column & df2$B1b= second column.

Create a new column with values from other columns in the dataset

Lily I want to create a column that contains information from one or the other column in my dataset, depending on the third column. My dataset is full of 0 and 1 values like this: df <- data.frame(PatientID = c("0002" ,"0004", "0005", "0006" ,"0009" ,"0010" ,"

Create new dataset from specific columns of other 2 datasets in Scala

Related

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Create variables from other values in other datasets

Create a new column at a specific location by summing other columns by row

Subset large dataset with strings in two columns from other datasets

Create a new specific value column from other columns

I have two datasets and need to merge strings from one dataset column with other dataset columns in R

Create a dataset from two different datasets

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Create a new column at a specific location by summing other columns by row

Create variables from other values in other datasets

Create a new specific value column from other columns

R - compare two columns from different datasets and get new dataset

Create a new column with values from other columns in the dataset

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Create a new specific value column from other columns

Create a new specific value column from other columns

Create a new column at a specific location by summing other columns by row

Subset large dataset with strings in two columns from other datasets

Create a dataset from two different datasets

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Concatenate two datasets in Python and create a "mismatch" dataset if the datetimes are within a specific time of each other

Create a new specific value column from other columns

Create a new specific value column from other columns

Create a new specific value column from other columns

Create a new column at a specific location by summing other columns by row

Create a new column at a specific location by summing other columns by row

Create a new column at a specific location by summing other columns by row

Ranking