All Questions
8 questions
4
votes
0
answers
808
views
Oracle Column type Number is showing Decimal value in spark
Using spark read jdbc option , i am reading oracle table and one of the column type is 'Number' type. After reading and writing into s3 bucket, dataframe printschema is showing decimal(38,10). I know ...
- 3,060
2
votes
0
answers
244
views
Error loading spark sql context for redshift jdbc url in glue
Hello I am trying to fetch month-wise data from a bunch of heavy redshift table(s) in glue job.
As far as I know glue documentation on this is very limited.
The query works fine in SQL Workbench which ...
- 1,467
0
votes
0
answers
655
views
How do I fix a driver error coming from writing a Dataset to redshift using scala spark?
def streamDSToRedshiftOutputIO[TIMELINE <: TimeLine : Encoder](tableName: String)(batchIntervalSeconds: Int)(dataset: Dataset[TIMELINE]): Unit = {
dataset
.writeStream
.foreachBatch(...
- 3,277
1
vote
1
answer
4k
views
Reading from Redshift into Spark Dataframe (Spark-Redshift Module)
I'm following along the spark-redshift tutorial to read from redshift into spark (databricks). I have the following code:
val tempDir = "s3n://{my-s3-bucket-here}"
val jdbcUsername = "usernameExample"...
0
votes
1
answer
3k
views
Pyspark - configure Amazon Redshift JDBC jar
I am trying to use the spark-redshift databricks package and cannot get the Redshift jdbc driver working correctly. I have downloaded the latest version from here and saved to an s3 bucket.
This is ...
- 455
0
votes
0
answers
1k
views
SQLFeatureNotSupportedException on Amazon Redshift
I am trying to run some ETL process on Amazon Redshift. It's written in Apache Spark. Same code works fine on Postgres but with Redshift is throwing SQLFeatureNotSupportedException: [Amazon][JDBC](...
- 13.6k
1
vote
1
answer
3k
views
spark and aws redshift: java.sql.SQLException: No suitable driver found for jdbc:redshift://xxx.us-west-2.redshift.amazonaws.com:5439
os: centos
spark:1.6.1
sbt: build.sbt
libraryDependencies ++= {
Seq(
"org.apache.spark" %% "spark-core" % "1.6.1" % "provided",
"com.amazonaws" % "aws-java-sdk" % "1.10.75",
"com....
- 2,318
2
votes
1
answer
3k
views
AWS Redshift driver in Zeppelin
I want to explore my data in Redshift using notebook Zeppelin. A small EMR cluster with Spark is running behind. I am loading databricks' spark-redshift library
%dep
z.reset()
z.load("com.databricks:...
- 1,143