com.splicemachine.spark.splicemachine

SplicemachineContext

Related Doc: package splicemachine

class SplicemachineContext extends Serializable

Context for Splice Machine.

Linear Supertypes
Serializable, Serializable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By inheritance
Inherited
  1. SplicemachineContext
  2. Serializable
  3. Serializable
  4. AnyRef
  5. Any
  1. Hide All
  2. Show all
Learn more about member selection
Visibility
  1. Public
  2. All

Instance Constructors

  1. new SplicemachineContext(url: String)

    Context for Splice Machine, specifying only the JDBC url.

    Context for Splice Machine, specifying only the JDBC url.

    url

    JDBC Url with authentication parameters

  2. new SplicemachineContext(options: Map[String, String])

    options

    Supported options are SpliceJDBCOptions.JDBC_URL, SpliceJDBCOptions.JDBC_INTERNAL_QUERIES, SpliceJDBCOptions.JDBC_TEMP_DIRECTORY

Value Members

  1. final def !=(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Definition Classes
    AnyRef → Any
  3. final def ==(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  4. def analyzeSchema(schemaName: String): Unit

    Analyze the scheme supplied via JDBC

    Analyze the scheme supplied via JDBC

    schemaName

  5. def analyzeTable(tableName: String, estimateStatistics: Boolean = false, samplePercent: Double = 0.10): Unit

    Analyze table provided.

    Analyze table provided. Will use estimate statistics if provided.

    tableName
    estimateStatistics
    samplePercent

  6. final def asInstanceOf[T0]: T0

    Definition Classes
    Any
  7. def bulkImportHFile(rdd: JavaRDD[Row], schema: StructType, schemaTableName: String, options: Map[String, String]): Unit

    Bulk Import HFile from a RDD into a schemaTableName(schema.table)

    Bulk Import HFile from a RDD into a schemaTableName(schema.table)

    rdd

    input data

    schemaTableName
    options

    options to be passed to --splice-properties; bulkImportDirectory is required

  8. def bulkImportHFile(dataFrame: DataFrame, schemaTableName: String, options: Map[String, String]): Unit

    Bulk Import HFile from a dataframe into a schemaTableName(schema.table)

    Bulk Import HFile from a dataframe into a schemaTableName(schema.table)

    dataFrame

    input data

    schemaTableName
    options

    options to be passed to --splice-properties; bulkImportDirectory is required

  9. def clone(): AnyRef

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  10. def createTable(tableName: String, structType: StructType, keys: Seq[String], createTableOptions: String): Unit

    Create Table based on the table name, the struct (data types), key sequence and createTableOptions.

    Create Table based on the table name, the struct (data types), key sequence and createTableOptions.

    We currently do not have any custom table options. These could be added if needed.

    tableName
    structType
    keys
    createTableOptions

  11. def delete(rdd: JavaRDD[Row], schema: StructType, schemaTableName: String): Unit

    Delete records in a dataframe based on joining by primary keys from the data frame.

    Delete records in a dataframe based on joining by primary keys from the data frame. Be careful with column naming and case sensitivity.

    rdd

    rows to delete

    schema
    schemaTableName

    table to delete from

  12. def delete(dataFrame: DataFrame, schemaTableName: String): Unit

    Delete records in a dataframe based on joining by primary keys from the data frame.

    Delete records in a dataframe based on joining by primary keys from the data frame. Be careful with column naming and case sensitivity.

    dataFrame

    rows to delete

    schemaTableName

    table to delete from

  13. def df(sql: String): Dataset[Row]

    SQL to Dataset translation.

    SQL to Dataset translation. (Lazy)

    sql

    SQL query

    returns

    Dataset[Row] with the result of the query

  14. def dropTable(schemaTableName: String): Unit

    Drop a table based on the schemaTableName (schema.table)

    Drop a table based on the schemaTableName (schema.table)

    schemaTableName

  15. def dropTable(schemaName: String, tableName: String): Unit

    Drop a table based on the schema name and table name.

    Drop a table based on the schema name and table name.

    schemaName
    tableName

  16. final def eq(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  17. def equals(arg0: Any): Boolean

    Definition Classes
    AnyRef → Any
  18. def execute(sql: String): Unit

    Execute SQL against Splice Machine via JDBC.

    Execute SQL against Splice Machine via JDBC.

    sql

  19. def executeUpdate(sql: String): Unit

    Execute an update statement via JDBC against Splice Machine

    Execute an update statement via JDBC against Splice Machine

    sql

  20. def export(dataFrame: DataFrame, location: String, compression: Boolean, replicationCount: Int, fileEncoding: String, fieldSeparator: String, quoteCharacter: String): Unit

    Export a dataFrame in CSV

    Export a dataFrame in CSV

    location

    - Destination directory

    compression

    - Whether to compress the output or not

    replicationCount

    - Replication used for HDFS write

    fileEncoding

    - fileEncoding or null, defaults to UTF-8

    fieldSeparator

    - fieldSeparator or null, defaults to ','

    quoteCharacter

    - quoteCharacter or null, defaults to '"'

  21. def exportBinary(dataFrame: DataFrame, location: String, compression: Boolean, format: String): Unit

    Export a dataFrame in binary format

    Export a dataFrame in binary format

    location

    - Destination directory

    compression

    - Whether to compress the output or not

    format

    - Binary format to be used, currently only 'parquet' is supported

  22. def finalize(): Unit

    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  23. final def getClass(): Class[_]

    Definition Classes
    AnyRef → Any
  24. def getConnection(): Connection

    Get internal JDBC connection

  25. def getSchema(schemaTableName: String): StructType

    Return a table's schema via JDBC.

    Return a table's schema via JDBC.

    schemaTableName

    table

    returns

    table's schema

  26. def hashCode(): Int

    Definition Classes
    AnyRef → Any
  27. def insert(rdd: JavaRDD[Row], schema: StructType, schemaTableName: String, statusDirectory: String, badRecordsAllowed: Integer): Unit

    Insert a RDD into a table (schema.table).

    Insert a RDD into a table (schema.table). The schema is required since RDD's do not have schema.

    The status directory and number of badRecordsAllowed allows for duplicate primary keys to be written to a bad records file. If badRecordsAllowed is set to -1, all bad records will be written to the status directory.

    rdd

    input data

    schema
    schemaTableName
    statusDirectory

    status directory where bad records file will be created

    badRecordsAllowed

    how many bad records are allowed. -1 for unlimited

  28. def insert(dataFrame: DataFrame, schemaTableName: String, statusDirectory: String, badRecordsAllowed: Integer): Unit

    Insert a dataFrame into a table (schema.table).

    Insert a dataFrame into a table (schema.table). This corresponds to an

    insert into from select statement

    The status directory and number of badRecordsAllowed allows for duplicate primary keys to be written to a bad records file. If badRecordsAllowed is set to -1, all bad records will be written to the status directory.

    dataFrame

    input data

    schemaTableName
    statusDirectory

    status directory where bad records file will be created

    badRecordsAllowed

    how many bad records are allowed. -1 for unlimited

  29. def insert(rdd: JavaRDD[Row], schema: StructType, schemaTableName: String): Unit

    Insert a RDD into a table (schema.table).

    Insert a RDD into a table (schema.table). The schema is required since RDD's do not have schema.

    rdd

    input data

    schema
    schemaTableName

  30. def insert(dataFrame: DataFrame, schemaTableName: String): Unit

    Insert a dataFrame into a table (schema.table).

    Insert a dataFrame into a table (schema.table). This corresponds to an

    insert into from select statement

    dataFrame

    input data

    schemaTableName

    output table

  31. def internalDf(sql: String): Dataset[Row]

    SQL to Dataset translation.

    SQL to Dataset translation. (Lazy) Runs the query inside Splice Machine and sends the results to the Spark Adapter app

    sql

    SQL query

    returns

    Dataset[Row] with the result of the query

  32. def internalRdd(schemaTableName: String, columnProjection: Seq[String] = Nil): RDD[Row]

    Table with projections in Splice mapped to an RDD.

    Table with projections in Splice mapped to an RDD. Runs the query inside Splice Machine and sends the results to the Spark Adapter app

    schemaTableName

    Accessed table

    columnProjection

    Selected columns

    returns

    RDD[Row] with the result of the projection

  33. final def isInstanceOf[T0]: Boolean

    Definition Classes
    Any
  34. final def ne(arg0: AnyRef): Boolean

    Definition Classes
    AnyRef
  35. final def notify(): Unit

    Definition Classes
    AnyRef
  36. final def notifyAll(): Unit

    Definition Classes
    AnyRef
  37. def pruneSchema(schema: StructType, columns: Array[String]): StructType

    Prune all but the specified columns from the specified Catalyst schema.

    Prune all but the specified columns from the specified Catalyst schema.

    schema

    - The Catalyst schema of the master table

    columns

    - The list of desired columns

    returns

    A Catalyst schema corresponding to columns in the given order.

  38. def rdd(schemaTableName: String, columnProjection: Seq[String] = Nil): RDD[Row]

    Table with projections in Splice mapped to an RDD.

    Table with projections in Splice mapped to an RDD.

    schemaTableName

    Accessed table

    columnProjection

    Selected columns

    returns

    RDD[Row] with the result of the projection

  39. def schemaString(schema: StructType, url: String): String

    Generate the schema string for create table.

    Generate the schema string for create table.

    schema
    url
    returns

  40. final def synchronized[T0](arg0: ⇒ T0): T0

    Definition Classes
    AnyRef
  41. def tableExists(schemaName: String, tableName: String): Boolean

    Determine whether a table exists given the schema name and table name.

    Determine whether a table exists given the schema name and table name.

    schemaName
    tableName
    returns

    true if the table exists, false otherwise

  42. def tableExists(schemaTableName: String): Boolean

    Determine whether a table exists (uses JDBC).

    Determine whether a table exists (uses JDBC).

    schemaTableName
    returns

    true if the table exists, false otherwise

  43. def toString(): String

    Definition Classes
    AnyRef → Any
  44. def truncateTable(tableName: String): Unit

    Truncate the table supplied.

    Truncate the table supplied. The tableName should be in the form schema.table

    tableName

  45. def update(rdd: JavaRDD[Row], schema: StructType, schemaTableName: String): Unit

    Update data from a RDD for a specified schemaTableName (schema.table) and schema (StructType).

    Update data from a RDD for a specified schemaTableName (schema.table) and schema (StructType). The keys are required for the update and any other columns provided will be updated in the rows.

    rdd

    rows for update

    schema
    schemaTableName

  46. def update(dataFrame: DataFrame, schemaTableName: String): Unit

    Update data from a dataframe for a specified schemaTableName (schema.table).

    Update data from a dataframe for a specified schemaTableName (schema.table). The keys are required for the update and any other columns provided will be updated in the rows.

    dataFrame

    rows for update

    schemaTableName

    table to update

  47. def upsert(rdd: JavaRDD[Row], schema: StructType, schemaTableName: String): Unit

    Upsert data into the table (schema.table) from an RDD.

    Upsert data into the table (schema.table) from an RDD. This will insert the data if the record is not found by primary key and if it is it will change the columns that are different between the two records.

    rdd

    input data

    schema
    schemaTableName

  48. def upsert(dataFrame: DataFrame, schemaTableName: String): Unit

    Upsert data into the table (schema.table) from a DataFrame.

    Upsert data into the table (schema.table) from a DataFrame. This will insert the data if the record is not found by primary key and if it is it will change the columns that are different between the two records.

    dataFrame

    input data

    schemaTableName

    output table

  49. final def wait(): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  50. final def wait(arg0: Long, arg1: Int): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  51. final def wait(arg0: Long): Unit

    Definition Classes
    AnyRef
    Annotations
    @throws( ... )

Inherited from Serializable

Inherited from Serializable

Inherited from AnyRef

Inherited from Any

Ungrouped