pyspark.pandas.read_delta¶
-
pyspark.pandas.read_delta(path: str, version: Optional[str] = None, timestamp: Optional[str] = None, index_col: Union[str, List[str], None] = None, **options: Any) → pyspark.pandas.frame.DataFrame[source]¶ Read a Delta Lake table on some file system and return a DataFrame.
If the Delta Lake table is already stored in the catalog (aka the metastore), use ‘read_table’.
- Parameters
- pathstring
Path to the Delta Lake table.
- versionstring, optional
Specifies the table version (based on Delta’s internal transaction version) to read from, using Delta’s time travel feature. This sets Delta’s ‘versionAsOf’ option. Note that this paramter and timestamp paramter cannot be used together, otherwise it will raise a ValueError.
- timestampstring, optional
Specifies the table version (based on timestamp) to read from, using Delta’s time travel feature. This must be a valid date or timestamp string in Spark, and sets Delta’s ‘timestampAsOf’ option. Note that this paramter and version paramter cannot be used together, otherwise it will raise a ValueError.
- index_colstr or list of str, optional, default: None
Index column of table in Spark.
- options
Additional options that can be passed onto Delta.
- Returns
- DataFrame
Examples
>>> ps.range(1).to_delta('%s/read_delta/foo' % path) >>> ps.read_delta('%s/read_delta/foo' % path) id 0 0
>>> ps.range(10, 15, num_partitions=1).to_delta('%s/read_delta/foo' % path, ... mode='overwrite') >>> ps.read_delta('%s/read_delta/foo' % path) id 0 10 1 11 2 12 3 13 4 14
>>> ps.read_delta('%s/read_delta/foo' % path, version=0) id 0 0
You can preserve the index in the roundtrip as below.
>>> ps.range(10, 15, num_partitions=1).to_delta( ... '%s/read_delta/bar' % path, index_col="index") >>> ps.read_delta('%s/read_delta/bar' % path, index_col="index") id index 0 10 1 11 2 12 3 13 4 14