Supported Pyspark API

See something that you would like to see supported? Open an issue!

pyspark.sql.Column

• ❌ asc_nulls_first (request feature )
• ❌ asc (request feature )
• ✅ alias
• ❌ between (request feature )
• ❌ asc_nulls_last (request feature )
• ❌ bitwiseAND (request feature )
• ❌ astype (request feature )
• ❌ bitwiseOR (request feature )
• ✅ cast
• ❌ bitwiseXOR (request feature )
• ❌ contains (request feature )
• ❌ desc (request feature )
• ❌ desc_nulls_first (request feature )
• ❌ desc_nulls_last (request feature )
• ❌ dropFields (request feature )
• ❌ endswith (request feature )
• ❌ getField (request feature )
• ❌ getItem (request feature )
• ❌ ilike (request feature )
• ✅ isNotNull
• ❌ isNull (request feature )
• ✅ isin
• ❌ like (request feature )
• ❌ eqNullSafe (request feature )
• ❌ over (request feature )
• ✅ rlike
• ❌ name (request feature )
• ❌ startswith (request feature )
• ❌ substr (request feature )
• ❌ when (request feature )
• ❌ otherwise (request feature )
• ❌ withField (request feature )

pyspark.sql.types

• ❌ ArrayType (request feature )
• ✅ BinaryType
• ✅ ByteType
• ✅ DataType
• ❌ CharType (request feature )
• ✅ DateType
• ✅ DecimalType
• ❌ DayTimeIntervalType (request feature )
• ✅ DoubleType
• ✅ FloatType
• ✅ IntegerType
• ✅ LongType
• ❌ MapType (request feature )
• ❌ NullType (request feature )
• ✅ BooleanType
• ✅ ShortType
• ✅ StructType
• ❌ TimestampNTZType (request feature )
• ✅ TimestampType
• ❌ YearMonthIntervalType (request feature )
• ✅ StringType
• ❌ VarcharType (request feature )

pyspark.sql.DataFrame

• ❌ agg (request feature )
• ✅ alias
• ❌ approxQuantile (request feature )
• ❌ cache (request feature )
• ❌ coalesce (request feature )
• ❌ checkpoint (request feature )
• ❌ colRegex (request feature )
• ❌ collect (request feature )
• ❌ corr (request feature )
• ❌ cov (request feature )
• ❌ count (request feature )
• ❌ createGlobalTempView (request feature )
• ❌ createOrReplaceGlobalTempView (request feature )
• ❌ createOrReplaceTempView (request feature )
• ❌ createTempView (request feature )
• ❌ crossJoin (request feature )
• ❌ cube (request feature )
• ❌ describe (request feature )
• ❌ crosstab (request feature )
• ❌ distinct (request feature )
• ❌ drop (request feature )
• ❌ dropDuplicates (request feature )
• ❌ dropDuplicatesWithinWatermark (request feature )
• ❌ drop_duplicates (request feature )
• ❌ dropna (request feature )
• ❌ exceptAll (request feature )
• ❌ explain (request feature )
• ✅ fillna
• ✅ filter
• ❌ first (request feature )
• ❌ foreachPartition (request feature )
• ❌ freqItems (request feature )
• ❌ foreach (request feature )
• ✅ groupBy
• ❌ head (request feature )
• ❌ hint (request feature )
• ❌ inputFiles (request feature )
• ❌ intersectAll (request feature )
• ❌ isEmpty (request feature )
• ❌ intersect (request feature )
• ❌ isLocal (request feature )
• ✅ join
• ❌ limit (request feature )
• ❌ mapInArrow (request feature )
• ❌ localCheckpoint (request feature )
• ❌ melt (request feature )
• ❌ observe (request feature )
• ❌ mapInPandas (request feature )
• ❌ offset (request feature )
• ✅ orderBy
• ✅ pandas_api
• ❌ persist (request feature )
• ❌ printSchema (request feature )
• ❌ randomSplit (request feature )
• ❌ registerTempTable (request feature )
• ❌ repartitionByRange (request feature )
• ❌ repartition (request feature )
• ❌ replace (request feature )
• ❌ sameSemantics (request feature )
• ❌ rollup (request feature )
• ❌ sampleBy (request feature )
• ❌ sample (request feature )
• ✅ select
• ❌ semanticHash (request feature )
• ❌ selectExpr (request feature )
• ✅ show
• ❌ sortWithinPartitions (request feature )
• ❌ subtract (request feature )
• ❌ summary (request feature )
• ❌ tail (request feature )
• ✅ sort
• ❌ take (request feature )
• ❌ to (request feature )
• ❌ toDF (request feature )
• ❌ toJSON (request feature )
• ❌ toLocalIterator (request feature )
• ✅ toPandas
• ❌ transform (request feature )
• ❌ union (request feature )
• ❌ unionAll (request feature )
• ❌ unionByName (request feature )
• ❌ unpersist (request feature )
• ❌ unpivot (request feature )
• ✅ where
• ✅ withColumn
• ✅ withColumnRenamed
• ❌ withColumns (request feature )
• ❌ withColumnsRenamed (request feature )
• ❌ withMetadata (request feature )
• ❌ withWatermark (request feature )
• ❌ writeTo (request feature )

pyspark.sql.functions

• ✅ abs
• ❌ acos (request feature )
• ❌ acosh (request feature )
• ❌ aes_decrypt (request feature )
• ❌ aes_encrypt (request feature )
• ❌ add_months (request feature )
• ❌ aggregate (request feature )
• ❌ any_value (request feature )
• ❌ approx_count_distinct (request feature )
• ❌ approx_percentile (request feature )
• ❌ array (request feature )
• ❌ array_agg (request feature )
• ❌ array_compact (request feature )
• ❌ array_append (request feature )
• ❌ array_contains (request feature )
• ❌ array_distinct (request feature )
• ❌ array_intersect (request feature )
• ❌ array_insert (request feature )
• ❌ array_join (request feature )
• ❌ array_max (request feature )
• ❌ array_min (request feature )
• ❌ array_position (request feature )
• ❌ array_prepend (request feature )
• ❌ array_remove (request feature )
• ❌ array_repeat (request feature )
• ❌ array_size (request feature )
• ❌ array_union (request feature )
• ❌ arrays_overlap (request feature )
• ❌ arrays_zip (request feature )
• ❌ array_except (request feature )
• ✅ asc_nulls_first
• ✅ asc_nulls_last
• ❌ ascii (request feature )
• ❌ asin (request feature )
• ❌ array_sort (request feature )
• ❌ asinh (request feature )
• ❌ assert_true (request feature )
• ❌ atan (request feature )
• ❌ atan2 (request feature )
• ❌ atanh (request feature )
• ❌ base64 (request feature )
• ✅ asc
• ❌ bit_and (request feature )
• ❌ bit_count (request feature )
• ❌ bit_get (request feature )
• ❌ bit_length (request feature )
• ❌ bit_or (request feature )
• ❌ bit_xor (request feature )
• ❌ avg (request feature )
• ❌ bitmap_bit_position (request feature )
• ❌ bitmap_bucket_number (request feature )
• ❌ bitmap_construct_agg (request feature )
• ❌ bitmap_count (request feature )
• ❌ bitmap_or_agg (request feature )
• ❌ bin (request feature )
• ❌ bitwise_not (request feature )
• ❌ bool_and (request feature )
• ❌ bool_or (request feature )
• ❌ broadcast (request feature )
• ❌ bround (request feature )
• ❌ btrim (request feature )
• ❌ call_function (request feature )
• ❌ call_udf (request feature )
• ❌ cardinality (request feature )
• ❌ cbrt (request feature )
• ❌ ceil (request feature )
• ❌ char (request feature )
• ❌ char_length (request feature )
• ❌ ceiling (request feature )
• ✅ coalesce
• ❌ character_length (request feature )
• ✅ col
• ❌ collect_list (request feature )
• ❌ collect_set (request feature )
• ❌ column (request feature )
• ❌ concat_ws (request feature )
• ❌ concat (request feature )
• ❌ contains (request feature )
• ❌ convert_timezone (request feature )
• ❌ corr (request feature )
• ❌ conv (request feature )
• ❌ cos (request feature )
• ❌ cosh (request feature )
• ❌ cot (request feature )
• ✅ count
• ❌ count_distinct (request feature )
• ❌ count_if (request feature )
• ❌ count_min_sketch (request feature )
• ❌ covar_pop (request feature )
• ❌ covar_samp (request feature )
• ❌ crc32 (request feature )
• ❌ create_map (request feature )
• ❌ csc (request feature )
• ❌ curdate (request feature )
• ❌ current_database (request feature )
• ❌ current_date (request feature )
• ❌ current_schema (request feature )
• ❌ current_timestamp (request feature )
• ❌ current_timezone (request feature )
• ❌ current_user (request feature )
• ❌ date_add (request feature )
• ❌ date_diff (request feature )
• ❌ cume_dist (request feature )
• ❌ date_format (request feature )
• ❌ date_from_unix_date (request feature )
• ❌ date_part (request feature )
• ❌ current_catalog (request feature )
• ❌ date_sub (request feature )
• ❌ date_trunc (request feature )
• ❌ dateadd (request feature )
• ❌ day (request feature )
• ❌ dayofmonth (request feature )
• ❌ dayofweek (request feature )
• ❌ dayofyear (request feature )
• ❌ datediff (request feature )
• ❌ datepart (request feature )
• ❌ decode (request feature )
• ❌ degrees (request feature )
• ✅ dense_rank
• ✅ desc
• ✅ desc_nulls_first
• ✅ desc_nulls_last
• ❌ e (request feature )
• ❌ elt (request feature )
• ❌ encode (request feature )
• ❌ endswith (request feature )
• ❌ equal_null (request feature )
• ❌ every (request feature )
• ❌ exists (request feature )
• ❌ exp (request feature )
• ❌ explode (request feature )
• ❌ expm1 (request feature )
• ❌ expr (request feature )
• ❌ extract (request feature )
• ❌ element_at (request feature )
• ❌ factorial (request feature )
• ❌ filter (request feature )
• ❌ find_in_set (request feature )
• ❌ first (request feature )
• ❌ first_value (request feature )
• ❌ flatten (request feature )
• ❌ floor (request feature )
• ❌ forall (request feature )
• ❌ format_number (request feature )
• ❌ format_string (request feature )
• ❌ explode_outer (request feature )
• ❌ from_csv (request feature )
• ❌ from_json (request feature )
• ❌ from_utc_timestamp (request feature )
• ❌ from_unixtime (request feature )
• ❌ get (request feature )
• ✅ get_json_object
• ❌ getbit (request feature )
• ❌ greatest (request feature )
• ❌ grouping_id (request feature )
• ❌ hash (request feature )
• ❌ grouping (request feature )
• ❌ hex (request feature )
• ❌ histogram_numeric (request feature )
• ❌ hll_sketch_agg (request feature )
• ❌ hll_sketch_estimate (request feature )
• ❌ hll_union (request feature )
• ❌ hll_union_agg (request feature )
• ❌ hour (request feature )
• ❌ hypot (request feature )
• ❌ ifnull (request feature )
• ❌ ilike (request feature )
• ❌ initcap (request feature )
• ❌ inline (request feature )
• ❌ inline_outer (request feature )
• ❌ input_file_block_length (request feature )
• ❌ input_file_block_start (request feature )
• ❌ input_file_name (request feature )
• ❌ instr (request feature )
• ❌ isnan (request feature )
• ❌ isnotnull (request feature )
• ❌ isnull (request feature )
• ❌ json_array_length (request feature )
• ❌ java_method (request feature )
• ❌ kurtosis (request feature )
• ❌ json_tuple (request feature )
• ❌ lag (request feature )
• ❌ last (request feature )
• ❌ last_value (request feature )
• ❌ lcase (request feature )
• ❌ lead (request feature )
• ❌ last_day (request feature )
• ❌ least (request feature )
• ❌ left (request feature )
• ✅ length
• ❌ levenshtein (request feature )
• ❌ like (request feature )
• ✅ lit
• ❌ ln (request feature )
• ❌ localtimestamp (request feature )
• ❌ locate (request feature )
• ❌ log (request feature )
• ❌ json_object_keys (request feature )
• ❌ log10 (request feature )
• ❌ log1p (request feature )
• ❌ lower (request feature )
• ❌ log2 (request feature )
• ❌ lpad (request feature )
• ❌ make_date (request feature )
• ❌ ltrim (request feature )
• ❌ make_dt_interval (request feature )
• ❌ make_timestamp (request feature )
• ❌ make_interval (request feature )
• ❌ make_timestamp_ltz (request feature )
• ❌ make_ym_interval (request feature )
• ❌ make_timestamp_ntz (request feature )
• ❌ map_concat (request feature )
• ❌ map_contains_key (request feature )
• ❌ map_filter (request feature )
• ❌ map_from_arrays (request feature )
• ❌ map_from_entries (request feature )
• ❌ map_keys (request feature )
• ❌ map_values (request feature )
• ❌ map_zip_with (request feature )
• ❌ mask (request feature )
• ✅ max
• ❌ max_by (request feature )
• ❌ map_entries (request feature )
• ❌ median (request feature )
• ✅ min
• ❌ md5 (request feature )
• ❌ min_by (request feature )
• ❌ minute (request feature )
• ❌ mode (request feature )
• ❌ monotonically_increasing_id (request feature )
• ❌ month (request feature )
• ❌ months_between (request feature )
• ❌ named_struct (request feature )
• ❌ nanvl (request feature )
• ❌ negative (request feature )
• ❌ negate (request feature )
• ❌ now (request feature )
• ❌ next_day (request feature )
• ❌ nth_value (request feature )
• ❌ ntile (request feature )
• ❌ nullif (request feature )
• ❌ nvl (request feature )
• ❌ nvl2 (request feature )
• ❌ octet_length (request feature )
• ❌ overlay (request feature )
• ❌ pandas_udf (request feature )
• ❌ parse_url (request feature )
• ❌ percent_rank (request feature )
• ❌ percentile (request feature )
• ❌ percentile_approx (request feature )
• ❌ pi (request feature )
• ❌ pmod (request feature )
• ❌ posexplode (request feature )
• ❌ posexplode_outer (request feature )
• ✅ mean
• ❌ position (request feature )
• ❌ positive (request feature )
• ❌ pow (request feature )
• ❌ power (request feature )
• ❌ printf (request feature )
• ❌ quarter (request feature )
• ❌ product (request feature )
• ❌ radians (request feature )
• ❌ rand (request feature )
• ❌ raise_error (request feature )
• ❌ randn (request feature )
• ✅ rank
• ❌ reduce (request feature )
• ❌ regexp_count (request feature )
• ❌ regexp (request feature )
• ❌ regexp_extract_all (request feature )
• ❌ regexp_extract (request feature )
• ❌ regexp_instr (request feature )
• ❌ regexp_like (request feature )
• ✅ regexp_replace
• ❌ regexp_substr (request feature )
• ❌ regr_avgx (request feature )
• ❌ regr_avgy (request feature )
• ❌ regr_count (request feature )
• ❌ regr_intercept (request feature )
• ❌ regr_r2 (request feature )
• ❌ regr_slope (request feature )
• ❌ regr_sxx (request feature )
• ❌ regr_sxy (request feature )
• ❌ regr_syy (request feature )
• ❌ repeat (request feature )
• ❌ replace (request feature )
• ❌ reflect (request feature )
• ❌ reverse (request feature )
• ❌ right (request feature )
• ❌ rlike (request feature )
• ❌ rint (request feature )
• ❌ rpad (request feature )
• ✅ row_number
• ❌ rtrim (request feature )
• ❌ schema_of_csv (request feature )
• ❌ schema_of_json (request feature )
• ❌ sec (request feature )
• ❌ second (request feature )
• ❌ sentences (request feature )
• ❌ sequence (request feature )
• ❌ session_window (request feature )
• ❌ sha (request feature )
• ❌ sha1 (request feature )
• ❌ sha2 (request feature )
• ✅ round
• ❌ shiftleft (request feature )
• ❌ shiftright (request feature )
• ❌ shiftrightunsigned (request feature )
• ❌ sign (request feature )
• ❌ shuffle (request feature )
• ❌ signum (request feature )
• ❌ sinh (request feature )
• ❌ sin (request feature )
• ❌ size (request feature )
• ❌ slice (request feature )
• ❌ some (request feature )
• ❌ sort_array (request feature )
• ❌ soundex (request feature )
• ❌ spark_partition_id (request feature )
• ❌ split (request feature )
• ❌ sqrt (request feature )
• ❌ split_part (request feature )
• ❌ stack (request feature )
• ❌ startswith (request feature )
• ❌ std (request feature )
• ❌ stddev (request feature )
• ❌ stddev_pop (request feature )
• ❌ stddev_samp (request feature )
• ❌ str_to_map (request feature )
• ❌ struct (request feature )
• ❌ substr (request feature )
• ❌ skewness (request feature )
• ❌ substring (request feature )
• ✅ sum
• ❌ sum_distinct (request feature )
• ❌ tan (request feature )
• ❌ tanh (request feature )
• ❌ timestamp_micros (request feature )
• ❌ timestamp_millis (request feature )
• ❌ timestamp_seconds (request feature )
• ❌ to_binary (request feature )
• ❌ to_char (request feature )
• ❌ to_csv (request feature )
• ❌ to_date (request feature )
• ❌ substring_index (request feature )
• ❌ to_json (request feature )
• ❌ to_number (request feature )
• ✅ to_timestamp
• ❌ to_timestamp_ltz (request feature )
• ❌ to_unix_timestamp (request feature )
• ❌ to_utc_timestamp (request feature )
• ❌ transform (request feature )
• ❌ to_varchar (request feature )
• ❌ transform_values (request feature )
• ❌ translate (request feature )
• ❌ trim (request feature )
• ❌ trunc (request feature )
• ❌ try_add (request feature )
• ❌ to_timestamp_ntz (request feature )
• ❌ try_aes_decrypt (request feature )
• ❌ try_avg (request feature )
• ❌ try_element_at (request feature )
• ❌ try_divide (request feature )
• ❌ try_multiply (request feature )
• ❌ transform_keys (request feature )
• ❌ try_subtract (request feature )
• ❌ try_sum (request feature )
• ❌ try_to_number (request feature )
• ❌ try_to_binary (request feature )
• ❌ typeof (request feature )
• ❌ ucase (request feature )
• ❌ udf (request feature )
• ❌ udtf (request feature )
• ❌ unbase64 (request feature )
• ❌ unhex (request feature )
• ❌ unix_date (request feature )
• ❌ unix_micros (request feature )
• ❌ unix_millis (request feature )
• ❌ unix_seconds (request feature )
• ❌ unwrap_udt (request feature )
• ❌ upper (request feature )
• ❌ try_to_timestamp (request feature )
• ❌ url_encode (request feature )
• ❌ user (request feature )
• ❌ var_pop (request feature )
• ❌ var_samp (request feature )
• ❌ variance (request feature )
• ❌ version (request feature )
• ❌ unix_timestamp (request feature )
• ❌ weekday (request feature )
• ❌ weekofyear (request feature )
• ✅ when
• ❌ width_bucket (request feature )
• ❌ window (request feature )
• ❌ window_time (request feature )
• ❌ url_decode (request feature )
• ❌ xpath (request feature )
• ❌ xpath_boolean (request feature )
• ❌ xpath_double (request feature )
• ❌ xpath_float (request feature )
• ❌ xpath_int (request feature )
• ❌ xpath_long (request feature )
• ❌ xpath_number (request feature )
• ❌ xpath_short (request feature )
• ❌ xpath_string (request feature )
• ❌ xxhash64 (request feature )
• ❌ year (request feature )
• ❌ zip_with (request feature )

pyspark.sql.GroupedData

• ✅ agg
• ❌ applyInPandas (request feature )
• ❌ apply (request feature )
• ❌ applyInPandasWithState (request feature )
• ❌ avg (request feature )
• ❌ cogroup (request feature )
• ✅ count
• ✅ mean
• ✅ min
• ❌ pivot (request feature )
• ✅ max
• ✅ sum

pyspark.sql.SparkSession

• ❌ addArtifact (request feature )
• ❌ addTag (request feature )
• ❌ clearTags (request feature )
• ❌ copyFromLocalToFs (request feature )
• ✅ createDataFrame
• ❌ getTags (request feature )
• ❌ interruptAll (request feature )
• ❌ interruptOperation (request feature )
• ❌ interruptTag (request feature )
• ❌ newSession (request feature )
• ❌ range (request feature )
• ❌ removeTag (request feature )
• ❌ sql (request feature )
• ❌ stop (request feature )
• ❌ table (request feature )
• ❌ addArtifacts (request feature )

pyspark.sql.Window

• ❌ rangeBetween (request feature )
• ✅ partitionBy
• ✅ orderBy
• ❌ rowsBetween (request feature )