Please check here to write pandas dataframe as parquet

df.to_parquet('df.parquet.gzip',
              compression='gzip') 

Most probably your DataFrame is the Pandas DataFrame object, not Spark DataFrame object.

try:

Copyspark.createDataFrame(df).write.saveAsTable("dashboardco.AccountList")

As you would have already guessed, you can fix the code by removing .schema(my_schema) like below

my_spark_df.write.format("delta").save(my_path)

I think you are confused where does the schema apply, you need to create a dataframe with the schema(use some dummy Seq or rdd), and during that point you need to mention the schema. While you call DataFrameWriter there is no option to provide schema, it infers the schema of the dataframe on which the writer API is called.

You could take your initial dataframe alter its schema like below and use this intermediate dataframe for the write api call

 df.withColumn("new_column_name",$"old_column_name".cast("new_datatype"))

The syntax is valid with Pandas DataFrames but that attribute doesn't exist for the PySpark created DataFrames. You can check out this link for the documentation.

Usually, the collect() method or the .rdd attribute would help you with these tasks.

You can use the following snippet to produce the desired result:

http_path = sdf.rdd.map(lambda row: row['http_path'].split('?'))
api_param_df = pd.DataFrame([[row[0], np.nan] if len(row) == 1 else row for row in http_path.collect()], columns=["api", "param"])
sdf = pd.concat([sdf.toPandas()['raw'], api_param_df], axis=1)

Note that I removed the comments to make it more readable and I've also substituted the regex with a simple split.