Hi all

I'm doing a course on Udemy so this code probably looks familiar to some of you. I'm trying to get a list of prices from the original dataset (shape is (506,), and convert it to log prices.

Converting it to log prices results in the same (506,0) shape, but when I try to convert this into a DataFrame, the shape ends up being (0,1) so I have issues later down the line trying to merge this with another data frame that's shaped (506,11).

Someone in course comments mentioned that using np.log1p instead of np.log will fix the issue, but unfortunately I'm having the same issue.

Any help would be greatly appreciated. I've tried asking in the course comments, and even tried google and troubleshooting with ChatGPT but no dice 🥲

Code:

# Gather Data
data = pd.DataFrame(data=boston_dataset.data, columns=boston_dataset.feature_names)
data.head()
features = data.drop(['INDUS', 'AGE'], axis=1)

# Check the shape of the target values in the dataset
print("Shape of boston_dataset.target:", boston_dataset.target.shape)

# Convert to log prices
#log_prices = np.log(boston_dataset.target)
log_prices = np.log1p(boston_dataset.target)

# Ensure log_prices has the correct shape
print("Shape of log_prices:", log_prices.shape)
print("First 5 elements of log_prices:", log_prices[:5])

# Create a new DataFrame for the target prices
target = pd.DataFrame(log_prices, columns=['PRICE'])

# Check the shape and contents of the target DataFrame
print("Target DataFrame shape:", target.shape)
print("First 5 rows of target DataFrame:\n", target.head())

# Check if the DataFrames are empty
if features.empty:
    print("Error: Features DataFrame is empty")
if target.empty:
    print("Error: Target DataFrame is empty")

# If 'CHAS' is categorical, convert it to numeric
if features['CHAS'].dtype == 'category':
    features['CHAS'] = features['CHAS'].astype('int')

# Convert RAD to integer
if features['RAD'].dtype == 'category':
    features['RAD'] = features['RAD'].astype('int')

The outputs for the above print statements are:

Shape of boston_dataset.target: (506,)
Shape of log_prices: (506,)
First 5 elements of log_prices: 0    3.218876
1    3.117950
2    3.575151
3    3.538057
4    3.616309
Name: MEDV, dtype: float64
Target DataFrame shape: (0, 1)
First 5 rows of target DataFrame:
 Empty DataFrame
Columns: [PRICE]
Index: []
Error: Target DataFrame is empty

Hey

I am pretty new to Pandas. Do you know what the first and second line do?

Specifically, I try to understand what .isnull(),:] is doing

and df.shape[0] and the 2 at the end ? :S

Thanks for your help.

salary_nan_df = survey_df.loc[survey_df['ConvertedSalary'].isnull(), :]

percentage = round((salary_nan_df.shape[0] / survey_df.shape[0]) * 100, 2)

print(str(percentage)+"% ("+str(salary_nan_df.shape[0])+") of responders have not filled in their salary.")

51.75% (51153) of responders have not filled in their salary.