팬더 데이터 프레임의 열 정규화

팬더 데이터 프레임의 열 정규화

---

각 열의 값 범위가 다른 팬더에 데이터 프레임이 있습니다. 예를 들면 다음과 같습니다.

df :

A     B   C
1000  10  0.5
765   5   0.35
800   7   0.09

각 값이 0과 1 사이 인이 데이터 프레임의 열을 정규화하는 방법에 대한 아이디어가 있습니까?

원하는 결과는 다음과 같습니다.

A     B    C
1     1    1
0.765 0.5  0.7
0.8   0.7  0.18(which is 0.09/0.5)

---

패키지 sklearn 및 관련 전처리 유틸리티를 사용하여 데이터를 정규화 할 수 있습니다.

import pandas as pd
from sklearn import preprocessing

x = df.values #returns a numpy array
min_max_scaler = preprocessing.MinMaxScaler()
x_scaled = min_max_scaler.fit_transform(x)
df = pd.DataFrame(x_scaled)

자세한 내용 은 전처리 데이터에 대한 scikit-learn 설명서 : 기능을 범위로 스케일링을 참조하십시오.

---

Pandas 를 사용하는 쉬운 방법 : (여기서 평균 정규화를 사용하고 싶습니다)

normalized_df=(df-df.mean())/df.std()

최소-최대 정규화를 사용하려면 :

normalized_df=(df-df.min())/(df.max()-df.min())

---

이 게시물을 기반으로 : https://stats.stackexchange.com/questions/70801/how-to-normalize-data-to-0-1-range

다음을 수행 할 수 있습니다.

def normalize(df):
    result = df.copy()
    for feature_name in df.columns:
        max_value = df[feature_name].max()
        min_value = df[feature_name].min()
        result[feature_name] = (df[feature_name] - min_value) / (max_value - min_value)
    return result

값이 음수인지 양수인지 걱정할 필요가 없습니다. 그리고 값은 0과 1 사이에 잘 ​​퍼져 있어야합니다.

---

sklearn 패키지 사용을 좋아하는 경우 팬더를 사용하여 열 및 색인 이름을 유지할 수 있습니다 loc.

from sklearn.preprocessing import MinMaxScaler

scaler = MinMaxScaler() 
scaled_values = scaler.fit_transform(df) 
df.loc[:,:] = scaled_values

---

문제는 실제로 열에 작용하는 간단한 변환입니다.

def f(s):
    return s/s.max()

frame.apply(f, axis=0)

또는 더 간결한 :

   frame.apply(lambda x: x/x.max(), axis=0)

---

단순하다 아름다운 :

df["A"] = df["A"] / df["A"].max()
df["B"] = df["B"] / df["B"].max()
df["C"] = df["C"] / df["C"].max()

---

정규화하려는 열 목록을 만들 수 있습니다.

column_names_to_normalize = ['A', 'E', 'G', 'sadasdsd', 'lol']
x = df[column_names_to_normalize].values
x_scaled = min_max_scaler.fit_transform(x)
df_temp = pd.DataFrame(x_scaled, columns=column_names_to_normalize, index = df.index)
df[column_names_to_normalize] = df_temp

이제 판다 데이터 프레임이 원하는 열에서만 정규화됩니다.

---

그러나 반대로 원하는 경우 정규화 하지 않으려 는 열 목록을 선택하면 모든 열 목록을 만들고 원하는 열을 제거 할 수 있습니다.

column_names_to_not_normalize = ['B', 'J', 'K']
column_names_to_normalize = [x for x in list(df) if x not in column_names_to_not_normalize ]

---

팬더에서 그렇게하는 더 좋은 방법은

df = df/df.max().astype(np.float64)

Edit If in your data frame negative numbers are present you should use instead

df = df/df.loc[df.abs().idxmax()].astype(np.float64)

---

The solution given by Sandman and Praveen is very well. The only problem with that if you have categorical variables in other columns of your data frame this method will need some adjustments.

My solution to this type of issue is following:

 from sklearn import preprocesing
 x = pd.concat([df.Numerical1, df.Numerical2,df.Numerical3])
 min_max_scaler = preprocessing.MinMaxScaler()
 x_scaled = min_max_scaler.fit_transform(x)
 x_new = pd.DataFrame(x_scaled)
 df = pd.concat([df.Categoricals,x_new])

---

You might want to have some of columns being normalized and the others be unchanged like some of regression tasks which data labels or categorical columns are unchanged So I suggest you this pythonic way (It's a combination of @shg and @Cina answers ):

features_to_normalize = ['A', 'B', 'C']
# could be ['A','B'] 

df[features_to_normalize] = df[features_to_normalize].apply(lambda x:(x-x.min()) / (x.max()-x.min()))

---

def normalize(x):
    try:
        x = x/np.linalg.norm(x,ord=1)
        return x
    except :
        raise
data = pd.DataFrame.apply(data,normalize)

From the document of pandas,DataFrame structure can apply an operation (function) to itself .

DataFrame.apply(func, axis=0, broadcast=False, raw=False, reduce=None, args=(), **kwds)

Applies function along input axis of DataFrame. Objects passed to functions are Series objects having index either the DataFrame’s index (axis=0) or the columns (axis=1). Return type depends on whether passed function aggregates, or the reduce argument if the DataFrame is empty.

You can apply a custom function to operate the DataFrame .

---

The following function calculates the Z score:

def standardization(dataset):
  """ Standardization of numeric fields, where all values will have mean of zero 
  and standard deviation of one. (z-score)

  Args:
    dataset: A `Pandas.Dataframe` 
  """
  dtypes = list(zip(dataset.dtypes.index, map(str, dataset.dtypes)))
  # Normalize numeric columns.
  for column, dtype in dtypes:
      if dtype == 'float32':
          dataset[column] -= dataset[column].mean()
          dataset[column] /= dataset[column].std()
  return dataset

---

Note that sklearn uses biased estimator for standard deviation. Consider following normalize example:

import pandas as pd
df = pd.DataFrame({
               'A':[1,2,3],
               'B':[100,300,500],
               'C':list('abc')
             })
print(df)
   A    B  C
0  1  100  a
1  2  300  b
2  3  500  c

When normalizing we simply subtract the mean and divide by standard deviation.

df.iloc[:,0:-1] = df.iloc[:,0:-1].apply(lambda x: (x-x.mean())/ x.std(), axis=0)
print(df)
     A    B  C
0 -1.0 -1.0  a
1  0.0  0.0  b
2  1.0  1.0  c

If you do the same thing with sklearn you will get DIFFERENT output!

import pandas as pd

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()


df = pd.DataFrame({
               'A':[1,2,3],
               'B':[100,300,500],
               'C':list('abc')
             })
df.iloc[:,0:-1] = scaler.fit_transform(df.iloc[:,0:-1].to_numpy())
print(df)
          A         B  C
0 -1.224745 -1.224745  a
1  0.000000  0.000000  b
2  1.224745  1.224745  c

The results are different. However, as per the official documentation of sklearn.preprocessing.scale using biased estimator is UNLIKELY to affect the performance of machine learning algorithms and we can safely use them.

---

It is only simple mathematics. The answer should as simple as below.

normed_df = (df - df.min()) / (df.max() - df.min())

---

You can do this in one line

DF_test = DF_test.sub(DF_test.mean(axis=0), axis=1)/DF_test.mean(axis=0)

it takes mean for each of the column and then subtracts it(mean) from every row(mean of particular column subtracts from its row only) and divide by mean only. Finally, we what we get is the normalized data set.

---

This is how you do it column-wise using list comprehension:

[df[col].update((df[col] - df[col].min()) / (df[col].max() - df[col].min())) for col in df.columns]

---

You can simply use the pandas.DataFrame.transform1 function in this way:

df.transform(lambda x: x/x.max())

참고URL : https://stackoverflow.com/questions/26414913/normalize-columns-of-pandas-data-frame