对kaggle入门题titanic的记录：成绩top 10%。
题目分析
通过查看给出的数据集，该问题是一个二分类问题。
准备工作
我的做法是将数据集下载到同目录的dataset文件夹下面，在本地进行特征工程和训练。
准备工作
打开jupyter或者熟悉的ide，推荐jupyter notebook。 首先是常用包的导入：
# 基本包的导入import numpy as npimport os# 画图相关%matplotlib inlineimport matplotlibimport matplotlib.pyplot as pltimport pandas as pdfrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import Pipelinefrom sklearn.preprocessing import Imputerfrom sklearn.linear_model import LogisticRegressionfrom sklearn.preprocessing import StandardScalerfrom sklearn.preprocessing import LabelEncoderimport seaborn as snsfrom sklearn.preprocessing import LabelBinarizer# measurefrom sklearn.metrics import accuracy_score# 忽略警告import warningsDATASET_DIR = "./dataset"复制代码
最后一行是数据集的目录。
下面是sklearn后续版本将会添加的一个功能：对类别数据进行one-hot编码，这里要求会使用。
# 上述函数，其输入是包含1个多个枚举类别的2D数组，需要reshape成为这种数组# from sklearn.preprocessing import CategoricalEncoder #后面会添加这个方法from sklearn.base import BaseEstimator, TransformerMixinfrom sklearn.utils import check_arrayfrom scipy import sparse# 后面再去理解class CategoricalEncoder(BaseEstimator, TransformerMixin): """Encode categorical features as a numeric array. The input to this transformer should be a matrix of integers or strings, denoting the values taken on by categorical (discrete) features. The features can be encoded using a one-hot aka one-of-K scheme (``encoding='onehot'``, the default) or converted to ordinal integers (``encoding='ordinal'``). This encoding is needed for feeding categorical data to many scikit-learn estimators, notably linear models and SVMs with the standard kernels. Read more in the :ref:`User Guide`. Parameters ---------- encoding : str, 'onehot', 'onehot-dense' or 'ordinal' The type of encoding to use (default is 'onehot'): - 'onehot': encode the features using a one-hot aka one-of-K scheme (or also called 'dummy' encoding). This creates a binary column for each category and returns a sparse matrix. - 'onehot-dense': the same as 'onehot' but returns a dense array instead of a sparse matrix. - 'ordinal': encode the features as ordinal integers. This results in a single column of integers (0 to n_categories - 1) per feature. categories : 'auto' or a list of lists/arrays of values. Categories (unique values) per feature: - 'auto' : Determine categories automatically from the training data. - list : ``categories[i]`` holds the categories expected in the ith column. The passed categories are sorted before encoding the data (used categories can be found in the ``categories_`` attribute). dtype : number type, default np.float64 Desired dtype of output. handle_unknown : 'error' (default) or 'ignore' Whether to raise an error or ignore if a unknown categorical feature is present during transform (default is to raise). When this is parameter is set to 'ignore' and an unknown category is encountered during transform, the resulting one-hot encoded columns for this feature will be all zeros. Ignoring unknown categories is not supported for ``encoding='ordinal'``. Attributes ---------- categories_ : list of arrays The categories of each feature determined during fitting. When categories were specified manually, this holds the sorted categories (in order corresponding with output of `transform`). Examples -------- Given a dataset with three features and two samples, we let the encoder find the maximum value per feature and transform the data to a binary one-hot encoding. >>> from sklearn.preprocessing import CategoricalEncoder >>> enc = CategoricalEncoder(handle_unknown='ignore') >>> enc.fit([[0, 0, 3], [1, 1, 0], [0, 2, 1], [1, 0, 2]]) ... # doctest: +ELLIPSIS CategoricalEncoder(categories='auto', dtype=<... 'numpy.float64'>, encoding='onehot', handle_unknown='ignore') >>> enc.transform([[0, 1, 1], [1, 0, 4]]).toarray() array([[ 1., 0., 0., 1., 0., 0., 1., 0., 0.], [ 0., 1., 1., 0., 0., 0., 0., 0., 0.]]) See also -------- sklearn.preprocessing.OneHotEncoder : performs a one-hot encoding of integer ordinal features. The ``OneHotEncoder assumes`` that input features take on values in the range ``[0, max(feature)]`` instead of using the unique values. sklearn.feature_extraction.DictVectorizer : performs a one-hot encoding of dictionary items (also handles string-valued features). sklearn.feature_extraction.FeatureHasher : performs an approximate one-hot encoding of dictionary items or strings. """ def __init__(self, encoding='onehot', categories='auto', dtype=np.float64, handle_unknown='error'): self.encoding = encoding self.categories = categories self.dtype = dtype self.handle_unknown = handle_unknown def fit(self, X, y=None): """Fit the CategoricalEncoder to X. Parameters ---------- X : array-like, shape [n_samples, n_feature] The data to determine the categories of each feature. Returns ------- self """ if self.encoding not in ['onehot', 'onehot-dense', 'ordinal']: template = ("encoding should be either 'onehot', 'onehot-dense' " "or 'ordinal', got %s") raise ValueError(template % self.handle_unknown) if self.handle_unknown not in ['error', 'ignore']: template = ("handle_unknown should be either 'error' or " "'ignore', got %s") raise ValueError(template % self.handle_unknown) if self.encoding == 'ordinal' and self.handle_unknown == 'ignore': raise ValueError("handle_unknown='ignore' is not supported for" " encoding='ordinal'") X = check_array(X, dtype=np.object, accept_sparse='csc', copy=True) n_samples, n_features = X.shape self._label_encoders_ = [LabelEncoder() for _ in range(n_features)] for i in range(n_features): le = self._label_encoders_[i] Xi = X[:, i] if self.categories == 'auto': le.fit(Xi) else: valid_mask = np.in1d(Xi, self.categories[i]) if not np.all(valid_mask): if self.handle_unknown == 'error': diff = np.unique(Xi[~valid_mask]) msg = ("Found unknown categories {0} in column {1}" " during fit".format(diff, i)) raise ValueError(msg) le.classes_ = np.array(np.sort(self.categories[i])) self.categories_ = [le.classes_ for le in self._label_encoders_] return self def transform(self, X): """Transform X using one-hot encoding. Parameters ---------- X : array-like, shape [n_samples, n_features] The data to encode. Returns ------- X_out : sparse matrix or a 2-d array Transformed input. """ X = check_array(X, accept_sparse='csc', dtype=np.object, copy=True) n_samples, n_features = X.shape X_int = np.zeros_like(X, dtype=np.int) X_mask = np.ones_like(X, dtype=np.bool) for i in range(n_features): valid_mask = np.in1d(X[:, i], self.categories_[i]) if not np.all(valid_mask): if self.handle_unknown == 'error': diff = np.unique(X[~valid_mask, i]) msg = ("Found unknown categories {0} in column {1}" " during transform".format(diff, i)) raise ValueError(msg) else: # Set the problematic rows to an acceptable value and # continue `The rows are marked `X_mask` and will be # removed later. X_mask[:, i] = valid_mask X[:, i][~valid_mask] = self.categories_[i][0] X_int[:, i] = self._label_encoders_[i].transform(X[:, i]) if self.encoding == 'ordinal': return X_int.astype(self.dtype, copy=False) mask = X_mask.ravel() n_values = [cats.shape[0] for cats in self.categories_] n_values = np.array([0] + n_values) indices = np.cumsum(n_values) column_indices = (X_int + indices[:-1]).ravel()[mask] row_indices = np.repeat(np.arange(n_samples, dtype=np.int32), n_features)[mask] data = np.ones(n_samples * n_features)[mask] out = sparse.csc_matrix((data, (row_indices, column_indices)), shape=(n_samples, indices[-1]), dtype=self.dtype).tocsr() if self.encoding == 'onehot-dense': return out.toarray() else: return out复制代码
加载数据
将题目提供的数据集下载并保存到相应的目录下。
dataset_train_tf = pd.read_csv("./dataset/train.csv")dataset_test_tf = pd.read_csv("./dataset/test.csv")combine = [dataset_train_tf, dataset_test_tf]复制代码 查看训练数据的前五条数据 ：
dataset_train_tf.head()复制代码
输出如下：
可以看到包含不同的信息，有数值特征和类别特征。其每个特征的含义请参考kaggle题目介绍。
下面查看一下其数据分布状态：
此图有很多重要的信息：
- 有接近38%的人存活。(平均值的计算)， 由0，1表示。
- silsp表示兄弟姐妹，parch表示配偶，一起表示随亲友出行的个数。75%的人都是0，说明都是单人出行。
- 将近30%的乘客有兄弟姐妹和/或配偶。
- 票价差别很大，几乎没有乘客(<1%)支付高达512美元。
- 几乎没有乘客是65-80岁的。
上面是数值特征分布状态，下面查一下类别特征的状态：
一些猜测
- 是否女性更容易存活
- 是否小孩更容易存活
- 票的类型跟存活有什么关系。
下面看一下数据的类型和缺失情况情况：
上面3步基本上是拿到一个数据集应该进行的必备步骤。下面进行简单分析：
简单分析
通过同groupby针对不同属性进行分组，查看与存活之间的关系。
dataset_train_tf[["Pclass", "Survived"]].groupby(['Pclass'],as_index=False).mean().sort_values(by="Survived", ascending=False)复制代码
其结果如下：
| Pclass | Survived | |
|---|---|---|
| 0 | 1 | 0.629630 |
| 1 | 2 | 0.472826 |
| 2 | 3 | 0.242363 |
不同的船票，其生还的可能也不同。
dataset_train_tf[["Sex", "Survived"]].groupby(['Sex'],as_index=False).mean().sort_values(by="Survived", ascending=False)复制代码
| Sex | Survived | |
|---|---|---|
| 0 | female | 0.742038 |
| 1 | male | 0.188908 |
等等，这里不做过多分析。
可视化部分可以参考kaggle的相关入门参考和titanic的文章。
通过corr函数可以计算不同特征之间的关联性。
# 数据间的相互关系corr_matrix = dataset_train_tf.corr()corr_matrix["Survived"].sort_values() #数据建的相互相关， 1， -1 最有用复制代码
其结果输出如下：
Pclass -0.338481Age -0.077221SibSp -0.035322PassengerId -0.005007Parch 0.081629Fare 0.257307Survived 1.000000Name: Survived, dtype: float64复制代码
后面介绍缺失数据的补全，Pipeline的使用，类别特征的编码，模型的训练与预测。