def toPandas(self, df):
cols = df.columns<EOL>matrixCols = []<EOL>def toscipy(v):<EOL><INDENT>if isinstance(v, DenseVector):<EOL><INDENT>return csr_matrix((v.values, np.array(range(v.size)), np.array([<NUM_LIT:0>, v.size])),<EOL>shape=(<NUM_LIT:1>, v.size))<EOL><DEDENT>elif isinstance(v, SparseVector):<EOL><INDENT>return csr_matrix((v.values, v.indices, np.array([<NUM_LIT:0>, len(v.indices)])),<EOL>shape=(<NUM_LIT:1>, v.size))<EOL><DEDENT>else:<EOL><INDENT>raise TypeError("<STR_LIT>" % type(v))<EOL><DEDENT><DEDENT>tosparse = udf(lambda v: toscipy(v), CSRVectorUDT())<EOL>for i in range(len(cols)):<EOL><INDENT>c = cols[i]<EOL>if isinstance(df.schema.fields[i].dataType, VectorUDT):<EOL><INDENT>cols[i] = tosparse(df[c]).alias(c)<EOL>matrixCols.append(c)<EOL><DEDENT>else:<EOL><INDENT>cols[i] = df[c]<EOL><DEDENT><DEDENT>return df.select(*cols).toPandas()<EOL>
This is similar to the Spark DataFrame built-in toPandas() method, but it handles
MLlib Vector columns differently.  It converts MLlib Vectors into rows of
scipy.sparse.csr_matrix, which is generally friendlier for PyData tools like scikit-learn.

.. note:: Experimental: This will likely be replaced in later releases with improved APIs.

:param df: Spark DataFrame
:return:  Pandas dataframe