Parquet - EDS-NLP

[giscus[bot]]

Parquet - EDS-NLP

https://aphp.github.io/edsnlp/latest/data/parquet/

[LoickChardon]

If you wish to write data in the HDFS, you may run into the following error messages :

OSError

If you encounter an OS error message :

OSError: HDFS connection failed

then you need to set up two environment variables before running your script. In a terminal session, run the following command :

export CLASSPATH="$($HADOOP_HOME/bin/hadoop classpath --glob)" && export ARROW_LIBHDFS_DIR="$HADOOP_HOME/lib/native"

Then run your python script as usual. Note that you will need to run this command every time you open a new terminal session.

You can make it persistent by editing the .bashrc file :

Open .bashrc with vim:

 vim ~/.bashrc

Press g (go to last line) then i (insert mode) and paste these three command lines with ctrl + shift + v :

export HADOOP_HOME=/usr/local/hadoop
export CLASSPATH=$($HADOOP_HOME/bin/hadoop classpath --glob)
export ARROW_LIBHDFS_DIR=$HADOOP_HOME/lib/native

Press Esc

Then type :wq

And press Enter

Apply changes immediately with :

source ~/.bashrc

PermissionError

If you encounter a permission error message :

PermissionError: [Errno 13] Cannot create directory '/user'.
Detail: [errno 13] Permission denied

then you need to specify a filesystem to the write_parquet() method as follows :

import fsspec
hdfs = fsspec.filesystem("hdfs", host="bbsedsi", port=8020)

edsnlp.data.write_parquet(data=docs, path=/path/to/file.parquet, filesystem=hdfs)

[percevalw]

⚠️ overwrite=True won't delete the existing directory, but append to it. While we work on improving the API, you can use shutil.rmtree to delete any existing data.