# preprocess module

`cranetoolbox.preprocess.preprocess.``count_per_day`(data: [<class 'str'>, <class 'str'>, <class 'str'>, <class 'str'>]) → pandas.core.frame.DataFrame

  Count occurences per day in a list of data records.

  Parameters

    data (list(list())) – List of data records, where the timestamp is the fourth column.

  Returns

    A DataFrame with the count of occurences per day.

  Return type

    pandas.DataFrame

`cranetoolbox.preprocess.preprocess.``merge_counts_dataframe`(counts_list: List[pandas.core.frame.DataFrame]) → Optional[pandas.core.frame.DataFrame]

  Get merged counts per day from a list of DataFrame

  Parameters

    counts_list (list(pandas.DataFrame)) – List of DataFrame with the count of occurences per day.

  Returns

    A DataFrame, sum of the info contained in the input list

  Return type

    pandas.DataFrame

`cranetoolbox.preprocess.preprocess.``preprocess_csv_file`(csv_reader: _csv.reader, file_path: str, output_path: str, replace_or_remove_url: bool, replace_or_remove_mentions: bool, remove_hashtag_or_segment: bool, replace_or_remove_punctuation: bool, replace_or_remove_numbers: bool) → Optional[pandas.core.frame.DataFrame]

  Preprocess a single CSV file.

  Parameters

    - csv_reader (csv.reader) – The reader for the input CSV file, without header.

    - file_path (str) – The path to the input file.

    - output_path (str) – The path to the output folder.

    - replace_or_remove_url (bool) – True to replace URLs, False to remove them.

    - replace_or_remove_mentions (bool) – True to replace mentions, False to remove them.

    - remove_hashtag_or_segment (bool) – True to remove ‘#’ in front of hashtags, False to segment hashtags.

    - replace_or_remove_punctuation (bool) – True to replace multiple punctuation, False to remove all punctuation.

    - replace_or_remove_numbers (bool) – True to replace numbers by their text version, False to remove them.

  Returns

    Dataframe of processed CSV file

  Return type

    pd.DataFrame

`cranetoolbox.preprocess.preprocess.``preprocessing_text`(text: str, replace_or_remove_url: bool, replace_or_remove_mentions: bool, remove_hashtag_or_segment: bool, replace_or_remove_punctuation: bool, replace_or_remove_numbers: bool) → str

  Preprocess the text content of a tweet for analysis.

  Parameters

    - text (str) – Text content of a tweet.

    - replace_or_remove_url (bool) – True to replace URLs, False to remove them.

    - replace_or_remove_mentions (bool) – True to replace mentions, False to remove them.

    - remove_hashtag_or_segment (bool) – True to remove ‘#’ in front of hashtags, False to segment hashtags.

    - replace_or_remove_punctuation (bool) – True to replace multiple punctuation, False to remove all punctuation.

    - replace_or_remove_numbers (bool) – True to replace numbers by their text version, False to remove them.

  Returns

    The clean version of the text.

  Return type

    str

`cranetoolbox.preprocess.preprocess.``preprocessing_tweet`(tweet: [<class 'str'>, <class 'str'>, <class 'str'>], replace_or_remove_url: bool, replace_or_remove_mentions: bool, remove_hashtag_or_segment: bool, replace_or_remove_punctuation: bool, replace_or_remove_numbers: bool) → [<class ‘str’>, <class ‘str’>, <class ‘str’>, <class ‘str’>]

  Preprocess the text content of a tweet for analysis.

  Parameters

    - tweet (list()) – An array with the tweet info, in format [id, original_text, timestamp].

    - replace_or_remove_url (bool) – True to replace URLs, False to remove them.

    - replace_or_remove_mentions (bool) – True to replace mentions, False to remove them.

    - remove_hashtag_or_segment (bool) – True to remove ‘#’ in front of hashtags, False to segment hashtags.

    - replace_or_remove_punctuation (bool) – True to replace multiple punctuation, False to remove all punctuation.

    - replace_or_remove_numbers (bool) – True to replace numbers by their text version, False to remove them.

  Returns

    An array with the tweet info, including clean text, in format [id, original_text, clean_text, timestamp].

  Return type

    list()

`cranetoolbox.preprocess.preprocessTools.``remove_at_user`(text: str) → str

  Removes “@user”

`cranetoolbox.preprocess.preprocessTools.``remove_escaped_unicode`(text: str) → str

  Removes escaped unicode characters from the text

`cranetoolbox.preprocess.preprocessTools.``remove_hashtag_in_front_of_word`(text: str) → str

  Removes hastag in front of a word

`cranetoolbox.preprocess.preprocessTools.``remove_non_ascii`(text: str) → str

  Removes non ascii characters from the text

`cranetoolbox.preprocess.preprocessTools.``remove_numbers`(text: str) → str

  Removes integers

`cranetoolbox.preprocess.preprocessTools.``remove_punctuation`(text: str) → str

  Removes punctuation symbols, except hyphens

`cranetoolbox.preprocess.preprocessTools.``remove_url`(text: str) → str

  Removes url address

`cranetoolbox.preprocess.preprocessTools.``replace_at_user`(text: str) → str

  Replaces “@user” with “atUser”

`cranetoolbox.preprocess.preprocessTools.``replace_contraction`(text: str) → str

  Replaces contractions from a string to their equivalents

`cranetoolbox.preprocess.preprocessTools.``replace_hashtags`(text: str) → str

  Removes hastag in front of a word and add hashtag segmentation

`cranetoolbox.preprocess.preprocessTools.``replace_multi_exclamation_mark`(text: str) → str

  Replaces repetitions of exlamation marks

`cranetoolbox.preprocess.preprocessTools.``replace_multi_question_mark`(text: str) → str

  Replaces repetitions of question marks

`cranetoolbox.preprocess.preprocessTools.``replace_multi_stop_mark`(text: str) → str

  Replaces repetitions of stop marks

`cranetoolbox.preprocess.preprocessTools.``replace_new_line`(text: str) → str

  Replaces new lines with spaces

`cranetoolbox.preprocess.preprocessTools.``replace_numbers`(text: str) → str

  Replaces numbers with their text version

`cranetoolbox.preprocess.preprocessTools.``replace_url`(text: str) → str

  Replaces url address with “url”

`cranetoolbox.preprocess.preprocessTools.``segment_hashtag`(text: str) → str

  Removes hastag in front of a word and add hashtag segmentation