# importTools module

class `cranetoolbox.importTools.transform.``TransformationOptions`(languagefilter: str, retweets: bool, max_in_mem: int, text_field_key: str, id_field_key: str, date_field_key: str)

  A simple class to handle ETL options as specified in the driver

`cranetoolbox.importTools.transform.``filter_lighten_chunk`(chunk, opts: cranetoolbox.importTools.transform.TransformationOptions) -> (typing.List[dict], <class 'int'>)

  Filter and lighten a given set of lines, keeping only important keys

  Parameters

    - chunk (list(str) or buffer of str) – The chunk of lines to process

    - opts (TransformationOptions) – Transformation options

  Returns

    List of filtered tweets and parse failure count

  Return type

    list(dict), int

`cranetoolbox.importTools.transform.``is_retweet`(tweet: dict, text_field_key: str) → bool

  Check whether a tweet is a retweet.

  Parameters

    - tweet (str or None) – A dictionary representing a single tweet

    - tweet – Oser-defined name for the “text” field

  Returns

    True/False if the tweet has been labeled as a retweet.

  Return type

    bool

  Note

  Check for automated retweets with the retweet flag and for manual retweets of the form “RT [original tweet]”

`cranetoolbox.importTools.transform.``lighten_tweet`(tweet: dict, text_field_key: str, id_field_key: str, date_field_key: str) -> (<class 'str'>, <class 'str'>, <class 'str'>)

  Lighten a tweet by returning only the fields required for analysis.

  Parameters

    - tweet (dict) – A parsed JSON tweet

    - text_field_key (str) – User-defined name for the “text” field

    - id_field_key (str) – User-defined name for the “id” field

    - date_field_key (str) – User-defined name for the “created_at” field

  Returns

    A tuple of the three values scraped from the passed tweet

  Return type

    tuple(str, str, str)

`cranetoolbox.importTools.transform.``matches_language_filter`(tweet: dict, opts: cranetoolbox.importTools.transform.TransformationOptions) → bool

  Check whether a tweet is in the desired language. If JSON key does not exist it assumes that the text matches the language filter(returns True)

  Parameters

    - tweet (dict) – A dictionary representing a full JSON tweet(with no data removed etc)

    - opts (TransformationOptions) – Transformation options

  Returns

    True/False if the tweet matches the user specified language filter

  Return type

    bool

`cranetoolbox.importTools.transform.``parse_tweet`(tweet: str) → dict

  Parse the passed JSON format tweet from str to dictionary.

  Parameters

    tweet (str) – JSON tweet as a string

  Returns

    Dictionary representing the JSON parse results of the passed tweet

  Return type

    dict

`cranetoolbox.importTools.transform.``process_files`(file_list: List[str], opts: cranetoolbox.importTools.transform.TransformationOptions, csv_output_path: str) -> (<class 'int'>, <class 'int'>)

  Top-level function to combine input set into a single CSV file.

  Parameters

    - file_list (list(str)) – paths to files to be processed

    - opts (TransformationOptions) – An instance of the transformation options, used to control filtering and parsing of tweets

    - csv_output_path (str) – Full output path, folder, filename and extension

  Returns

    A tuple of (successes, failures) that represents the number of lines written to the file

  Return type

    tuple(int, int)

`cranetoolbox.importTools.transform.``process_tar_file`(file: str, opts: cranetoolbox.importTools.transform.TransformationOptions, csv_output_path: str) -> (<class 'int'>, <class 'int'>)

  Process any uncompressed nested files contained within a single tar file.

  Parameters

    - file (str) – Path to tar file

    - opts (TransformationOptions) – Transformation options

    - csv_output_path (str) – Output path for the combined CSV file

  Returns

    Pass/fail counts

  Return type

    tuple(int, int)

  Warning

  This cannot handle nested compression, ie a tar inside a tar.

`cranetoolbox.importTools.transform.``write_tweets_by_chunk`(lines, csv_output_path: str, opts: cranetoolbox.importTools.transform.TransformationOptions) -> (<class 'int'>, <class 'int'>)

  Process an arbitrary number of lines and save them to the CSV outfile

  Parameters

    - lines (list(str) or buffer) – Lines of tweets to process and write to file

    - csv_output_path (str) – Full output path of CSV file, including filename and extension

    - opts (TransformationOptions) – Transformation options

  Returns

    Tuple of write pass/failures

  Return type

    tuple(int, int)