Text Guide

Module Overview

This module aims to support the consistent extraction of key features in O&M data:

  • timestamp information

  • characteristic categorical information

  • a concise synopsis of the issue for context

Implemented functions include those for filling in data gaps (text.preprocess submodule), machine learning analyses to fill in gaps in categorical information and to generate concise summary strings (text.classify submodule), functions to prepare data for natural language processing (text.nlp_utils submodule), and a visualization suite (text.visualize submodule).

An example implementation of all capabilities can be found in text_class_example.py for specifics, and tutorial_textmodule.ipynb for basics.

Text pre-processing

preprocess

These functions process the O&M data into concise, machine learning-ready documents. Additionally, there are options to extract dates from the text.

  • preprocessor() acts as a wrapper function, utilizing the other preprocessing functions, which prepares the data for machine learning.

    • See text_class_example.prep_data_for_ML for an example.

  • preprocessor() should be used with the keyword argument extract_dates_only = True if the primary interest is date extraction instead continuing to use the data for machine learning.

    • See text_class_example.extract_dates module for an example.

Text classification

classify

These functions process the O&M data to make an inference on the specified event descriptor.

  • classification_deployer() is used to conduct supervised or unsupervised classification of text documents. This function conducts a grid search across the passed classifiers and hyperparameters.

  • Once the model is built and selected, classification (for supervised ML) or clustering (for unsupervised ML) analysis can be conducted on the best model returned from the pipeline object.

    • See text_class_example.predict_best_model module for an example.

Utils

utils

These helper functions focus on performing exploratory or secondary processing activities for the O&M data.

  • pvops.text.nlp_utils.remap_attributes() is used to reorganize an attribute column into a new set of labels.

NLP Utils

utils

These helper functions focus on processing in preparation for NLP activities.

  • summarize_text_data() prints summarized contents of the O&M data.

  • Doc2VecModel performs a gensim Doc2Vec transformation of the input documents to create embedded representations of the documents.

  • DataDensifier is a data structure transformer which converts sparse data to dense data.

  • create_stopwords() concatenates a list of stopwords using both words grabbed from nltk and user-specified words

Visualizations

These functions create visualizations to get a better understanding about your documents.