CLAIM: A Contextual Language Model for Accurate Imputation of Missing Tabular Data | HackerNoon

Table of Links

Abstract and 1 Introduction

2 Method

2.1 Problem Formulation and 2.2 Missingness Patterns

2.3 Generating Missing Values

2.4 Description of CLAIM

3 Experiments

3.1 Results

4 Related Work

5 Conclusion and Future Directions

6 Limitations and References

Abstract. This paper introduces the Contextual Language model for Accurate Imputation Method (CLAIM), a novel strategy that capitalizes on the expansive knowledge and reasoning capabilities of pre-trained large language models (LLMs) to address missing data challenges in tabular datasets. Unlike traditional imputation methods, which predominantly rely on numerical estimations, CLAIM utilizes contextually relevant natural language descriptors to fill missing values. This approach transforms datasets into natural language contextualized formats that are inherently more aligned with LLMs’ capabilities, thereby facilitating the dual use of LLMs: first, to generate missing value descriptors, and then, to fine-tune the LLM on the enriched dataset for improved performance in downstream tasks. Our evaluations across diverse datasets and missingness patterns reveal CLAIM’s superior performance over existing imputation techniques. Furthermore, our investigation into the effectiveness of context-specific versus generic descriptors for missing data highlights the importance of contextual accuracy in enhancing LLM performance for data imputation. The results underscore CLAIM’s potential to markedly improve the reliability and quality of data analysis and machine learning models, offering a more nuanced and effective solution for handling missing data.

1 Introduction

‘Well! I’ve often seen a cat without a grin,’ thought Alice; ‘but a grin without a cat! It’s the most curious thing I ever saw in all my life!’

Lewis Carroll, Alice’s Adventures in Wonderland (1865)

A compelling real-world example of how context-unaware estimation of missing data can defy reality and compromise the integrity of downstream tasks is highlighted in [35]. This account describes a scenario where a predictive machine learning (ML) model, developed to process tabular demographic data including individuals’ ages, faced challenges due to missing age entries. The imputation strategy employed involved substituting missing age values with zeros—a common default for initializing integers in several programming languages. This approach inadvertently led the model to categorize individuals with unspecified ages as “toddlers”, resulting in aberrant model behavior. Numerous instances echoing this type of bias in ML models, resulting from context-unaware imputation of missing data, are reported in the literature [35,14,38,43,34,1].

These incidents prompt a critical inquiry into more sophisticated and reality-congruent methods for estimating missing tabular data. While simple statistical replacements such as the mean or median might suffice under the assumption of a normal distribution, predictive ML techniques like k-Nearest Neighbors (k-NN), random forest (RF), or even deep learning (DL)-based generative models offer alternative strategies [20,13,45,8]. These ML/DL methods typically presuppose that missingness in an attribute correlates with observable values in other features. However, this raises fundamental questions: What if the missing data is independent of observed values? Or if the absence of data is influenced solely by unobserved variables? In scenarios where missingness is not attributable to external factors or other observed data, the challenge then becomes how to accurately estimate the missing values. To date, no single imputation method has proven universally effective, underscoring the complexity and variety of missing data scenarios encountered in practice [20].

This paper introduces a novel approach, leveraging the capabilities of pre-trained large language models (LLMs) [6,9,39,26], to innovatively address the challenge of missing data in tabular datasets. Our method, the Contextual Language model for Accurate Imputation Method (CLAIM), diverges significantly from traditional imputation techniques that predominantly estimate missing values through numerical methods. Instead, CLAIM harnesses LLMs’ expansive knowledge [28,29] and reasoning capabilities [9,42,4] in a dual-phase process: initially, it employs LLMs to generate contextually relevant natural language descriptors for missing values, effectively transforming datasets into natural language contextualized formats. This transformation is crucial, as it aligns the data with the inherent strengths of LLMs, making it more amenable to their processing capabilities.

Subsequently, these enriched datasets serve as the foundation for fine-tuning LLMs to enhance performance in downstream tasks (e.g., classification), showcasing a unique and effective use of language models beyond their conventional applications. By incorporating contextually relevant descriptors for missing data, CLAIM not only addresses the variability and specificity inherent in data across different domains but also adeptly navigates the complexities introduced by various missingness mechanisms. Through this innovative integration of LLMs into the data imputation process, CLAIM aims to deliver a more nuanced, accurate, and reliable method for data recovery, essential for improving the quality of subsequent data analysis and machine learning tasks.

To assess the effectiveness of CLAIM, we undertake a comprehensive analysis across three standard missing data mechanisms—MCAR (Missing Completely at Random), MAR (Missing at Random), and MNAR (Missing Not at Random) [30], and comparing CLAIM against a wide range of existing imputation methods spanning single and multiple imputation techniques, non-ML and ML methods, and discriminative and generative ML approaches. Our empirical studies, aimed at evaluating the impact of CLAIM on LLM-based downstream classification tasks, are guided by two principal research questions (RQs):

– [RQ1]: How effective is CLAIM in imputing missing values across the distinct missingness mechanisms (MCAR, MAR, and MNAR) and how does it compare with existing imputation methods in terms of accuracy and robustness across varied datasets and missing data scenarios?

– [RQ2]: How does the choice of phrasing for missingness descriptors in CLAIM affect the performance of LLM-based downstream tasks?

The main contributions of this work are multifaceted. Firstly, CLAIM represents a departure from traditional imputation methods by using LLMs to generate context specific descriptors for missing data, establishing a new benchmark in data imputation. Secondly, through extensive empirical evaluation, we demonstrate CLAIM’s superior performance over existing methods across varied datasets and missingness patterns. Lastly, our analysis of context-specific versus generic descriptors provides key insights into optimizing LLM performance for imputation tasks, highlighting the significance of contextual accuracy. Collectively, these contributions advance data preprocessing techniques and open novel pathways for applying LLMs in complex data science challenges.