Table of Links
Abstract and 1 Introduction
2 COCOGEN: Representing Commonsense structures with code and 2.1 Converting (T,G) into Python code
2.2 Few-shot prompting for generating G
3 Evaluation and 3.1 Experimental setup
3.2 Script generation: PROSCRIPT
3.3 Entity state tracking: PROPARA
3.4 Argument graph generation: EXPLAGRAPHS
4 Analysis
5 Related work
6 Conclusion, Acknowledgments, Limitations, and References
A Few-shot models size estimates
B Dynamic prompt Creation
C Human Evaluation
D Dataset statistics
E Sample outputs
F Prompts
G Designing Python class for a structured task
H Impact of Model size
I Variation in prompts
C Human Evaluation
Out of the four tasks used in this work, PROSCRIPT edge prediction and PROPARA have only one possible correct value. Thus, following prior work, we report the automated, standard metrics for these tasks. For EXPLAGRAPHS, we use model-based metrics proposed by Saha et al. (2021), which were found to have a high correlation with human judgments. For PROSCRIPT graph generation, we conducted an exhaustive automated evaluation that separately scores the correctness of the nodes and the correctness of the edges.
However, automated metrics are limited in their ability to evaluate model-generated output. Thus, to further investigate the quality of outputs, we conduct a human evaluation to compare the outputs generated by COCOGEN and DAVINCI. We sampled 20 examples, and three of the authors performed the evaluation. Annotators were shown two graphs (generated by COCOGEN and DAVINCI) and were asked to select one they thought was better regarding relevance and correctness. The selection for each criterion was made independently: the same graph could The annotations were done separately: the same graph could have more relevant nodes (higher relevance) but may not be correct. The identity of the model that generated each graph (COCOGEN or DAVINCI) was shuffled and unknown to the evaluators.
The results in Table 11 indicate that human evaluation is closely correlated with the automated metrics: for EXPLAGRAPHS, annotators found the graphs generated by COCOGEN to be more relevant and correct. We find that DAVINCI often fails to recover semantic relations between nodes in the argument graphs. For example, consider a belief (B) urbanization harms natural habitats for the animals in the world. We want to generate a graph that can counter this belief with the argument (A) urbanization causes increase in jobs.
For the same prompt, COCOGEN generated (urbanization; causes; increase in jobs); (increase in jobs; has context; good); (good; not capable of; harms) whereas DAVINCI generated (jobs; not harms; natural habitats) → (natural habitats; not part of; animals). Note that DAVINCI successfully recovered relevant events (“natural habitat” “animals”) but arranged them in incorrect relations. For PROSCRIPT, the human evaluation shows that COCOGEN and DAVINCI have complementary strengths, while COCOGEN generally produces more relevant and correct outputs.
D Dataset statistics
Dataset statistics are shown in Table 12. The test split for EXPLAGRAPHS is not available, so we evaluate on the validation split. For PROSCRIPT, we obtained the test splits from the authors.
E Sample outputs
Sample outputs from COCOGEN for all the tasks are located at https://github.com/ madaan/CoCoGen/tree/main/outputs. Representative examples from each task are presented in Figure 5. Surprisingly, COCOGEN (CODEX with a Python prompt) generates syntactically valid Python graphs that are similar to the task graphs/tables in nearly 100% of the cases.
F Prompts
The prompts for each tasks are present at this anonymous URL:
-
PROSCRIPT script-generation: https://github.com/madaan/CoCoGen/tree/main/data/proscript_ script_generation/prompt.txt
-
PROSCRIPT edge-prediction: https://github.com/madaan/CoCoGen/tree/main/data/proscript_edge_ prediction/prompt.txt
-
PROPARA: https://github.com/madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt
-
EXPLAGRAPHS: https://github.com/madaan/CoCoGen/tree/main/data/ explagraphs/prompt.txt
These prompts are also present in the attached supplementary material, and can be found in the data folder under respective task sub-directories.
Authors:
(1) Aman Madaan, Language Technologies Institute, Carnegie Mellon University, USA ([email protected]);
(2) Shuyan Zhou, Language Technologies Institute, Carnegie Mellon University, USA ([email protected]);
(3) Uri Alon, Language Technologies Institute, Carnegie Mellon University, USA ([email protected]);
(4) Yiming Yang, Language Technologies Institute, Carnegie Mellon University, USA ([email protected]);
(5) Graham Neubig, Language Technologies Institute, Carnegie Mellon University, USA ([email protected]).
