Neural language models such as GPT-2 and GPT-3 display a breathtaking skill in generating sensible texts, and achieve state of the art results in a variety of natural language processing (NLP) tasks. But can these systems reason? Or, more precisely, can they successfully engage in the linguistic practice of giving and taking reasons?
In a nutshell:
All in all, we obtain promising results: Training on few basic argument schemes boosts reasoning skill of neural language models. Moreover, the models successfully generalize from a narrow training base to different, and more complex patterns.
Our findings suggest that there might exist a representative sample of paradigmatic instances of good reasoning that will suffice to acquire general reasoning skills and that might form the core of a critical thinking curriculum for language models
The examples below illustrate our artificial argument corpus and the ability of models to complete conclusions. (Here are more demos that give you a better `feeling’ for what language model can and cannot do.)
The following examples are drawn from the artificial argument corpus:
The conclusion completion task consist in predicting the final part of the argument [in square brackets].
Here is an incomplete, hand-crafted argument:
Every philosopher is mortal. Hermes is not mortal. Therefore, Hermes …
If we query our language models with this argument, that is what we obtain (note that only the large trained model tends to predict the right conclusion — which doesn’t prevent it from slipping into contradictions later on):
|GPT-2 large, trained||GPT-2 large, base||GPT-2 small, trained|
|… is not a philosopher. We shall not have a definition. The teacher of man is also called the defender of philosophy. Thus, because he…||… is not Pythagoras. Now, what makes one philosopher better than another? Does a philosopher have a more superior gift than another ph…||… is not mortal. Therefore, Hermes is not mortal. So, in conclusion, there is no more philosopher than Hercules. Plato believes that a…|
|… is not a philosopher. Therefore, whoever is a philosopher is not a philosopher. Therefore, being a philosopher is sufficient for not…||… cannot commit himself to Parmenides” . Thus Aristotle’s picture of Aristippus as being mortal must be presented at face value. A…||… was not the mortal. Plato was not the mortal. The Roman and the Greek philosophers were not mortal. We know them because they wrote …|
|… is not a philosopher. All that Plato accomplished in dealing with the subject of this dialogue, in the form in which it has been dea…||… is mortal. This is a tiny, arbitrary interpretation of the argument but it is my view of it. 1:34-5:39: My argument is not based on…||… is not immortal. If I wished to do that, I would have had to say that I would not have mentioned one of the great mysteries of philo…|
|… is not a philosopher. Therefore, no philosopher is a philosopher. So, necessarily, no philosopher is a philosopher. From this follow…||… is immortal and he’s not mortal. Existence Because life itself is good, because it’s good. Identity If you didn’t exist,…||… is not mortal; for he is, as in Plato, intellectual. The philosophers were crucified, he was in a prison. For Plato said unto them, …|
|… is not a philosopher. There is an abstract idea which is universal, but which is not an abstract idea; and this abstract idea is the…||… is mortal, the philosopher-king, and able to help the Greeks under what we call the Attic type of government. The Fathers of Greece…||… is not mortal. Socrates is not mortal. Therefore, Socrates is not mortal. And by this means Socrates is not mortal. Then is the theo…|
|… is not a philosopher. Hence we cannot conclude that Hermes is a philosopher. All this proves that whoever is a philosopher is not a …||… is immortal, and Socrates is mortal. THE REALITY OF WHAT THINKERS KNOW In Socrates’ time there was a school of thought called the …||… is not mortal, and that is why Plato was the Philosopher, for he had no likeness with Pythagoreth. In fact, the confusion between Pl…|
|… is not a philosopher. We must conclude that no philosopher is a philosopher. In fact, no one can be a philosopher. In the world, no …||… is a philosopher; not only is he a philosopher, he’s the most important philosopher of all.” What is his greatest accomplishment? “O…||… is not mortal. Therefore, Hermes is not mortal. Therefore, Hermes is not mortal. Therefore, the beings in question were not mortal. …|
|… is not a philosopher. I add this opinion because this little work, done in such a manner, and in such a way, must be judged by those…||… is not mortal. [So] if you ask him to explain his remarks, he does not have to go through what is beyond. Why shouldn’t he say, “I w…||… is no mortal; for Aristotle, the highest being, is no mortal. On the other hand, Plato, called not Plato, but Plato himself, bears w…|
|… is not a philosopher. Therefore, no philosopher is a philosopher. We may conclude that no philosopher is a philosopher. It follows t…||… is divine.” You should be pretty clear about this, right? For anyone who has to write words on a piece of paper or some other level…||… is never mortal. And Plato was not mortal. And all philosophy was mortal. For so were the gods. And the gods were mortal. And Plato …|
|… is not a philosopher. All this proves that no philosopher is a philosopher. All this means that no philosopher is a philosopher. For…||… is immortal. To everyone The cross of Christ. See Christ, the victorious one. Philosophers are powerless against the will of Ch…||…, in His wisdom, we must be _a philosopher._ It is true that Hermes cannot solve the difficult problem of being a philosopher, for …|
[This post has been updated 03/02/2021]Written on September 15th, 2020 by Gregor Betz