I just learned about the Phi-1 model, and it is amazing!
Here's a link to a video about it by the excellent channel AI Explained:
https://youtu.be/7S68y6huEpU
It seems to be a breakthrough in the AI community, as a new perspective on how to utilize the full power of datasets.
And in my mind it confirms the validity/effectiveness of an idea I've had for a little while now:
Instead of just training the LLM on a large dataset, ask a teacher LLM to give explanations of *why* something is the way it is.
This gives an orca like approach, but on entirely new datasets, not just those already fully absorbed by something like GPT4/ChatGPT.
Then train the new model on those explanations and extrapolated examples that came from an analysis of the dataset.
So in whole, this idea allows the LLM to actually *learn* like a human, rather then simply being trained.
My original intention for the technique was to help an AI become better at replicating a person's actions by understanding their inner thoughts and motivational.
In other words, to let the AI learn a Theory of Mind of the person it's copying.
It's kind of a mashup of the most powerful discovered training techniques:
Wizard's Evol-instruct,
Orca's Explanation tuning,
Phi-1's Textbook synthesis,
And somewhat similar to the philosophy behind AlphaGo's improvement technique, but applied to reasoning and language.
(as shown in this video by Robert Miles: https://youtu.be/v9M2Ho9I9Qo )
My current going name for the technique is:
MUOLD: Mental Understanding Of Learned Datasets
- EagleP
