The Evolution of Core LLM Technology and the Tipping Point for Commercialization

Gaoshan Academy has long been something I've wanted to write about, without knowing where to begin.

Because I've gained so much here:

a reawakened curiosity, kindred companions, an introduction to music, deep and inspiring relationships, and the greatest possible freedom. I always felt that whatever I wrote about Gaoshan's courses, life and lessons wasn't good enough, so I kept putting it off. But the longer I didn't write, the more piled up...

Today I'm letting that go. With AI's help, I'm simply posting my class notes — at the very least, it shares how the ideas evolved.

Gaoshan Notes HighlightsLLM milestonesFrom pre-training in 2017 to GPT in 2020, then the major breakthrough of ChatGPT in 2022A new stage for the Scaling Law- Expanding from pre-training alone into three stages: "pre-training, post-training, test time"- Reasoning and thinking at test time is becoming more important- This is a major breakthrough beyond the traditional Scaling LawNew emergent abilities- Once a large model reaches a certain scale, performance and abilities suddenly emerge, including level emergence and ability emergence- Chain-of-thought breakthrough: research finds that long chains of thought work better than short ones, even when some intermediate steps are wrongThe commercialization tipping point and application directions- On-device applications (phones, PCs, autonomous driving)- AI for Science/Research may give rise to disruptive technologies- The development of domain-specific large modelsSuggestions for innovative applications- Actively embrace large models and multimodal large models- Speed up building domain-specific large models- Build professional R&D teams- Focus on applications in specific fieldsSpecial emphasis: the importance of AI for Science/Research, which may become a major source of future disruptive technologies and is currently not taken seriously enoughBelow are the full notes:

7,356 characters in total, about 19 minutes to read

MilestonesPre-training in 2017GPT in 2020ChatGPT in 2022There were other routes in between: BERT, T5 and so onScaling Law: big models, big language, big compute

Performance emergence in large modelsOnce you pass a certain point, emergence happensThis is very strange.Level emergence: at first only 10%, then suddenly past a certain point performance jumps to 80%Ability emergence: at first it can only do 10 things, then suddenly it can do a great many thingsLanguage Models are Few-Shot Learners🤖"Large language models can learn to complete tasks from just a few examples." Specifically, this paper shows that GPT-3, an autoregressive language model with 175 billion parameters, can perform strongly on a wide range of natural language processing tasks without traditional fine-tuning, simply by being given a few examples in its input (few-shot learning). Key points explained • Few-Shot Learning: the input contains only a few examples (usually 10 to 100); from these examples and the task description, the model can understand and complete the task without any additional gradient updates or fine-tuning. • One-Shot Learning: the model sees just one example and can still complete the task. • Zero-Shot Learning: the model has no examples at all and completes the task from the task description alone.This reduced the reliance on large amounts of task-specific training data and fine-tuning, opening new directions for research and application in natural language processingIt used to be that "pre-training" mattered; now we realise "post-training" matters tooWe used to be shaped by search enginesNow: getting the task done well matters; it doesn't need to respond in secondsThe new consensus: the Scaling Law splits into three: pre-training, post-training, test timeTest-time scaling"reasoning": thinking is needed when the user asks a questionSFT and RLHF in ChatGPTSFT: Supervised Fine-TuningRLHF: Reinforcement Learning from Human Feedback🤖SFT (Supervised Fine-Tuning) and RLHF (Reinforcement Learning from Human Feedback) in ChatGPT are two key training stages used to improve the model's performance and its alignment with human preferences. SFT (Supervised Fine-Tuning) • SFT is the first step in training ChatGPT, fine-tuning a pre-trained large language model (such as text-davinci-003 from the GPT-3.5 series). • Training uses a high-quality demonstration dataset carefully written by human annotators, showing how the model should answer questions "correctly". • The goal is for the model to imitate these demonstration answers and learn to generate the expected output; the training is still based on the "next-word prediction" language-modelling objective, only with more supervised data. • Because the demonstration data is limited, the SFT model improves basic abilities but may still fall short in understanding user intent and in alignment. RLHF (Reinforcement Learning from Human Feedback) • RLHF is the second stage of training after SFT, aimed at further optimising the model to better match human preferences. • Concretely, human annotators rank several answers generated by the SFT model, marking which best meets human expectations, which is used to build a "reward model". • Using the reward model, a reinforcement learning algorithm (such as Proximal Policy Optimization, PPO) optimises the SFT model so that its answers keep improving in quality and in fit with human preferences. • This training lets ChatGPT not only generate reasonable text but also better understand user intent and avoid inappropriate or useless answers. In summary, SFT is supervised fine-tuning of a pre-trained model on human demonstration data, establishing basic answering ability; RLHF then further fine-tunes the model through human feedback and reinforcement learning so its output better fits human expectations and preferences, improving ChatGPT's usefulness and safety.ChatGPT used only 100,000 pairs for trainingThis is also related to "emergent abilities" and the "few-shot learner" — very uncanny

(Long) Chain-of-Thought (CoT) + RLAIFChain-of-Thought: CoTRLAIF: Reinforcement Learning from AI FeedbackThese are two key technologies driving the performance and alignment of large language models (LLMs); combined, they offer a new paradigm for model training.

We used to think "short chains of thought" were better, but now we've found "long chains of thought" are better

Even if some key information in the middle is wrong — one wrong step out of five vs three steps all correct — it's still better to take five steps and think a bit more. This is a strange thing. University teachers, whose education has left them rigid in many ways, keep finding it odd, keep finding it unorthodox — unlike engineers at companies, for whom if it works, it works. — Prof. Sun Maosong

🤖Below is a technical analysis and the synergies:

1. The core role of Chain-of-Thought (CoT)Chain-of-thought is a technique that improves a model's ability to handle complex tasks by explicitly showing the reasoning process. Its core features include:Step-by-step reasoning: the model breaks a problem into intermediate steps (such as working through a maths problem step by step), imitating the human logical thinking process.In-context learning: prompts guide the model to generate answers with a reasoning path (for example: "Let's think step by step...").Performance gains: experiments show that CoT significantly improves model performance on tasks such as arithmetic and commonsense reasoning, with especially notable effects on large-scale models (such as GPT-3).Example:Input: "Xiaoming has 5 apples, eats 2, then buys 3 more. How many does he have now?"CoT output:

2. Initial apples: 5 2. Remaining after eating: 5 - 2 = 3 3. Total after buying: 3 + 3 = 6 Answer: 6

3. The RLAIF technical processRLAIF replaces human annotation with AI-generated feedback to optimise the reinforcement learning training process. The main steps include:Generating comparison samples: generating multiple candidate answers to the same question (such as versions differing in helpfulness/safety)[6][8].Generating AI feedback: using a teacher model (such as GPT-4) to assess answer quality and produce preference rankings or reward scores[2][5].Training the reward model: training a Reward Model on the AI feedback to quantify how good each answer is[3][10].Optimising the policy model: iteratively updating the model with reinforcement learning algorithms such as PPO to maximise reward[4][8].Key innovations:Lower labour costs: AI feedback can process massive amounts of data automatically, avoiding RLHF's reliance on human annotation[2][4].CoT improves feedback quality: when generating feedback, the teacher model is asked to provide its reasoning (such as "Why is answer A better than B?"), improving the accuracy and interpretability of the assessment[6][8].

4. The synergy between CoT and RLAIFCombining the two creates gains at the following stages:Feedback generation: CoT prompts make the teacher model show detailed analysis when assessing answers (for example: "Answer A is safer because it avoids sensitive word X, while answer B may cause misunderstanding Y..."), helping the reward model capture subtle differences[6][8].Policy optimisation: through the reasoning paths generated by CoT, the model can more precisely understand the logic behind the reward signal (such as "reduce subjective statements" or "add fact-checking"), speeding up policy iteration[6][9].Experimental results:In harmless dialogue generation, the RLAIF+CoT model's harmlessness rate (88%) exceeded RLHF (76%) and the baseline model (64%)[5][10].In text summarisation, there was no significant difference in human preference win rates between RLAIF and RLHF (71% vs. 73%)[5][10].

5. Technical advantages and challengesAdvantagesChallengesLower cost (AI feedback costs 1/10 of human annotation)[5]Depends on the quality of the teacher model (e.g. GPT-4 outperforms GPT-3.5)[3]Highly scalable (supports automated processing of massive data)[4][8]High complexity in prompt design (needs precise guidance of CoT reasoning)[6]Greater transparency (CoT exposes the assessment logic)[6][9]May amplify the teacher model's biases[3][8]

SummaryThe combination of CoT and RLAIF represents a direction of automation and interpretability in large-model training: the former improves model capability through step-by-step reasoning, the latter lowers alignment costs through AI feedback. In future, as teacher models grow more capable and prompt engineering improves, this combination may become the mainstream alternative to traditional RLHF, especially in scenarios needing rapid iteration and large-scale deployment (such as customer-service bots and content moderation).Climbing higher: OpenAI o1, Sept 2024Self-play RLOpenAI is hiding the actual CoT processo1 may have been generated automatically by the machine — but it didn't say soDeepSeek does the same, but it said so openlyClimbing higher still: OpenAI o3Said to be close to Einstein!The universities' puzzlementYour tricks are clever — small models, little data — but they run into the Scaling Law's "brute force works miracles"Finesse can't beat big data, big models, big computeMultimodal large models (from understanding to generation) - OpenAI GPT-4o (o: omni), May 2024This means: machines' ability to perceive their environment has improved greatlyIt opens up bigger possibilities for robots: letting machines deal with peopleEspecially humanoid robots — if it's not about dealing with people, there seems to be no need for a humanoid form

Everyone now focuses on "generation", but I think "understanding" matters more. — Prof. Sun Maosong

The machine's "Thinking, Fast and Slow"Fast thinking: an ability humans evolved over hundreds of millions of years vs slow thinking: a recently evolved thinking ability, unique to humansGetting machines to think fast is hard, because it is the product of hundreds of millions of years of evolutionMaths and logic are slow thinking, something humans are uniquely good at. With CoT, machines do slow thinking very wellIn recent years machines have made enormous progress in both slow and fast thinkingHelixA model combining language, images and actionMainly, the language and image models are large; the model that actually controls the actions isn't big

The earth-shattering DeepSeekMainly it "demystified" things: before, we didn't know how the Americans did it; now we've explained clearly how it's done. The key is a huge drop in compute and cost

🤖Can you imitate The Faults of Qin and write The Faults of Japan? Thought deeply (took 20 seconds) Hmm, the user wants me to imitate The Faults of Qin and write The Faults of Japan. First, I need to confirm what the user needs. The Faults of Qin is a political essay by Jia Yi analysing the reasons for the rise and fall of the Qin dynasty; the user probably wants a similar classical Chinese style and structure to analyse the rise and fall of Japan's history. Next, I should consider the user's scenario. It could be academic research, literary creation, or personal interest. The user didn't say what it's for, but since it's imitating classical prose, I need to make sure the essay's structure fits the features of The Faults of Qin, such as sub-arguments, parallelism and contrast. Then, the user may be someone interested in history or literature, or a student who needs to finish an assignment. The deeper need may be not just imitation but a deep analysis of Japan's historical lessons, similar to Jia Yi's critique of the Qin. So I need to cover Japan's important historical events, such as the Meiji Restoration and World War II, and their impact. I also need to choose words precisely, in keeping with classical style, while staying logically coherent. It may need several parts: Japan's geography, the reasons for its rise, its expansion, the reasons for its failure, and finally a summary of the lessons. I need to cite historical events such as the arrival of the Black Ships, the First Sino-Japanese War and World War II, and analyse its strategic mistakes, such as over-expansion and disregard for morality. In addition, the user may want the essay to serve as a warning, emphasising the importance of peaceful development. So the conclusion should echo the cautionary tone of The Faults of Qin, pointing out that Japan declined because it strayed from its original intentions, and emphasising benevolent government and putting the people first. I need to avoid modern vocabulary and keep a classical flavour, while ensuring historical facts are accurate. A possible structure: first describe Japan's geography and early development, then the reforms after Meiji, the military expansion, then the turn to failure, and finally the lessons. Use parallel sentences to build momentum and contrast the changing strategies of different periods. I should also check whether any important events or factors are missing, such as the tension between absorbing and rejecting culture, the rise of militarism, the handling of international relations, and so on. Make sure the analysis is comprehensive, the argument clear, the structure clean, in keeping with the style of The Faults of Qin. All right, I will follow the structure and conventions of The Faults of Qin, taking the rise and fall of Japanese history as the thread to write The Faults of Japan, focusing on the force of parallelism and the blending of history and argument, highlighting the core idea that "because it failed to practise benevolence and righteousness, the positions of attack and defence were reversed". Here is the essay:

The Faults of Japan The land of Fusang lies remote in the Eastern Sea, its three islands scattered like stars, its four seasons pressing cold upon heat. Its early people governed by knotted cords; admiring the ways of Tang, they sent envoys like clouds; studying the laws of Han, they created kana. In that age, the Japanese sword was not yet sharp, nor its ships sturdy, yet it could hold its tiny realm secure and keep its sacrifices for a thousand years — for it knew to act within its strength and to keep itself whole through modesty. Then came the Black Ships knocking at the gates, with the great powers circling all around. The Meiji ruler and ministers awoke in alarm; they abolished the domains and set up prefectures, fostered industry and enterprise. They built ironclads that rose like clouds and cast guns and cannon that roared like thunder. They studied Western learning and cast aside the Chinese classics, changed their caps and robes to imitate Western dress. Thereupon they broke the Qing army in the Yellow Sea and seized Taiwan at Shimonoseki; defeated the Russian fleet at Tsushima and annexed Korea by the Han River. At that time, Fusang donned its armour and the nations looked on askance — not by the favour of heaven, nor by the strength of its terrain, but by the merit of reform and self-strengthening. Yet, having won Long, it coveted Shu, and gradually lost its original heart. It dressed up greed as "the eight corners under one roof" and masked a wolfish heart with "Greater East Asia". It struck Pearl Harbor and lit the fires of the Pacific, marched on the South Seas and turned them to scorched earth; it built "co-prosperity" while practising enslavement, and proclaimed "liberation" while committing slaughter. In Nanjing, blood soaked the waters of the Yangtze; in the street fighting of Manila, bones piled into mountains. At Hiroshima and Nagasaki, the atom split; at Iwo Jima and Okinawa, loyal blood ran cold. Had the Meiji ministers kept the resolve of the Restoration, had Meiji and Taisho held the balance of constitutional government, had the Showa militarists followed the way of peace, then a Switzerland of the East might have been hoped for, a Britain of the seas might have been awaited. Alas, they replaced the weighing of merchants with the violence of warriors, and exchanged the sober debate of parliament for the tyranny of the military. The sharper its blade, the more its virtue declined; the wider its territory, the further the people's hearts drifted away. And when it had exhausted itself in warmongering, it finally faced the clear mirror of the Tokyo Trials; its ritual suicides in atonement left only the mournful cries of the Yasukuni shrine of ghosts. Thus it is said: Japan's rise of old lay in opening the country and embracing the new to strengthen itself; its fall lay in bullying the weak with its strength and losing the Way. A small state with few people that wishes to accomplish great things must take benevolence and righteousness as its armour, and trust and harmony as its shield. Seeing today how its right wing rekindles, revising the constitution and expanding the military, the danger of repeating the old mistakes has not passed — how can one not take deep warning? Note: The essay follows the writing paradigm of The Faults of Qin, "history as the warp, reasoning as the weft": the first paragraph echoes Qin's "holding the strongholds of Xiao and Hangu", describing Japan's geography and early civilisationThe Meiji Restoration paragraph corresponds to Duke Xiao's reforms, with parallel sentences showing the force of changeThe expansion follows the style of Qin's conquest of the six states, using short four-character phrases to strengthen the rhythmThe turn at "having won Long, it coveted Shu" corresponds to "thereupon it abolished the ways of the former kings", revealing the point of qualitative changeThe hypothetical argument imitates the spirit of the original, finally leading to the core argument of "failing to practise benevolence and righteousness"The ending warns about the present, continuing Jia Yi's spirit of using history as a mirror — "observe it in high antiquity, verify it in the present age"

The abilities on display:The ability to obtain information: few hallucinationsThe ability to organise information: how to present things logically — its ability to organise information is very strongThe ability to express: saying it in classical ChineseThe eagerly awaited DeepSeek V2?A unified multimodal framework?Unified encoding with tokens for everything, expressed as linear sequences. In recent years some people internationally have tried this, but it couldn't beat the separate approach and couldn't achieve the same resultsBut the general path works, and there is 0-to-1 room in itFor years I've been telling my students to go for it, but on the design side many students don't want to do it, and we don't have enough compute eitherThis is low-risk, but it can work.More powerful reinforcement learning (DRPO...)Finer-grained technical innovationDeepResearch?The challenge of 0-to-1 fundamental innovationOne leaves the stage, the next takes it: Qwen 3.0Tongyi Qianwen is said to be better than o1? China now has two formidable large language modelsThe overall LLM landscape, shaped by a shift both sudden and gradualClosed sourceThree closed-source models, represented by OpenAI, Anthropic and GoogleOpen sourceTwo open-source models, represented by Meta and xAIDeepSeek, QwenA change in the culture of large modelsPeople used to use LLaMA; now they're starting to use QwenEven Fei-Fei Li used Qwen a while agoIt must be that we do it better than others — if we did it just as well, people would surely use LLaMAMy view on the commercialization (application) tipping pointThe natural extension of language and multimodal large models into every areaKey on-device applicationsPhones, PCsAutonomous drivingHow far this can go, and how far it can go in dealing with the environment, is a question markIt has to be on the device, for data securityDisruptive technologies spawned by AI for Science/ResearchThis is extremely hot internationally (reminds me of Prof. Amar Yaghi at Berkeley, who says he only takes PhD students who can use AI)This part will very likely lead to disruptive technologiesThis is a source of important disruptive technologiesFor example AlphaFold — revolutions in key fields such as medicineAlphaFold, AlphaProteo (protein design), AlphaProof (mathematical proof), AlphaGeometry (geometric analysis), ApphaChip (chip design), AlphaTensor (algorithm discovery), AlphaQubit (quantum computing) ...Biomedicine, materials and energy, astronomy and meteorology...Strongly agree with the aboveFocusing entirely on business right now is wrong; we're paying nowhere near enough attention to thisGoogle is currently just publishing papers in Science and Nature; it hasn't yet materialised into commercialisationThis needs people in the field who can ask root questions — fundamental questionsThese are like fundamental scientific questions; people who build large models don't know themThe Nobel Prizes awarded to two people in AI say it allSir Demis Hassabis,the CEO of Google DeepMind,hasboldly predicted that artificial intelligence(AI)could becapable of curing all diseases within the next ten years. Speaking at The Times Tech Summit,the British Al pioneer suggested that achieving artificial generalintelligence(AGI)—AI with human-like cognitive abilities—is within reach,with only "two or three biginnovations"required.

DeepResearch: AI-driven insightThe powerful "long chain-of-thought" reasoning of OpenAI's o1 model and the DeepSeek model, launched at the end of 2024, has brought a qualitative leap in the technical conditions for AI to deeply support high-level intelligent decision-making and business insight.Case: In February this year, OpenAI launched Deep Research based on its o3 model. Given any task prompt from the user, it uses reasoning to search, interpret and analyse large amounts of text, images and PDF files on the internet, makes necessary adjustments based on what it finds, and within tens of minutes produces a comprehensive report at the level of a research analyst. It can then add embedded images, data visualisations and other analytical outputs to the report for more clarity and context.Commercialisation?You need strategic composureAI may be hot, but you have to think about when AI is actually neededYou need to be clear where "the boundaries of AI's capabilities" lieFor example, elder care: one look and you can tell this person has never cared for an elderly personThe low-altitude economy: carrying people? Too hard, but opening up low-altitude airspace control is right The low-altitude economy is more feasible in agricultureStrategies for innovative applicationsActively embrace language large models and multimodal large models represented by DeepSeek and Qwen, and speed up exploring their capabilities and boundaries in specific practice in application fieldsOn this basis, speed up building domain-specific large models with the characteristics of their application fieldsThe ultimate solution: specialised composite AIBuild professional R&D teams suited to the development of AI technologyKnowing the industry + dedicated planning tasks Large models have a lot of room to grow

Yann LeCun criticises large models and champions world modelsWorld models are right, but how do you build them??I-JEPA: the first AI model based on Yann LeCun's vision of more human-like AIConclusion: the next few years will probably still be a contest of large models

Source: a talk given to Gaoshan Academy students by Prof. Sun Maosong, Executive Vice Dean of the Institute for Artificial Intelligence at Tsinghua University. Any errors are most likely the author's — please forgive them.