There’s a lot to query in the analysis highlighted in the note reviewed today, but I’ll come to that later. First the fun part.
Lisheng Yu (et al.) from the Sun Yat Sen University – School of Business wanted to know if Chinese Large Language Models (LLMs) were better at predicting stock price movement based on news flow than ChatGPT.
That notion, that news flow predicts stock price movement is hardly new. It’s been observed for decades but the trick is how to interpret the headlines, their likely effect and how to weight the initial information.
The paper provides two valuable insights.
First, in the analysis a trio* of Chinese LLMs knocked ChatGPT out of the ring in terms of their predictive superiority. Over the sample period of Jan. ’22 to Dec. ’22 a long-short strategy using the China models generated a return of 87.5% whilst ChatGPT’s return was a ‘mere’ 13.9%.
[*Chat GLM created by the Tsinghua University and Zhipu AI, ERNIE Bot – created by Baidu and QWEN – created by Alibaba]
The main reason for this superior performance seems, unsurprisingly, to be the volume of Chinese language data used to train the home teams.
The second insight is perhaps the more valuable one. The researchers wanted to see if the LLMs could justify their analysis and then place a confidence number in percentage terms on their predictions. We know this process in the real world as reasoning.
Not only could the models supply weighted numbers but incorporating the confidence levels into the test improved the returns significantly.
The analysis struck me as flawed in a number of ways: the sample period is short, the theoretical buy and sell decisions couldn’t possibly work in the real world, adjusting for transaction costs of 25-bp doesn’t capture liquidity constraints and the analysis describes only the here and now. The speed with which the space is developing guarantees instant obsolescence for this and other work like it. Having said all that….
That Chinese models do better on Chinese data is an advantage that isn’t likely to go away and getting models to ‘reason’ is a fascinating development of which we’re sure to hear much more.
You can read the work in full via this link Beyond the Global Giant.
Happy Sunday.