Angelie Kraft: AI as a ‘Knowledge Technology’? On the Modelling and Benchmarking of ‘Knowledge’ in LLMs and Its Implications for Science

OpenAI solving a Millennium Prize Problem with an LLM has caused a stir in the mathematics community and well beyond. Meanwhile, AI services are heavily advertised as scientific tools and even as end-to-end science automation devices.

In this talk, Angelie Kraft argues that the debate on the pros and cons of AI for science can draw valuable insights from looking at the science behind AI. How do AI researchers define ‘knowledgeability’ in LLMs? How do they think it is represented, and how do they test for it?

Why do AI researchers and developers seem so optimistic that LLMs are a suitable ‘knowledge technology’, and are their reasons ‘good enough’?

The talk critically discusses practices around knowledge representation and measurement (i.e., benchmarking) in AI, an underlying culture that embraces a ‘view from nowhere’, and what might be taken away from the ‘mathematics crisis’.

Zoom details

Link
Meeting-ID: 621 4863 7282 
Kenncode: 057478