跳到正文
只看一手
9月2日周二
9月1日周一
  1. Berkeley AI Research

    What exactly does word2vec learn?

    【RSS 原文 · 未经 AI 筛选或翻译】What exactly does word2vec learn, and how? Answering this question amounts to understanding representation learning in a minimal yet interesting language modeling task. Despite the fact that word2vec is a well-known precursor to modern language models, for many years, researchers lacked a quantitative and predictive theory describing its learning process. In our new paper , we finally provide such a theory. We prove that there are realistic, practical regimes in which the learning problem reduces to unweighted least-squares matrix factorization . We solve the gradient flow dynamics in closed form; the final learned representations are simply given by PCA. Learning dynamics of word2vec . When trained from small initialization, word2vec learns in discrete, sequential steps. Left: rank-incrementing learning steps in the weight matrix, each decreasing the loss. Right: three time slices of the latent embedding space showing how embedding vectors expand into subspaces of increasing dimension at each learning step, continuing until model capacity is saturated. Before elaborating on this result, let’s motivate the problem. word2vec is a well-known algorithm for learning dense vector representations of words. These embedding vectors are trained using a contrastive algorithm; at the end of training, the semantic relation between any two words is captured by the angle between the corresponding embeddings. In fact, the learned embeddings empirically exhibit striking linear structure in their geometry: linear subspaces in the latent space often encode interpretable concepts such as gender, verb tense, or dialect. This so-called linear representation hypothesis has recently garnered a lot of attention since LLMs exhibit this behavior as well , enabling semantic inspection of internal representations and providing for novel model ste

8月28日周四
8月27日周三
8月26日周二
8月25日周一
8月22日周五
8月21日周四
  3. OpenAI News

    Mixi reimagines communication with ChatGPT

    【RSS 原文 · 未经 AI 筛选或翻译】Discover how MIXI, a leader in digital entertainment and lifestyle services in Japan, uses ChatGPT Enterprise to transform productivity, boost AI adoption across teams, and create a secure environment for innovation.

8月19日周二
8月18日周一
8月14日周四
8月13日周三
8月12日周二
8月8日周五
8月7日周四
  10. OpenAI News

    Introducing GPT-5

    【RSS 原文 · 未经 AI 筛选或翻译】We are introducing GPT‑5, our best AI system yet. GPT‑5 is a significant leap in intelligence over all our previous models, featuring state-of-the-art performance across coding, math, writing, health, visual perception, and more.

  13. OpenAI News

    GPT-5 System Card

    【RSS 原文 · 未经 AI 筛选或翻译】This GPT-5 system card explains how a unified model routing system powers fast and smart responses using gpt-5-main, gpt-5-thinking, and lightweight versions like gpt-5-thinking-nano, optimized for different tasks and developer use.