Welcome to Disarray

This summer, I was lucky enough to being accepted to ICML 2024 as spotlight poster. I told this story on reddit (you can see the original post here). However, now that I have a blog, I thought it would be nice to keep it here as well. Well, enough with the introduction, let's get to the story behind the paper titled By tying embedding you are assuming the distributional hypothesis.

For anyone who read through the toy models of superposition. This is mostly a rehash of the same ideas presented in the third section of Elhage et al. 's work. However, to keep things new, I will introduce a regularization term to the loss function. This will allow us to explore the phase diagram in a novel scenario. This analysis suggests that regularization inhibits superposition.

In this post, I will discuss T-Free. A recent paper from aleph-alpha that introduces a new method for learning embedding representations that promises to be both memory and computationally efficient. Of course this post is based from their original paper. However, I will share some of my insights and thoughts on their work.

In this post, I will explore how to find the population minimizer for the conditional risk associated with Categorical Cross Entropy. We will use only basic fundamental probability principles in order to make the derivation accessible to anyone with a basic understanding of probability. The only analytical tool needed are the Lagrange multipliers for constrained optimization problems.