Preheat the oven to 350°F (175°C) with a rack in the center.
Melt the unsalted butter in a medium saucepan over medium heat until foamy, about 1 minute. Add the yellow onion and cook, stirring, until soft and translucent, 5–7 minutes. Sprinkle in the all-purpose flour and cook, stirring constantly, until foamy and lightly nutty, about 2 minutes. Slowly whisk in the whole milk; bring to a gentle simmer and cook, stirring, until the sauce thickens to coat the back of a spoon, 3–5 minutes. Season with the kosher salt and black pepper, then remove from the heat.
Arrange the russet potatoes in a 9×13-inch (3-quart) baking dish in tight, overlapping layers. Pour the hot sauce evenly over the top, tilt the dish to help it seep between slices, and tap the dish firmly on the counter to settle, 10–15 seconds.
Cover the dish tightly with foil and bake until bubbling around the edges, 40–50 minutes at 350°F.
Uncover and continue baking until a knife slides through the center with little resistance and the top is lightly browned, 25–35 minutes.
Let the dish rest on a rack for 15 minutes to set before serving hot.