Steve's Explanation of Lagrangian Multipliers

Lagrangian multipliers are used frequently in optmization situations. In machine learning cases, these situations usually occur when one is trying to find the most likely value for a certain probability, where the value is constrained by equations that limit the variables in question. An example is if one wanted to optimize the value A, but with the constraints that 2*A + 5*B = 20, and 6*B + 7*C + 9*D = 5, and so on.

Without going too deeply into the math, to accomplish this optimization, the Lagrangian multipliers are introduced. They are a set of new variables, one for each constraint equation, and each of the constraint equations is multiplied by one of these variables, before being summed together with the original equation to optimize.

Once one has this new equation made up of the old equation and a linear sum of the constraints, the next step is to partially differentiate this new equation with respect to both the original variable and the Lagrangian multipliers. This produces a set of N+1 equations in N+1 unknowns, where N is the number of constraint equations present. The solution to these equations will then produce the solution to the original problem to be optimized.

This explanation is derived from my interpretation of explanations given in lectures and explanations found over the web.