Opened in 1926, the original restaurant at 3427 Wilshire Boulevard remains the most famous due to its distinctive shape. Whimsical architecture was popular at the time, and the restaurant was designed to catch the eye of passing motorists. The Brown Derby name originated from a Malverne, New York-based restaurant of the same name that had been a popular hang-out for vaudevillians in the 1920s. It was founded by Wilson Mizner as a small cafe, across the street from the popular Hollywood hot spot the Cocoanut Grove at the Ambassador Hotel. Wilson was the front man; Herbert K. Somborn owned the property and Jack L. Warner put up the money. Wilson Mizner sat in booth 50 almost every day.

The cafe was successful enough to warrant building a second branch later. The original, derby-shaped building was moved in 1937 to 3377 Wilshire Boulevard at the northeast corner of Wilshire Boulevard and Alexandria Avenue, about a block from its previous location (and directly across the street from the Ambassador Hotel).

In September 1980, the restaurant closed without warning. Local preservationists unsuccessfully tried to stop the building from being bulldozed, but the demolition was completed in November and replaced by a parking lot.