That’s a damn good question that I’m not sure of the answer to but am immediately fascinated by.
As you said, obviously incumbents who lost their bid for reelection (with the exception of John Adams and John Quincy Adams who both bolted town before their respective successors were inaugurated), attend the inauguration of their opponent. Same with Vice Presidents like Richard Nixon in 1960 or Al Gore in 2000 who not only have to attend the inauguration of their victorious opponent but, in one of the more brutal and heartbreaking traditions in American History, had to preside over the Joint Session of Congress which certified their loss in the Presidential election.
However, I don’t know the protocol for other defeated opponents. In the case of John McCain (2008) or Paul Ryan, the unsuccessful Vice Presidential candidate in 2012, they are in attendance because they remained members of Congress despite their loss in the national election. With losing candidates like Mitt Romney, who wasn’t a member of Congress or any sort of incumbent office-holder, I just don’t know. Part of me would imagine that, out of courtesy, they would receive an invitation to the Inauguration. However, another part of me thinks that might be like rubbing salt in the wound: “Hey, no hard feelings, but do you want to come sit on the frigid steps of the Capitol and watch what you could have done if you had won?” I really don’t know what the protocol is for that, but I’ll try to find out the answer.
Very good question. You get a gold star.