Should ‘natural-born’ matter?

The past few news cycles have drawn attention to a clause in the American constitution that the President must be a natural-born citizen. I feel that vigorous debate over this could be healthy, but I am disturbed by the large number of people who shrug off the issue with a "Who cares?" I also am troubled by the way that many people interpret this clause and the rest of the constitution opportunistically.
I would not object to a constitutional amendment that allowed any U.S. citizen, of any age, gender, race, or digital format to become president. I would be willing to support the direct election of any citizen who the people chose — if the constitution were amended to prescribe that.