I think you make a really good point about ignorance and its dangers in this context. I think that the most significant inhibitor to social change might actually be fear, which you mention as well. As Baldwin writes in his letter, "people find it very difficult to act on what they know (...) the danger, in the minds of most white Americans is the loss of their identity." As Yancy explains through the work of Sarah Ahmed and others, white people have not had to think about their whiteness. Whiteness is "invisible," and addressing their privilege would be a threat to a white person's comfortable position as the default, or the normative existence. The unwillingness to look critically at whiteness, or to allow others to do so, perpetuating whiteness as the default, is what inhibits real systemic change.