July 4th is Independence Day in the United States, a day commemorating the ratification of the Declaration of Independence, when the thirteen British colonies stated that they no longer recognized British rule. Today, we associate the holiday with the concept of “freedom.”

The events of July 1776 represented one of the most dramatic periods in world history when people took the principle of freedom into their own hands. But not all Americans were free on July 4, 1776—and not all are free today, either. What do we celebrate when we set off fireworks and have cook outs on Independence Day, and what histories are swept under the rug in the process?

Perhaps most obviously, many people in the colonies were still enslaved in 1776. By the Civil War, there were nearly four million people enslaved in the US. These people did not have the rights of American citizens, even though many of them were born on American soil, and all of them were foundational to the new nation’s economic power. This map drawn after the Mexican-American War shows free and slave states. It also shows territories that had yet to be carved into states in 1848; the debate over whether these territories should permit slavery was one of the sparks that led to the Civil War. The map is surrounded by state emblems, emphasizing the tenuous balance between states which did and did not perpetuate a system of unfreedom through bondage.

Discussions of slavery often focus on the south, including in this famous map of enslaved populations in southern states. However, even in states which did not permit slavery outright, a lot of wealth derived from the slave system. In Massachusetts, for example, although slavery was illegal starting in 1783, the clothing mills still operated based on slave-picked cotton, rum for distilleries came from plantations in the Caribbean, and many more of the goods that were shipped through Boston and made it a wealthy port city were created through slave labor.

The Fifteenth Amendment is one of the amendments to the Constitution made during Reconstruction, after the Civil War and the national abolition of chattel slavery. The amendment sought to secure voting rights for Black men following their release from enslavement. This lithograph has a celebratory feel to it, illustrated as it is with famous people of the time who contributed to what was seen as the advancement of African Americans. These include Frederick Douglass, Abraham Lincoln, Martin Dulany, and Hiram Rhodes Revels. In spite of the congratulatory air of this piece, voting was still denied to most Black Americans through poll taxes and tests. And while the 13th Amendment prohibited slavery, it carved out an exemption if the servitude was set as punishment for a crime. The United States has the highest population of incarcerated people of any country in the world. Though laws vary state by state, many incarcerated people are prohibited from voting, and some perform uncompensated labor.

Also illustrated in the lithograph above are scenes of Black children attending school, a freedom denied to them during slavery. Schools were not universally a source of freedom, though. Across the continent, young Indigenous people were forced to attend boarding schools or reservation schools that would teach them to be American rather than Native. Recent news from Canada about the bodies of children found at residential schools is appalling. Canada is not alone in its genocidal practices against Indigenous people in the Americas. The US also had residential schools which functioned as a way to “kill the Indian, save the man” in each child torn from their families and culture and forced to assimilate or suffer the consequences. This map of American reservations marks different types of schools with several red symbols.

The right to vote is supposedly a founding principle of the United States, but who could vote at the outset of our country? Not Black men, as discussed above, and certainly not Native Americans, who were not citizens until 1924. Women also did not have the vote in 1776; only white, propertied men could choose their representatives in government. This 1919 Victory Map shows each state’s position on women’s suffrage and includes the year women obtained the vote in each place.

Today, populations in some cities are still de facto prevented from voting because of long lines and limited hours at polling places. Residents without citizenship, people convicted of felonies, residents in some territories, and others do not have the vote.

For all of these reasons and more, the project of freedom that was set in motion on July 4, 1776 took the work of many generations to perfect, and it’s a project that’s still ongoing today. As the fireworks go off today, think about both the hard work which has been done to “let freedom ring,” and how much work remains to be done. Happy Independence Day!