A lot of the history has to do with crabbing, oystering and fishing. there was also a massive fight over the bays oysters at a time and Annapolis was a hub for that  Earlier in its life it was actually also known for horse racing but that has died out as time went on. I would go on and on about the history but if you want all the details head to. https://www.annapolis.gov/588/History-of-Annapolis