Following this blog post, I was able to shrink my repo size from 110MB to 3.8MB, .git directory went from 107MB to 1.3MB. In addition to instructions on how to cleanup your repo history, the blog post also provides a script for finding big files in your repository, which was not necessary for me as I knew the location of the files/directory to be deleted.
A short summary of what I did, so you do not need to go through the complete blog post (Following these instructions will lead to irreversible changes in your commit history and potentially to unwanted loss of data. Only apply when you are 100% sure that you know what you are doing):
$ git filter-branch --tree-filter 'git rm -rf .config --ignore-unmatch' HEAD $ rm -rf .git/refs/original/ $ git reflog expire --expire=now --all $ git gc --aggressive --prune=now $ git push origin master --force