Was your website already listed in google and how old was it? I've seen it only impact sites that were super new or did not already have some sort of page linking most pages together. For instance a website that was near 6 years old didn't see very much impact when I submitted a sitemap and it got accepted.