とはThat..., (Surprised) that..., (Never expected) that..., (Shocked) that...

なんて can overlap in meaning with とは when used in phrases like 優勝できる(とは)・（なんて）夢にも思わなかった ('Never even in my dreams did I think we could win the championship.'). なんて however has a variety of usages that do not apply to とは, such as to comment (often negatively) on 'things like (A)', e.g. 嘘なんて大嫌い ('I hate things like lies.').

Both expressions can be used to convey shock and surprise, though how they function in a sentence differs. まさか precedes whatever the speaker is surprised about, e.g. まさか勝てるなんて ('To think they could win!'), or even on its own as an interjection, i.e. まさか！ ('No way!). In contrast, とは always follows what the speaker is surprised about, and can pair with まさか for further emphasis, e.g. まさか勝てるとは (same meaning as the above example).

まさか and とは are both used to express surprise, but function in different ways. まさか comes at the beginning of a sentence to emphasize that what follows goes against the speaker's expectations, while とは appears at the end of a sentence or phrase to express the same. Because they occupy different positions, they are often used together. For example, まさか勝てるとは！ ('To think we could win!' / 'I never thought we could win').