Apparently the stock starter doesn’t like the heat coming from the headers.
After spending some time reading about heat soak issues, I decided that a mini starter hopefully was the answer to the problem.
Apparently the mini starter which I bought from a swedish dealer originated from 440Source (which I didn’t know when ordering it) and according to what I’ve been reading online about these starters, some people has had heat soak problems with these as well.
Just to make sure that I can rely on that this starter won’t fail on me, I got a starter heat shield from Thermo Tec which we wrapped up the starter in. Hopefully this is the end of my heat soak problems!