During a challenge
When the challenge is active, you can remove your submission by withdrawing from the challenge. Your submission will not go into judging and will not show up on your profile or on ArtStation.
After a challenge is complete
Once a challenge is completed, you can only hide a submission from your profile. Go to your profile, click on "Challenges", then on the challenge submission there should be a drop down that allows you to hide the submission from public.
Note that hiding the submission on your profile does not remove the challenge submission itself. People can still access it by visiting the challenge page and browsing the submissions. When a challenge ends, it is in an archived state and because we have to preserve the standings and results of the challenge, submissions cannot be removed from the challenge itself. It can only be hidden on your profile.