By "I have to worry" you mean: "Will I risk a memory leak?" or "could it lead to data loss?" or "does this affect how the client sees and closes the connection?"
Saying, it is considered polite to issue a closing command to the client. The client may see a sharp gap in the connection instead of closing. For example, if it is a web server and you do not send a close, then some browsers will read it as a line break (error) and immediately try to connect again. Closing the connection correctly is just polite, and it will help to logically organize your code (you know exactly how all your resources are processed).
I am not 100% sure how erlang handles garbage collection, but I can say that for most programming languages (and thus I also assume erlang), by killing the process, it will take care of all the memory associated with this, therefore there should not be memory leaks, just letting the process die.
If, however, there is potential content left in the buffer that will be sent to the client, you can make your exit procedure so that it deletes the buffer or connection before it is closed, otherwise you may risk data loss.
Edit : as mentioned in Legoscia, any open file descriptors should be handled erlang gracefully, but it will never hurt (and I would call it “good practice”) to close them myself before leaving the process.