Adding @Thomas to the answer, there is a post on the MSDN forum that says that
"You can only remove it from the store, removing applications from the control panel is not possible after they are published."
The link added to the response also provides additional information about the app’s visibility in the store (scroll down to Distribution and visibility ).
Update
At the moment, a new option appears when you go to the developer's toolbar and click on the corresponding application (on the left). On the next page, a new link appeared in the upper right corner with the name Make app unavailable , which you can click. After confirming the pop-up message
Making this app unavailable will prevent anyone from finding it in the Store, and no new customers will be able to get it. This will override any availability settings in your submissions. To undo this later, click Make app available from the app overview page. Learn more Are you sure you want to make this app unavailable?
You will remove the application from the store.
Gorgsenegger
source share