So I went to Azure portal, selected directory, chose Directory Integration tab and clicked Deactivate button. Azure started to work and status turned to Deactivating... I waited and waited, but it was stuck.
As it turned out this process can take up to 72 hours, I didn't measure exactly it in about three days Directory Integration status switched to Deactivated.
So waiting is the answer :)