Any procedure that follows a pattern:
BEGIN TRAN check if row exists with SELECT if row doesn't exist INSERT COMMIT
going to face difficulties in production, because there is nothing to prevent the simultaneous execution of two protectors, and both reach the conclusion that they should be inserted. In particular, at the isolation level of serialization (as in your case) this template is guaranteed to be a dead end.
A much better template is to use unique database restrictions and always INSERT, to fix errors duplicating key errors. It is also significantly more productive.
Another alternative is to use the MERGE statement :
create procedure usp_getOrCreateByMachineID @nMachineId int output, @fkContract int, @lA int, @lB int, @lC int, @id int output as begin declare @idTable table (id int not null); merge Machines as target using (values (@nMachineID, @fkContract, @lA, @lB, @lC, GETDATE())) as source (MachineID, ContractID, lA, lB, lC, dteFirstAdded) on (source.MachineID = target.MachineID) when matched then update set @id = target.MachineID when not matched then insert (ContractID, iA_Metric, iB_Metric, iC_Metric, dteFirstAdded) values (source.contractID, source.lA, source.lB, source.lC, source.dteFirstAdded) output inserted.MachineID into @idTable; select @id = id from @idTable; end go create procedure usp_getOrCreateByMetrics @nMachineId int output, @fkContract int, @lA int, @lB int, @lC int, @id int output as begin declare @idTable table (id int not null); merge Machines as target using (values (@nMachineID, @fkContract, @lA, @lB, @lC, GETDATE())) as source (MachineID, ContractID, lA, lB, lC, dteFirstAdded) on (target.iA_Metric = source.lA and target.iB_Metric = source.lB and target.iC_Metric = source.lC) when matched then update set @id = target.MachineID when not matched then insert (ContractID, iA_Metric, iB_Metric, iC_Metric, dteFirstAdded) values (source.contractID, source.lA, source.lB, source.lC, source.dteFirstAdded) output inserted.MachineID into @idTable; select @id = id from @idTable; end go
This example separates the two cases, since T-SQL queries should never try to resolve two different solutions in the same query (the result is never optimized). Since the two tasks at hand (get by mahcine id and get by metrics) are completely separate, there must be separate procedures, and the caller must call apropiate, and not pass the flag. This example assumes how to achieve the (possibly) desired result using MERGE, but, of course, the correct and optimal solution depends on the actual scheme (determining the table, indices and coordinates in place) and on the real requirements (it is not clear that the procedure should be performed if the criteria have already been agreed upon, and @id is not displayed?).
By eliminating SERIALIZABLE isolation, this is no longer guaranteed by a deadlock, but can still slow down. Of course, the solution to the deadlock completely depends on the scheme that was not indicated, therefore, in this context it is impossible to provide a solution to the deadlock. There is a sledgehammer of blocking all possible strings (the power of UPDLOCK or even TABLOCX), but such a solution will kill throughput with heavy use, so I can not recommend it without knowing the use case.