Do not read your code because I am not familiar with the classes you use, but I know how to fix this problem. The problem occurs when the deleted row item is not the last due to an index mismatch between the catalytic index (your ArrayList for storing data) and the final int position parameter in mehod onBindViewHolder . Let me explain the problem graphically:
Suppose we have a RecyclerView with three rows, and we delete the 3rd row in the datalist 2 index (note that the deleted one is not the last element) after removing the datalist index for the last element reduces 3 out of 2, however, final int position still retrieving its positon as 3, and this causes a problem. Thus, after deletion, you cannot use final int position as the index of the element to be removed from the datalist.
To fix this, enter int shift=0 and in the while loop, try deleting the third element in position (position shift), and if it does not work (in my case, it will) increase the offset and try to delete it again until an exception occurs.
public void onBindViewHolder(RecyclerView.ViewHolder holder, final int position) { ... holder.removeButton.setOnClickListener(new View.OnClickListener(){ //button used to remove rows @Override public void onClick(View view) { if (position == dataList.size() - 1) { // if last element is deleted, no need to shift dataList.remove(position); notifyItemRemoved(position); } else { // if the element deleted is not the last one int shift=1; // not zero, shift=0 is the case where position == dataList.size() - 1, which is already checked above while (true) { try { dataList.remove(position-shift); notifyItemRemoved(position); break; } catch (IndexOutOfBoundsException e) { // if fails, increment the shift and try again shift++; } } } } }); ... }