Try it (just cut and paste this):
Needs["Notation`"] Notation[ParsedBoxWrapper[ RowBox[{"A_", " ", "\[FilledSmallCircle]", " ", "B_"}]] \[DoubleLongLeftRightArrow] ParsedBoxWrapper[ RowBox[{"Which", "[", RowBox[{ RowBox[{ RowBox[{"(", RowBox[{ RowBox[{"MatrixQ", "[", "A_", "]"}], "||", RowBox[{"VectorQ", "[", "A_", "]"}]}], ")"}], "&&", RowBox[{"(", RowBox[{ RowBox[{"MatrixQ", "[", "B_", "]"}], "||", RowBox[{"VectorQ", "[", "B_", "]"}]}], ")"}]}], ",", RowBox[{"A_", " ", ".", "B_"}], ",", RowBox[{ RowBox[{"!", RowBox[{"(", RowBox[{ RowBox[{"MatrixQ", "[", "A_", "]"}], "||", RowBox[{"VectorQ", "[", "A_", "]"}]}], ")"}]}], "&&", RowBox[{"(", RowBox[{ RowBox[{"MatrixQ", "[", "B_", "]"}], "||", RowBox[{"VectorQ", "[", "B_", "]"}]}], ")"}]}], ",", RowBox[{"A_", "[", "B_", "]"}], ",", RowBox[{ RowBox[{"(", RowBox[{ RowBox[{"MatrixQ", "[", "A_", "]"}], "||", RowBox[{"VectorQ", "[", "A_", "]"}]}], ")"}], "&&", RowBox[{"!", RowBox[{"(", RowBox[{ RowBox[{"MatrixQ", "[", "B_", "]"}], "||", RowBox[{"VectorQ", "[", "B_", "]"}]}], ")"}]}]}], ",", RowBox[{"Transpose", "[", RowBox[{"B_", "[", RowBox[{"Transpose", "[", "A_", "]"}], "]"}], "]"}]}], "]"}]]]
Now I entered this with the Notation palette, so on the screen it looks like this:
(if necessary, the palette inserts various boxes). It looks awful when I cut and paste due to the explicit string representation of everything.
EDIT: That is: type "Needs["Notation "]`, resulting in a palette. Click on the first button, and then

appears. Inside the first yellow rectangle is A_ \[FilledSmallCircle] B_ , and in the second -
Which[(MatrixQ[A_]||VectorQ[A_])&&(MatrixQ[B_]||VectorQ[B_]),A_ .B_,!(MatrixQ[A_]||VectorQ[A_])&&(MatrixQ[B_]||VectorQ[B_]),A_[B_],(MatrixQ[A_]||VectorQ[A_])&&!(MatrixQ[B_]||VectorQ[B_]),Transpose[B_[Transpose[A_]]]]
The result looks like this: 
and, when evaluated, determines what you want. Also, after the Needs bit, just cut and paste what I gave above.