Fully agreed. I, too, won't consider this an error.
This only makes sense if all the terms in the matrix are of the same type.
And thats the reason the numeric processor fails - its aware of units. While I don't consider it a bug or error I can see that it could be convenient if Prime would see the unit not as a real unit but rather as a simple factor in that case.
The symbolic processor is not aware of units and treats it as normal (unknown) variables. Consequently a symbolic evaluation works.
