[x1,y1] = solve(dot(a,b)-1,dot(a,c)-2)
When you pass a string into solve(), the string is interpreted strictly in terms of MuPAD calls. MuPAD does not have a function named dot(). In MuPAD, the dot product routine is named linalg::scalarProduct()
Also keep in mind that when you pass a string to solve() and the string refers to a variable that has been defined at the MATLAB level, then unless you somehow transfer the variable to MuPAD then MuPAD is not going to know about the value of the variable.
Basically the only time you should pass strings to solve() is as variable names or when you know you are writing a MuPAD expression that the interface to the Symbolic Toolbox does not provide an easy way to write. For example in MATLAB up to about R2011b, MATLAB did not recognize the "==" operator on symbolic expressions in any useful way, so it was necessary to pass in strings to solve() that used the MuPAD equivalent, the "=" operator
solve('zeta(cos(x), 5) = besselJ(3,sin(x))')
but otherwise do not pass in strings.