expr1 =
Let's look at the systems of equations you're trying to solve.
Ax = [0 1 0 0; 0 -0.1 0 0.8; 0 0 -4 9; -1 -9 2 -8];
Bx = [0 0 0 0; 0 0 -0.5 0; -2 -2 0 2; 4 0 0 0];
syms a01 a02 a03 a04 a11 a12 a13 a23 a14
expr1 = Ax*Ax*Bx(:,1) == a01*Bx(:,1) + a11*Ax*Bx(:,1)
This one clearly has no solution, since 16/5 is not equal to 0.
expr2 = Ax*Ax*Bx(:,2) == a02*Bx(:,2)+a12*Ax*Bx(:,2)
Again, no solution since -16/5 is not equal to 0.
expr3 = Ax*Ax*Ax*Bx(:,3) == a03*Bx(:,3)+a13*Ax*Bx(:,3)+a23*Ax*Ax*Bx(:,3)
This one might have a solution. We can use the third equation to get the value of a_23 and then use either the first or fourth equation to get the value of a_13. We can then use the other of the fourth or first equation to check if those values of a_23 and a_13 are consistent.
a_23 = solve(expr3(3))
a_13 = solve(subs(expr3(1), a23, a_23))
check = subs(expr3(4), [a23 a13], [a_23, a_13])
simplify(check)
So no, the equations are not consistent.
expr4 = Ax*Ax*Bx(:,2) == a04*Bx(:,4)+a14*Ax*Bx(:,4)
And again, no solution because of the second equation.
So MATLAB returning the empty 0-by-1 sym for each of those systems is correct.