Can I get help optimizing a signal space?

I am trying to optimize a signal constellation but I am new to optimization. Let's say I am transmitting three frequencies simultaneously. I am able to uniquely identify each component as coming from a separate, specific transmitter. I want to treat each of the three signals as having their own individual constellations. I want to design the constellations as to minimize the probability of bit error, or design it so that the minimum distance between any two symbols is maximized. An individual symbol or code word consists of a point from each of the three constellations simultaneously. In other words, I want to design a "code book" or set of symbols which will maximize the minimum distance between symbols. I have been trying to use optimtool and fmincon in the optimization toolbox, but I can't seem to set my problem up correctly. Here is what I have.

Objective function script file:

function obj = QRS_OBJ( x )
%QRS_OBJ Summary of this function goes here
% Detailed explanation goes here
M = length(x)/3;
%in this example, M==4
d = [];
for idx= 1:M
    Q(idx) = x(3*idx-2);
    R(idx) = x(3*idx-1);
    S(idx) = x(3*idx);
end
d(1) = -sqrt( ((Q(1)-Q(2)).^2) + ((R(1)-R(2)).^2) + ((S(1)-S(2)).^2)); % "-" because we want to maximize
d(2) = -sqrt( ((Q(1)-Q(3)).^2) + ((R(1)-R(3)).^2) + ((S(1)-S(3)).^2));
d(3) = -sqrt( ((Q(1)-Q(4)).^2) + ((R(1)-R(4)).^2) + ((S(1)-S(4)).^2));
d(4) = -sqrt( ((Q(2)-Q(3)).^2) + ((R(2)-R(3)).^2) + ((S(2)-S(3)).^2));
d(5) = -sqrt( ((Q(2)-Q(4)).^2) + ((R(2)-R(4)).^2) + ((S(2)-S(4)).^2));
d(6) = -sqrt( ((Q(3)-Q(4)).^2) + ((R(3)-R(4)).^2) + ((S(3)-S(4)).^2));

%use max(d) to find maximum using fmincon
min_d = max(d)
obj = min_d;
%the objective is to find the maximum possible minimum value between symbols

In optimtool I entered:

Aeq: [1,1,1,0,0,0,0,0,0,0,0,0;0,0,0,1,1,1,0,0,0,0,0,0;0,0,0,0,0,0,1,1,1,0,0,0;0,0,0,0,0,0,0,0,0,1,1,1]
Beq:
[0,0,0,0]
%These constraints are so that each symbol's individual components will sum to 0

Lower bound: [-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1]
Upper bound: [1,1,1,1,1,1,1,1,1,1,1,1]
%I want each symbol's components constrained between -1 and 1

Starting value is: [1,0,-1,.1,0,-1,1,0,-1,1,0,-1]

After doing all of this, it doesn't seem to be working. I end up with repeat constellation points, which I do not want. I think I need a uniqueness constraint somehow, and that may not be all. Any help would be appreciated.

Answer 1

[Matt J]

Well, first of all, I recommend using FMINIMAX instead of FMINCON and, for differentiability's sake, getting rid of the sqrt's. Other than that, it appears this problem has some local minima, so I recommend not intializing with repeat points, if you want a solution with the same. I get reasonable-looking results from the following:

Aeq=kron(eye(4),[1 1 1]);
beq=[0;0;0;0];

lb=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1];
ub=-lb;

S=[1 -1 0 0; 1 0 -1 0; 1 0 0 -1; 0 1 -1 0;0 1 0 -1; 0 0 1 -1].';
     %differencing matrix

F=@(x) -sum((x*S).^2   ,1);

x0=rand(3,4),

[x,fval]=fminimax(F, x0,[],[],Aeq,beq,lb,ub),

[Matt J]

Christopher Hansen commented:

Thank you, that does yield something very similar to what I thought an ideal solution would resemble. By extension, what would you have to change for an x vector with twice as many points, or M==8? Would it look something like this?

Aeq=kron(eye(8),[1 1 1]);
beq=[0;0;0;0;0;0;0;0];

lb=[-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1,-1];
ub=-lb;

S=[1 -1 0 0; 1 0 -1 0; 1 0 0 -1; 0 1 -1 0;0 1 0 -1; 0 0 1 -1].';
     %differencing matrix

F=@(x) -sum((x*S).^2   ,1);

x0=rand(3,8),

[x,fval]=fminimax(F, x0,[],[],Aeq,beq,lb,ub),

I'm not sure what to set S to, and is x0 correct in this case?

Thank you, I wish had a better grasp of optimization but I'm only just beginning.

[C Hansen]

Looking at the code, I'm not understanding how lines 5 and 6 work. Once again, sorry, I'm new to this.

[Matt J]

The result of D=x*S is a matrix whose columns D(:,i) are the pairwise differences of the columns of x. In the 6th line, when you take the (negative) square norms of those differences, you get the objective function vector F(x) whose elements you are trying to maxmin.

To generalize the construction of S for M points, you might do something like this

 I=nchoosek(1:M,2);
 N=size(I,1);

 S=sparse( I(:,1), 1:N, 1, M,N) - sparse( I(:,2), 1:N, 1, M,N);

[C Hansen]

Thank you for your response. That worked very well. Here is the general solution, and using M as a parameter:

M=8;

Aeq=kron(eye(M),[1 1 1]);

beq = repmat(0, M, 1);

lb=repmat(-1, 1, M*3)

ub=-lb;

I=nchoosek(1:M,2);

N=size(I,1);

S=sparse( I(:,1), 1:N, 1, M,N) - sparse( I(:,2), 1:N, 1, M,N);

F=@(x) -sum((x*S).^2 ,1);

x0=rand(3,M),

[x,fval]=fminimax(F, x0,[],[],Aeq,beq,lb,ub),