How to form final template for entire database?

hello,i have a database with some files.. step 1: I calculated row vector for all the files and stored in variable 'Rowvector'. Following is the code to form row vector.

D = 'dir name;                                   
S = dir(fullfile(D,'*.txt'));
Rowvector = cell(numel(S),1);                                               
for k = 1:numel(S)
    M = dlmread(fullfile(D,S(k).name));
    Rowvector{k} =[de2bi(M(:,1),9),de2bi(M(:,2),10),de2bi(M(:,3),9)];      %converting 3 elements from a row to 9,10,9 bits,totally 28 bits for a single row
end

step 2: Now i need to calculate how many 0's and 1's are in each row in a file..if it have 10 0's and 18 1's in a single row,then p =10/28 and q= 18/28..so,likewise i need tto calculate each row from individual files and write theor probability in L and O variable's..so L and O must have same cell size as row vector,but each file inturn size changes as there size varies for different files... i calculated for a single file..i want for all the files..

Z=Rowvector{1,1}(:,:);                                                   %one file
[m n]=size(Z);
L=cell(m,1);                                                     %0th probability cell
O=cell(m,1);                                                     %1th probability cell
p=0,q=0;
for j=1:m
for i=1:n
    if(Z(j,i)==0)
        p=p+1;
    else
        q=q+1;
    end
end
p=p/m;
q=q/m;
L{j}=p;
O{j}=q;
end

STEP 3: next,after calculating probability values,now i have a threshold valu of 0.3 if suppose,now if a consider a row in a file,if the 1st bit is 0 then i need to compare the probability value of first row of first file in L and some calculations and if bit is 1,then i need to compare with the probability value of first row of first file in O..So,last i must have 'Const' cell with same size of row vector... i have done for single file...need to do for all files. CODE for 1 file :

z1=Rowvector{1,1}(:,:);        
Const=cell(1,m);                                       %Consistent vector of 1st file
for b=1:m
for a=1:n
    if(z1(b,a)==0)
        if(L{b}>=0.3)
            Const{b}{a}=1;
        else
            Const{b}{a}=0;
        end
    else
        if(O{b}>=0.3)
            Const{b}{a}=1;
        else
            Const{b}{a}=0;
        end
    end
end
end

STEP 4:Now,if a consider a single file in Const cell,i have to divide the cell into 16 or 8 bit size.if file size is 1*1624.now i have to convert every 8 binary elements from 1624 elements into a decimal number and store in an array..so,the final output must be of size 1*203(each element in 203 must consists of the value of each 8 binary elements since 8*203=1624)..but if i need to divide this 1624 bits into each 16 cells,then i will have 16*101 cell and 8 bits will be remaining..these 8 bits must store in another cell..so,finally the output must be 16*102..since 101 columns will have each 16 bit values and 102th column must have remaining 8 bits..now i need to convert these 102 column values into decimal values...the final output must be a row vector of size 1*102.. CODE: i did for 1 file,i need to calculate for whole database..

F={[Const{:}]};                                         
 r=F{1,1}(:,:);                                         %consistent row vector
 M1  = reshape([r{:}], 8, []);                          %converting a single row into a column wise with 8 bits.
Decvector = [128, 64, 32, 16, 8, 4, 2, 1] * M1;         %decvector of size 203 since each elemnt in decvec is of converted decimal num of 8 bit binary num from constrow vector

   LUT = mod( reshape(randperm(256*8), 256, 8), 2 );      %lookup table with size 256,to map each decimal number of size 2^8 from decvec(since each dec num is 8 bit bin num)
   FT=cell(1,812);                                        %final template with size 812 since every 4 bits from 203 cells of decimal vector
   for i1=1:203
       a1=Decvector(1,i1);
       ft{i1}=LUT(a1+1,(3:6));                            %extracting 3:6 bits from lookup table

   end

I have done for a single file and i got the final template,but problem is i have to do for entire databse,if i do with my code,then system will surely hang up..so,please solve it for me by reducing lines of code..

Answer 1

[Guillaume]

Well, you certainly like to overcomplicate the code and overuse cell arrays.

Step 2: p is simply the sum of 1s in each row divided. By definition q is 28-p. And therefore L is p/28 and O is 1-L. In other word no need to calculate q or O. The whole step 2 is simply:

p = sum(Rowvector{idx}, 2);
L{idx} = p/28;

Next, step 3, Remember that if O is 1-L by definition, so if L>=0.3 then O<0.3, so really all your step 3 does, is:

• if L >= 0.3, invert the bits of RowVector;
• otherwise leaves as is

That can be coded very simply:

Const{idx} = mod(Rowvector{idx} + (L{idx} >= 0.3), 2);

And note that there is no point making the content of Const{idx} a cell array as your original code did.

Step 4: yourlast bit doesn't make much sense. You create a cell array of 812 columns but only populate the first 203 of them, each with 4 elements. I'm assuming what your really want is a row vector of bits:

DecVector = 2.^(7:-1:0) * reshape(Const{idx}', 8, []);
LUT = mod(reshape(randperm(256 * 8), 256, 8), 2 ); but what is the point of an 8 column lut if you only use columns 3:6?
FT{idx} = reshape(LUT(DecVector', 3:6)', 1, []);

So all in all, the whole code:

D = dirname;                                   
S = dir(fullfile(D, '*.txt'));
LUT = mod(reshape(randperm(256 * 8), 256, 8), 2 );  %assume you use the same LUT for all files
binmatrices = cell(numel(S),1); %Changed name from RowVector as binmatrices explain better what's in it
L = cell(numel(S), 1);   %not a very good name
Const = cell(numel(S), 1);  %not a very good name
FT = cell(numel(S), 1);   %not a very good name
for fidx = 1:numel(S)
    M = dlmread(fullfile(D, S(k).name));
    binmatrices{fidx} =[de2bi(M(:, 1), 9), de2bi(M(:, 2), 10), de2bi(M(:, 3), 9)];      %converting 3 elements from a row to 9,10,9 bits,totally 28 bits for a single row
    p = sum(binmatrices{fidx}, 2);
    L{fidx} = p/28;
    Const{fidx} = mod(binmatrices{fidx} + (L{fidx} >= 0.3), 2);
    DecVector = 2.^(7:-1:0) * reshape(Const{fidx}', 8, []);
    FT{fidx} = reshape(LUT(DecVector', 3:6)', 1, []);  
end

That's all

[Jyothi Alugolu]

In step 4,if i want to divide 1624 bits into each 16 cells,then i will have 16*101 cell and 8 bits will be remaining..these 8 bits must store in another cell..so,finally the output must be 16*102..since 101 columns will have each 16 bit values and 102th column must have remaining 8 bits..now i need to convert these 102 column values into decimal value....and this 1624 file size is not common for all the files,since there are many files in the folder.

[Guillaume]

You need to explain yourself a lot better. You say you have 1624 bits, but then say that you don't. My understanding:

You want to group the Const bit into 16 bits and convert these into decimal values. There may be leftover bits (if your rows as 28 bits, it can be 4, 8, or 12 leftover bits) which must also be converted (one assumes with padding, but you've not explained properly). In which case:

padding = mod(numel(Const{fidx}, 16)
groupedbits = reshape([Const{fidx}(:); zeros(padding, 1)], 16, []);
DecVector = 2.^(15:-1:0) * groupedbits;

Again there's absolutely no need for cells. It just complicates everything.