Info
Cette question est clôturée. Rouvrir pour modifier ou répondre.
Code to search for a word in a file taking too long to execute.
2 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
I have a code that has to process a very large amount of textual data. There is a file, say A which has around 0.4 million sentences, and another file, say B with around 15000 words. For every word in file B, I need to search for that word in file A and so I need to do a strcmp and use the data in file A to return some result. I have currently defined a function which takes a word(from file B) as an argument and searches through all the words in file A to return. This function is called around 15000 times (There are that many words in file B). This is taking ages to complete in MATLAB. However, Python is able to do it with the same method in much less time. Is there a way to improve the speed? The code that I have written :
global Table3;
count_word_tag = 0;
count_tag = 0;
for i = 1:size(Table3,2)
if strcmpi(word,Table3{i})
if mod(i,4)==0
if strcmpi(tag, Table3{i-1})
i
count_word_tag = count_word_tag + str2num(Table3{i-3});
break;
end
end
end
end
for i = 1:size(Table3,2)
if strcmp(tag,Table3{i})
if mod(i,4) == 3
count_tag = count_tag + str2num(Table3{i-2});
end
if isempty(count_tag)
break;
end
end
end
e = count_word_tag./count_tag;
end
And the code where I have called the function is
for i = 1:size(inputTable,2)
e1 = emission(inputTable{i},'O');
e2 = emission(inputTable{i},'I-GENE');
f(i) = max(e1,e2);
disp('iteration no.'),i
end
0 commentaires
Réponses (1)
Cedric
le 16 Août 2013
Modifié(e) : Cedric
le 16 Août 2013
Have you tried using regular expressions? For counting occurrence in file A of words from file B, you would do something like
content_A = fileread('file_A.txt') ;
counts = zeros(1e6, 1) ; % Overshoot prealloc.
words = cell(1e6, 1) ;
wordId = 0 ;
fid_B = fopen('file_B.txt', 'r') ;
while ~feof(fid_B)
wordId = wordId + 1 ;
words{wordId} = fgetl(fid_B) ;
starts = regexpi(content_A, words{wordId}) ; % If case doesn't matter.
%starts = strfind(content_A, words{wordId}) ; % If case matters.
counts(wordId) = length(starts) ;
end
fclose(fid_B) ;
counts = counts(1:wordId) ; % Truncate to true size.
words = words(1:wordId) ;
But I don't really understand what you are doing in your code. How do you read files, how is emission() defined, how do you define word, tag, Table3, inputTable?
2 commentaires
Cedric
le 16 Août 2013
Modifié(e) : Cedric
le 16 Août 2013
Why iteratively? Couldn't you work on the whole cell arrays in one shot? And what happens for example when you use the tag 'I-GENE'? It looks like when strcmpi(tag, Table3{i-1}) is true, which means that Table3{i-1} is 'I-GENE', you are converting this string to num and adding the result to a counter.. which makes little sense. Also, it seems that Table3 has two dimensions; what are they? Then you address this cell array with a unique index i..
I think that the simplest would be to make a small example which shows e.g. 20 entries of Table3, a word and a tag, and explain what you would like to obtain with that.
Cette question est clôturée.
Voir également
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!