Hi all!
I have two files. One of which, I read into a SQLite database and the other I read line by line and search the SQLite database for the lines from the file. It works, but it's really slow, so I'm wondering if it's my process or just the nature of the beast.
File2 (FSTEST.dat) is 8mb and File1 (ESTEST.dat) is 1.5mb. These are the smaller of the sets of files (the other sets can be upwards of 50mb or so).
Is there a way to make this run faster? Thanks for the help as always!
AutoIt
#include <File.au3> #include <SQLite.au3> #include <SQLite.dll.au3> Global $hQuery, $aRow, $fl_array2 Global $sTempDB = @ScriptDir & "\TempDB.db" Global $iTotalCount = 0, $iTotalSoFar = 0 Global $iKey1_a = 1 Global $iKey1_b = 12 Global $iKey2_a = 18 Global $iKey2_b = 25 Global $st = TimerInit() Global $iOrphans, $iDupes, $iMatches ; --------------------------------------------- ; Start SQLite ; --------------------------------------------- _SQLite_Startup() If @error Then ConsoleWrite('error loading sqlite.dll' & @LF) Exit EndIf If FileExists($sTempDB) Then FileDelete($sTempDB) _SQLite_Open($sTempDB) If @error Then ConsoleWrite('Unable to open DB' & @LF) _Exit() EndIf ; --------------------------------------------- ; Get the files to compare ; --------------------------------------------- $sFile1 = FileOpenDialog("Select EXTEST.DAT", @ScriptDir, "Dat files (*.dat)", 1 + 2, "") If @error Then _Exit() $sFile2 = FileOpenDialog("Select FSTEST.DAT", @ScriptDir, "Dat files (*.dat)", 1 + 2, "") If @error Then _Exit() ; --------------------------------------------- ; Load the data from File2 into the db ; --------------------------------------------- _ReLoad() ; --------------------------------------------- ; Compare the fields from File1 to those in File2 ; --------------------------------------------- CompareItems($sFile1) ; --------------------------------------------- Func CompareItems($sFileContents) Dim $aSplit _FileReadToArray($sFileContents, $aSplit) If IsArray($aSplit) Then If $aSplit[1] = "" Then _ArrayDelete($aSplit, 1) $aSplit[0] -= 1 EndIf $iTotalCount += $aSplit[0] For $i = 1 To $aSplit[0] $sPart1 = StringMid($aSplit[$i], $iKey1_a, ($iKey1_b - $iKey1_a) + 1) $sPart2 = StringMid($aSplit[$i], $iKey2_a, ($iKey2_b - $iKey2_a) + 1) If $sPart1 <> "" Then $iCount = 0 _SQLite_Query(-1, 'select * from compare where PART1 = "' & $sPart1 & '" and PART2 = "' & $sPart2 & '";', $hQuery) While _SQLite_FetchData($hQuery, $aRow) = $SQLite_OK If $aRow[0] <> "" Then $iCount += 1 WEnd ConsoleWrite($sPart1 & " $iCount = " & $iCount & @CRLF) If $iCount = 0 And $i <> $aSplit[0] Then $iOrphans += 1 EndIf If $iCount = 1 Then $iMatches += 1 EndIf If $iCount > 1 Then $iDupes += 1 EndIf EndIf $iTotalSoFar += 1 Next ConsoleWrite("---------------------------" & @CRLF) ConsoleWrite("Orphans: " & $iOrphans & @CRLF) ConsoleWrite("Matches: " & $iMatches & @CRLF) ConsoleWrite("Dupes: " & $iDupes & @CRLF) ConsoleWrite("Total: " & $iTotalCount & @CRLF) EndIf EndFunc ;==>CompareItems Func _ReLoad() ; drop "parts" table if it exists, re-define and reload it If _SQLite_Exec(-1, 'drop table if exists compare;') <> $SQLite_OK Then ConsoleWrite('Drop table failed' & @LF) _Exit() Else ConsoleWrite('Compare table dropped for refresh' & @LF) EndIf If _SQLite_Exec(-1, 'create table compare (PART1, PART2);') <> $SQLite_OK Then ConsoleWrite('Create Table Failed' & @LF) _Exit() EndIf _FileReadToArray($sFile2, $fl_array2) Switch @error Case 1 ConsoleWrite('Input file failed to open' & @LF) _Exit() Case 2 ConsoleWrite('Unable to split file' & @LF) _Exit() EndSwitch Local $aLine, $sql ProgressOn('Loading Compare Table', 'Please Wait', '', Default, Default, 2 + 16); 2 - without always on top / 16 - Window can be moved _SQLite_Exec(-1, "begin immediate;") For $1 = 1 To $fl_array2[0] ProgressSet(($1 / $fl_array2[0]) * 100) $sql = 'insert into compare values ("' & _ StringMid($fl_array2[$1], $iKey1_a, ($iKey1_b - $iKey1_a) + 1) & '", "' & _ StringMid($fl_array2[$1], $iKey2_a, ($iKey2_b - $iKey2_a) + 1) & '");' If _SQLite_Exec(-1, $sql) <> $SQLite_OK Then ConsoleWrite('Table insert failed STMT = ' & $sql & @LF) _Exit() EndIf Next _SQLite_Exec(-1, "commit;") ProgressOff() ConsoleWrite('Table loaded with ' & UBound($fl_array2) - 1 & ' records in ' & Round(TimerDiff($st) / 1000, 3) & ' seconds' & @LF) EndFunc ;==>_ReLoad Func _Exit() _SQLite_Close() _SQLite_Shutdown() Exit EndFunc ;==>_Exit
(I don't know why the code box always loses my tab structure...weird)